This is also high on my wish list! I find the annotation summary feature in this way actually useless. It would be a great functionality if it shows all annotations, so that you immediately see all relevant sections of the document together in the summary.
Thanks for considering to include this feature!