Consider adding folds support to aurumannotate buffers. Copying folding information to this buffer is trivial, but opening/closing folds is not. As there is no autocommand for catching fold open/close events, it makes sense to use CursorMoved event for this job.
Alternative: disable folds when using annotate.
Alternative 2: add short version of annotation where different revisions use different colors and markers in signs column. Also make it possible to use folds to fold lines changed in the same revision and statusline to show description of the revision where line under cursor was changed (with powerline support, of course). This alternative deserves to be implemented even if there will be full fold support.