We've had performance issues in the past of the form
'string': [ ('"', String, '#pop'), ('\\n', String), ('.', String), ],
where the single
. not being
[^"\\]+ causes many more tokens to be output. This should be fairly easy to check with the existing example files (at least ~4 consecutive single-character tokens of the same type), assuming adequate coverage.