`escapeinside` is fragile with some lexers

Issue #1118 new
Geoffrey Poore created an issue

If I understand correctly, escapeinside works by tokenizing to obtain strings and comments, then merging all non-string and non-comments tokens, then scanning these merged tokens for escapes, and finally scanning everything that wasn't escaped for normal tokens. So it isn't possible to have escapes inside of strings and comments. This triple-scan approach can cause escapeinside to fail in non-obvious ways.

For example, in the bash lexer the backslash \ and immediately following character is treated as a String.Escape token. Because of this, standard LaTeX commands of the form \command will always be treated as starting with a type of string, even when they occur in a region "protected" by escapeinside. More details about this particular case are here.

Basically, any time that the LaTeX code contained within an escape looks like a string or comment for the current lexer, escapeinside may fail. If it would be possible to tokenize everything in a single stage, with escapes having the highest priority, that might be a solution.

Comments (0)

  1. Log in to comment