1. Georg Brandl
  2. pygments-main
  3. Issues
Issue #768 duplicate

wrong position yielded for default eol (\n) in RegexLexer

Matthieu Gautier
created an issue

In RegexLexer (get_tokens_unprocessed), if there is no token matching and we reach a '\n' then a Token.Text is yielded.

However, the pos is incremented before yielding the token. This leads to a wrong token's position being yielded. The pos variable should be incremented after yielding the token.

Most of the time this is not a problem cause the next token will start at the right position. This is a problem if the client program uses the token pos to do some action.

Solve is easy : move the "pos +=1" line after the yield instruction.

Comments (1)

  1. Log in to comment