Position bug in RubyLexer's heredoc handling

Issue #366 resolved
created an issue

I already have a fix ready. Here is a real world example of how this confuses the irb lexer.


{{{ irb(main):001:0> puts <<END irb(main):002:0" a irb(main):003:0" b irb(main):004:0" c irb(main):005:0" END a b c => nil }}}

Buggy output (notice all the prompts at the end of the first line):

{{{ irb(main):001:0> puts <<ENDirb(main):002:0" irb(main):003:0" irb(main):004:0" a b c irb(main):005:0" END a b c => nil }}}

Test script:

{{{ from pygments.lexers.agile import RubyLexer curcode= u'puts <<XYZ\na\nb\nXYZ\n' r = RubyLexer().get_tokens_unprocessed(curcode)

pos = 0 for i, t, v in r: if i != pos: print "continuity error", pos, i, t, repr(v) pos += len(v) print "done" }}}

This prints:

{{{ continuity error 10 15 Token.Text u'\n' done }}}

There's a newline whose position is being incorrectly reported, which causes all the insertions to happen too early.

Comments (4)

  1. Log in to comment