Issue #812 resolved

Ruby 1.9 hash syntax not recognised

Tom Stuart
created an issue

In Ruby 1.8 a hash with a symbol as a key is written { :a => b }, which the lexer handles correctly, but Ruby 1.9's alternative { a: b } syntax doesn't get properly tokenised:

$ echo '{ a: b }' | pygmentize -l ruby -f raw --
Token.Punctuation       u'{'
Token.Text              u' '
Token.Name              u'a'
Token.Punctuation       u':'
Token.Text              u' '
Token.Name              u'b'
Token.Text              u' '
Token.Punctuation       u'}'
Token.Text              u'\n'

In this context the a: should be treated as a Token.Literal.String.Symbol, not as a separate Token.Name and Token.Punctuation.

This incorrect treatment of the : interferes with, for example, the tokenisation of a subsequent regular expression:

$ echo '{ a: /=/ }' | pygmentize -l ruby -f raw --
Token.Punctuation               u'{'
Token.Text                      u' '
Token.Name                      u'a'
Token.Punctuation               u':'
Token.Text                      u' '
Token.Operator                  u'/='
Token.Literal.String.Regex      u'/'
Token.Literal.String.Regex      u' }\n'

Obviously this is wrong; the /=/ should all be treated as Token.Literal.String.Regex, but instead the /= is being consumed as a Token.Operator, with unpleasant consequences for the rest of the string.

Comments (6)

  1. Log in to comment