Hy doesn't lex unicode literals

Issue #1422 new
Simon Gomizelj
created an issue

I've updated the Hy documentation to use hylang in its code blocks, and we've found a few places where it fails to lex valid Hy code. Most of the problems revolve around REPL output, which is understandable (trying to figure out what's the best practice here), but there are a few snippets that fail.

Consider the following snippet:

=> (deftag  [code]
...  (setv op (last code) params (list (butlast code)))
...  `(~op ~@params))
=> #(1 2 3 +)

When pushing this through pymentize, I hit this:

An unhandled exception occurred while highlighting.
Please report the whole traceback to the issue tracker at

=> (deftag Traceback (most recent call last):
  File "/home/simon/.virtualenvs/hy-eeIqtnUk/bin/pygmentize", line 11, in <module>
    load_entry_point('Pygments', 'console_scripts', 'pygmentize')()
  File "/home/simon/src/pygments-main/pygments/cmdline.py", line 548, in main
    return main_inner(popts, args, usage)
  File "/home/simon/src/pygments-main/pygments/cmdline.py", line 512, in main_inner
    highlight(code, lexer, fmter, outfile)
  File "/home/simon/src/pygments-main/pygments/__init__.py", line 85, in highlight
    return format(lex(code, lexer), formatter, outfile)
  File "/home/simon/src/pygments-main/pygments/__init__.py", line 67, in format
    formatter.format(tokens, outfile)
  File "/home/simon/src/pygments-main/pygments/formatters/terminal.py", line 101, in format
    return Formatter.format(self, tokensource, outfile)
  File "/home/simon/src/pygments-main/pygments/formatter.py", line 95, in format
    return self.format_unencoded(tokensource, outfile)
  File "/home/simon/src/pygments-main/pygments/formatters/terminal.py", line 121, in format_unencoded
    for ttype, value in tokensource:
  File "/home/simon/src/pygments-main/pygments/filter.py", line 20, in _apply
    for token in filter_.filter(lexer, stream):
  File "/home/simon/src/pygments-main/pygments/filters/__init__.py", line 196, in filter
    raise self.exception(value)
pygments.filters.ErrorToken: ↻

Link to Hy issue: https://github.com/hylang/hy/pull/1522

Comments (1)

  1. Log in to comment