Japanese characters are mangled when guessing lexer type

Issue #838 resolved
Scott Patten
created an issue

If I take a file with Japanese characters in it, then the output is correct when I tell Pygments what lexer to use, but the characters are mangled when I ask Pygments to guess the lexer.

Using the attached "test.rb" file

cat test.rb | pygmentize -l ruby -O encoding=utf-8


pygmentize -O encoding=utf-8 test.rb

work correctly, while

cat test.rb | pygmentize -g -O encoding=utf-8

results in mangled characters.

This happens in all of the formatters.

Tested on Pygments 1.5 and Pygments 1.6, rc1. It happens on multiple lexers, as well.

Comments (1)

  1. Log in to comment