Commits

gbrandl  committed b4567cf

[svn] Fix unittest.

  • Participants
  • Parent commits dbcaf85
  • Branches trunk

Comments (0)

Files changed (1)

File pygments/lexers/special.py

         elif self.compress == 'bz2':
             import bz2
             text = bz2.decompress(text)
-        return Lexer.get_tokens(self, text)
+
+        # do not call Lexer.get_tokens() because we do not want Unicode
+        # decoding to occur, and stripping is not optional.
+        text = text.strip('\n') + '\n'
+        for i, t, v in self.get_tokens_unprocessed(text):
+            yield t, v
 
     def get_tokens_unprocessed(self, text):
         length = 0
             try:
                 ttypestr, val = match.group().split('\t', 1)
             except ValueError:
-                val = match.group()
+                val = match.group().decode(self.encoding)
                 ttype = Error
             else:
                 ttype = _ttype_cache.get(ttypestr)