Anonymous avatar Anonymous committed 659ee15

[svn] Make RawTokenFormatter/Lexer use unicode escaped strings.

Comments (0)

Files changed (2)


         lasttype = None
         lastval = u''
         for ttype, value in tokensource:
-            value = value.encode(self.encoding)
+            value = repr(value)
             if ttype is lasttype:
                 lastval += value


                     for ttype_ in ttypes:
                         ttype = getattr(ttype, ttype_)
                     _ttype_cache[ttypestr] = ttype
-                val = val[1:-2].decode('string-escape')
+                val = val[2:-2].decode('unicode-escape')
             yield length, ttype, val
             length += len(val)
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.