Token compatibility with Python2
When using the CodeEditor with Python2, pressing a non-ascii character generates an exception in the _findNextToken function when trying to get the string.
There is an effort in the Token class to handle Python2 unicode strings as we can read at line 39:
self.line = ustr(line). But this is not enough, a
__unicode__ function should also be provided. In order to have a Token class compatible with both versions of Python, I am suggesting to replace the
__str__ method by the following lines:
44 if sys.version_info >= 3: # Python 3 45 def __str__(self): 46 return self.__unicode__() 47 else: # Python 2 48 def __str__(self): 49 return self.__unicode__().encode('utf8') 50 51 def __unicode__(self): 52 return self.line[self.start:self.end]