Issue #255 resolved

Token compatibility with Python2

Maxime Chéramy
created an issue

When using the CodeEditor with Python2, pressing a non-ascii character generates an exception in the _findNextToken function when trying to get the string.

There is an effort in the Token class to handle Python2 unicode strings as we can read at line 39: self.line = ustr(line). But this is not enough, a __unicode__ function should also be provided. In order to have a Token class compatible with both versions of Python, I am suggesting to replace the __str__ method by the following lines:

 44     if sys.version_info[0] >= 3:  # Python 3
 45         def __str__(self):
 46             return self.__unicode__()
 47     else:  # Python 2
 48         def __str__(self):
 49             return self.__unicode__().encode('utf8')
 51     def __unicode__(self):
 52         return self.line[self.start:self.end]

Comments (6)

  1. Log in to comment