Valid Python3 non-ascii identifiers being highlighted as errors in source view

Issue #14449 closed
Brenden Bain staff created an issue

Python3 now allows non-ascii characters to be used as identifiers. The current Python highlighting for Python2. This leads to the highlighting showing invalid errors:

python3.png

Either the detection of the difference between Python2 or Python3 should be better or we should let the user chose Python3 from the UI.

Comments (7)

  1. Brenden Bain staff reporter

    We need to use the Python3Lexer for Python3. For example:

    In [1]: from pygments.lexers import Python3Lexer, PythonLexer
    
    In [3]: data = 'langue_système = something'
    
    In [6]: py2, py3 = PythonLexer(encoding='utf-8'), Python3Lexer(encoding='utf-8')
    
    In [7]: for token in py2.get_tokens(data):
       ...:     print(token)
       ...:     
    (Token.Name, u'langue_syst')
    (Token.Error, u'\xe8')
    (Token.Name, u'me')
    (Token.Text, u' ')
    (Token.Operator, u'=')
    (Token.Text, u' ')
    (Token.Name, u'something')
    (Token.Text, u'\n')
    
    In [8]: for token in py3.get_tokens(data):
       ...:     print(token)
       ...:     
    (Token.Name, u'langue_syst\xe8me')
    (Token.Text, u' ')
    (Token.Operator, u'=')
    (Token.Text, u' ')
    (Token.Name, u'something')
    (Token.Text, u'\n')
    
    In [9]: 
    
  2. Amber Van Hecke staff

    Hi Benjamin, This issue seems to have been fixed in the new source browser. You can turn the new source browser on, by going to Bitbucket Labs in your settings.

    @Brenden Bain if you continue to see people with this issue, please open another ticket so we can look into a proper fix. Thank you!

  3. Log in to comment