annotate command hits unicode happy fun time

Issue #363 resolved
Tom Callaway created an issue

The annotate command chokes if unicode characters are parsed in the file. Here's a sample file (test.py):

# -*- coding: utf-8 -*-
#
# é

print "foobar"
[spot@localhost master]$ coverage run test.py
foobar
[spot@localhost master]$ coverage annotate -d covdir
Traceback (most recent call last):
  File "/usr/bin/coverage", line 9, in <module>
    load_entry_point('coverage==4.0a5', 'console_scripts', 'coverage')()
  File "/usr/lib64/python2.7/site-packages/coverage/cmdline.py", line 676, in main
    status = CoverageScript().command_line(argv)
  File "/usr/lib64/python2.7/site-packages/coverage/cmdline.py", line 451, in command_line
    directory=options.directory, **report_args)
  File "/usr/lib64/python2.7/site-packages/coverage/control.py", line 924, in annotate
    reporter.report(morfs, directory=directory)
  File "/usr/lib64/python2.7/site-packages/coverage/annotate.py", line 42, in report
    self.report_files(self.annotate_file, morfs, directory)
  File "/usr/lib64/python2.7/site-packages/coverage/report.py", line 80, in report_files
    report_fn(cu, self.coverage._analyze(cu))
  File "/usr/lib64/python2.7/site-packages/coverage/annotate.py", line 92, in annotate_file
    dest.write(line)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 2: ordinal not in range(128)

It looks like the obvious fix is to force encode the line being written to utf-8, a patch is attached against 4.0a5 which does this. This resolves the issue in my testing, but I wanted to make sure I had it right first.

Note: I do not see this issue in the 3.7 code, only in 4.0.

Comments (12)

  1. Vjacheslav Fyodorov

    In python 3 line.encode('utf-8') produces bytes, not string, so we have:

      File "/usr/lib64/python3.4/site-packages/coverage/annotate.py", line 92, in an
    notate_file                                                                    
        dest.write(line.encode('utf-8'))
    TypeError: must be str, not bytes
    

    I have reverted this line back, and all works fine.

  2. Ned Batchelder repo owner

    @fvabit Help me out here. Please provide complete details. What Python program were you measuring? What version of Python were you using? How did you run coverage?

  3. Ned Batchelder repo owner

    @fvabit it looks like you are running the pull request code, not the final code? Can you try the tip on bitbucket and see if it works for you?

  4. Vjacheslav Fyodorov
    $ uname -a
    Linux localhost.localdomain 4.0.6-300.fc22.x86_64 #1 SMP Tue Jun 23 13:58:53 UTC
     2015 x86_64 x86_64 x86_64 GNU/Linux                                           
    
    $ python3 -V
    Python 3.4.2
    
    $ dnf info python3-coverage
    Last metadata expiration check performed 1:54:12 ago on Mon Jul 13 11:44:10 2015.
    Installed Packages
    Name        : python3-coverage
    Arch        : x86_64
    Epoch       : 0
    Version     : 4.0
    Release     : 0.6.a5.fc22
    Size        : 708 k
    Repo        : @System
    From repo   : fedora
    Summary     : Code coverage testing module for Python 3
    URL         : http://nedbatchelder.com/code/modules/coverage.html
    License     : BSD and (MIT or GPLv2)
    Description : Coverage.py is a Python 3 module that measures code coverage
                : during Python execution. It uses the code analysis tools and
                : tracing hooks provided in the Python standard library to determine
                : which lines are executable, and which have been executed.
    
    $ coverage3 help
    Coverage.py, version 4.0a5
    Measure, collect, and report on code coverage in Python programs.
    
    usage: coverage <command> [options] [args]
    
    Commands:
        annotate    Annotate source files with execution information.
        combine     Combine a number of data files.
        erase       Erase previously collected coverage data.
        help        Get help on using coverage.py.
        html        Create an HTML report.
        report      Report coverage stats on modules.
        run         Run a Python program and measure code execution.
        xml         Create an XML report of coverage results.
    
    Use "coverage help <command>" for detailed help on any command.
    For full documentation, see https://coverage.readthedocs.org/en/coverage-4.0a5
    
    $ cat tmp.py
    print ('Hello World!')
    
    $ coverage3 run tmp.py
    Hello World!
    
    $ coverage3 report
    Name     Stmts   Miss  Cover
    ----------------------------
    tmp.py       1      0   100%
    
    $ coverage3 annotate
    Traceback (most recent call last):
      File "/usr/bin/coverage3", line 9, in <module>
        load_entry_point('coverage==4.0a5', 'console_scripts', 'coverage')()
      File "/usr/lib64/python3.4/site-packages/coverage/cmdline.py", line 676, in ma
    in                                                                             
        status = CoverageScript().command_line(argv)
      File "/usr/lib64/python3.4/site-packages/coverage/cmdline.py", line 451, in co
    mmand_line                                                                     
        directory=options.directory, **report_args)
      File "/usr/lib64/python3.4/site-packages/coverage/control.py", line 924, in an
    notate                                                                         
        reporter.report(morfs, directory=directory)
      File "/usr/lib64/python3.4/site-packages/coverage/annotate.py", line 42, in re
    port                                                                           
        self.report_files(self.annotate_file, morfs, directory)
      File "/usr/lib64/python3.4/site-packages/coverage/report.py", line 80, in repo
    rt_files                                                                       
        report_fn(cu, self.coverage._analyze(cu))
      File "/usr/lib64/python3.4/site-packages/coverage/annotate.py", line 92, in an
    notate_file                                                                    
        dest.write(line.encode('utf-8'))
    TypeError: must be str, not bytes
    
    $ 
    
  5. Vjacheslav Fyodorov

    Bitbucket code differs from my (which is from Fedora 22). Sorry. It should be updated in Fedora 22 distribution.

  6. Log in to comment