TokenError when generating html report

Chris AtLee avatarChris AtLee created an issue

I'm trying to run coverage on the buildbot source like this:

coverage run --branch =trial buildbot.test
coverage html -d cover -i 

And get the following exception:

Traceback (most recent call last):
  File "/home/catlee/.virtualenvs/buildbot.git/bin/coverage", line 9, in <module>
    load_entry_point('coverage==3.4a1', 'console_scripts', 'coverage')()
  File "/home/catlee/src/coveragepy/coverage/cmdline.py", line 639, in main
    status = CoverageScript().command_line(argv)
  File "/home/catlee/src/coveragepy/coverage/cmdline.py", line 531, in command_line
    directory=options.directory, **report_args)
  File "/home/catlee/src/coveragepy/coverage/control.py", line 522, in html_report
    include=self.config.include
  File "/home/catlee/src/coveragepy/coverage/html.py", line 47, in report
    self.report_files(self.html_file, morfs, directory, omit, include)
  File "/home/catlee/src/coveragepy/coverage/report.py", line 63, in report_files
    report_fn(cu, self.coverage._analyze(cu))
  File "/home/catlee/src/coveragepy/coverage/control.py", line 458, in _analyze
    return Analysis(self, it)
  File "/home/catlee/src/coveragepy/coverage/results.py", line 30, in __init__
    self.statements, self.excluded = self.parser.parse_source()
  File "/home/catlee/src/coveragepy/coverage/parser.py", line 188, in parse_source
    self._raw_parse()
  File "/home/catlee/src/coveragepy/coverage/parser.py", line 91, in _raw_parse
    for toktype, ttext, (slineno, _), (elineno, _), ltext in tokgen:
  File "/usr/lib/python2.5/tokenize.py", line 292, in generate_tokens
    raise TokenError, ("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (36, 0))

I think this is because there's one of the .html files in the coverage report for some reason. I can attach the .coverage file if that helps.

This patch seems to fix the problem:

diff --git a/coverage/parser.py b/coverage/parser.py
--- a/coverage/parser.py
+++ b/coverage/parser.py
@@ -179,17 +179,20 @@ class CodeParser(object):
 
         Return values are 1) a sorted list of executable line numbers, and
         2) a sorted list of excluded line numbers.
 
         Reported line numbers are normalized to the first line of multi-line
         statements.
 
         """
-        self._raw_parse()
+        try:
+            self._raw_parse()
+        except:
+            raise NoSource("Couldn't parse file: %s" % self.filename)
 
         excluded_lines = self.first_lines(self.excluded)
         ignore = excluded_lines + list(self.docstrings)
         lines = self.first_lines(self.statement_starts, ignore)
 
         return lines, excluded_lines
 
     def arcs(self):

Comments (21)

  1. Chris AtLee

    buildbot uses Jinja as its web template engine. I wonder if Jinja is faking out the file/line numbers in the generated python files during execution?

  2. Ned Batchelder

    Yes, "coverage debug data" shows this entry:

    /home/catlee/src/buildbot/master/buildbot/status/web/templates/revmacros.html: 51 lines

    Any chance you could make me a small .tar with a reproducible case?

  3. Chris AtLee

    I tarred up my virtual env...and it's 31MB. I can put that up somewhere, otherwise, here are the steps to reproduce.

    virtualenv --no-site-packages coveragetest
    cd coveragetest
    source bin/activate
    git clone http://github.com/djmitche/buildbot.git
    cd buildbot/master
    python setup.py develop
    pip install mock
    pip install coverage
    
    coverage run --branch $VIRTUAL_ENV/bin/trial buildbot.test
    coverage html -d cover -i
    
  4. Nathan Yergler

    We're experiencing this, as well, when attempting to generate the XML report. I've attached a small test case that you can use by:

    $ virtualenv --no-site-packages jinjatest
    $ cd jinjatest
    $ source bin/activate
    $ ./bin/easy_install Jinja2
    $ ./bin/easy_install coverage
    $ tar zxvf template_pkg.tgz
    $ ./bin/coverage run ./template_pkg/__init__.py
    $ ./bin/coverage xml
    

    Ka-boom!

    I initially thought this was due to our use of PackageLoader, which loads templates from within packages, but rewrote my simple example to use the FileSystemLoader.

  5. Alexandre Conrad
    • changed status to open

    I have the same problem. The following change worked for me:

    if filename.endswith(".html") or filename.endswith(".jinja2"):
    

    as my templates have a .jinja2 extention name.

    Would there be a way to exclude patterns from the command line? Or maybe just skip files that don't compile and report it at the end of the XML conversion? (don't know if that's acceptable though).

  6. Anonymous

    I'm experiencing this problem with a Jinja2 template that ends in ".js.inc" I believe this is a valid use case. The Jinja template documentation at http://jinja.pocoo.org/docs/templates/ says: "A template is simply a text file. It can generate any text-based format (HTML, XML, CSV, LaTeX, etc.). It doesn’t have a specific extension, .html or .xml are just fine."

    As a workaround I can either include=["*.py"] or omit=["*.js.inc"]

    But it seems worthwhile to investigate a fix in coverage.py. Some thoughts: - The PyTrace._trace() function gets a frame object. Is there anything in that object that indicates that the file isn't an actual python file? Maybe something in frame.f_code? - It seems like the report generation is only able to generate annotated source code for .py source. Would it make sense to ignore files with other extensions? Or ignore any exceptions that happen while attempting to exec the code?

  7. Anonymous

    (Sorry, last comment was from me. I guess I don't have a bitbucket account. My name is Mark Doliner. Email address is mark@kingant.net, if there's anything I can help with.)

  8. Ned Batchelder

    This seems to still be happening with Jinja:

    pelican)...nt/pelican% ~/.virtualenvs/pelican/bin/coverage run `which unit2` discover                                                                                                                                             Julian@air
    sh: pandoc: command not found
    ........No handlers could be found for logger "pelican.contents"
    ................s.sss.................
    ----------------------------------------------------------------------
    Ran 46 tests in 1.580s
    
    OK (skipped=4)
    (pelican)...nt/pelican% ~/.virtualenvs/pelican/bin/coverage html                                                                                                                                                                   Julian@air
    Traceback (most recent call last):
      File "/Users/Julian/.virtualenvs/pelican/bin/coverage", line 9, in <module>
        load_entry_point('coverage==3.5.2', 'console_scripts', 'coverage')()
      File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/cmdline.py", line 657, in main
        status = CoverageScript().command_line(argv)
      File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/cmdline.py", line 549, in command_line
        directory=options.directory, **report_args)
      File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/control.py", line 603, in html_report
        reporter.report(morfs)
      File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/html.py", line 87, in report
        self.report_files(self.html_file, morfs, self.config.html_dir)
      File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/report.py", line 83, in report_files
        report_fn(cu, self.coverage._analyze(cu))
      File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/control.py", line 543, in _analyze
        return Analysis(self, it)
      File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/results.py", line 30, in __init__
        self.statements, self.excluded = self.parser.parse_source()
      File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/parser.py", line 202, in parse_source
        self._raw_parse()
      File "/Users/Julian/.virtualenvs/pelican/lib/python2.7/site-packages/coverage/parser.py", line 106, in _raw_parse
        for toktype, ttext, (slineno, _), (elineno, _), ltext in tokgen:
      File "/usr/local/Cellar/python/2.7.3/lib/python2.7/tokenize.py", line 357, in generate_tokens
        raise TokenError, ("EOF in multi-line statement", (lnum, 0))
    tokenize.TokenError: ('EOF in multi-line statement', (62, 0))
    
  9. Davide Setti

    I have the same issue with coverage 3.5.3 and Jinja 2.6. My templates ends with .jinja2. Do i need to give them .html extension?

    Traceback (most recent call last):
      File "manage_ve.py", line 72, in <module>
        execute_from_command_line(sys.argv)
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
        utility.execute()
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
        self.execute(*args, **options.__dict__)
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
        output = self.handle(*args, **options)
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django_jenkins/management/commands/__init__.py", line 70, in handle
        if test_runner.run_tests(test_labels):
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django_jenkins/runner.py", line 345, in run_tests
        self.teardown_test_environment()
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django_jenkins/runner.py", line 325, in teardown_test_environment
        signals.teardown_test_environment.send(sender=self)
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 172, in send
        response = receiver(signal=self, sender=sender, **named)
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/django_jenkins/tasks/with_coverage.py", line 59, in teardown_test_environment
        self.coverage.xml_report(morfs=morfs, outfile=os.path.join(self.output_dir, 'coverage.xml'))
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/control.py", line 630, in xml_report
        reporter.report(morfs, outfile=outfile)
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/xmlreport.py", line 56, in report
        self.report_files(self.xml_file, morfs)
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/report.py", line 83, in report_files
        report_fn(cu, self.coverage._analyze(cu))
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/control.py", line 543, in _analyze
        return Analysis(self, it)
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/results.py", line 30, in __init__
        self.statements, self.excluded = self.parser.parse_source()
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/parser.py", line 207, in parse_source
        self._raw_parse()
      File "/var/lib/jenkins/jobs/timu-develop-ci/workspace/.ve/local/lib/python2.7/site-packages/coverage/parser.py", line 110, in _raw_parse
        for toktype, ttext, (slineno, _), (elineno, _), ltext in tokgen:
      File "/usr/lib/python2.7/tokenize.py", line 351, in generate_tokens
        ("<tokenize>", lnum, pos, line))
      File "<tokenize>", line 12
        <span class="famfamfamflag famfamfamflags-{{ lang_code }}"></span>
        ^
    IndentationError: unindent does not match any outer indentation level
    
  10. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.