json parse error

After upgrading to 4.0, we started getting the following stacktrace on all of our builds:

Traceback (most recent call last):
  File "/testing/tests/runtests.py", line 476, in <module>
    main()
  File "/testing/tests/runtests.py", line 469, in main
    parser.finalize(0)
  File "/usr/local/lib/python2.7/dist-packages/salttesting/parser/cover.py", line 233, in finalize
    self.stop_coverage(save_coverage=True)
  File "/usr/local/lib/python2.7/dist-packages/salttesting/parser/cover.py", line 204, in stop_coverage
    self.code_coverage.combine()
  File "/usr/local/lib/python2.7/dist-packages/coverage/control.py", line 787, in combine
    self.data_files.combine_parallel_data(self.data, aliases=aliases, data_paths=data_paths)
  File "/usr/local/lib/python2.7/dist-packages/coverage/data.py", line 699, in combine_parallel_data
    new_data.read_file(f)
  File "/usr/local/lib/python2.7/dist-packages/coverage/data.py", line 293, in read_file
    filename, exc.__class__.__name__, exc,
CoverageException: Couldn't read data from '/testing/.coverage.li1177-156.14191.086480': ValueError: Expecting object: line 1 column 16164 (char 16163)

You can find the referenced salttesting code on github.

Downgrading to 3.7.1 fixes this problem for us. I am going to spend some time investigating further what the problem is and will report back what I find. Thanks.

@jfindlay Can you provide one of these files that isn't parsable? Also, are you running in a clean test environment? Is it possible you're combining data files from 3.7.1 with 4.0?

The environment is constructed from a cloud image from Linode or Rackspace using salt states, and in fact coverage is installed as part of that process, so I don't think that there would be any data from 3.7.1. It may be a few days before I have some more info. We have downgraded for now so we can get some releases out and then I'll be back on this.

It would be consistent with other coverage behavior if we convert these hard errors to warnings.

Eg: if there's a warning when there's nothing to save why wouldn't there be a warning when data can't be loaded?

Plus, due to external factors (things that can't be controlled) processes can fail to save the data: someone (like a time limiter) sends SIGKILL, linux OOM killer sends SIGKILL etc

@jfindlay if you can provide me with steps to reproduce, that would be great.

Commit aa0afc855566 adds "combine -i"

BTW, not sure why, but coverage.py 3.7.1 had "except Exception: pass" around the reading of the data files... :( It's possible that under 3.7.1, you also have invalid data files, but they wouldn't stop the processing.

I've changed the behavior: Commit ebfc51a026f4 makes warnings on unreadable files unconditional. I don't know why your data files are unreadable, but this makes coverage.py work with them as it did under 3.7.1, with the addition that it lets you know it is skipping those files.