Only first line of multi-line statements reported by coverage xml

Issue #571 closed
Former user created an issue

This is a issue first submit in codecov:

but I find that this sonar-python plugins sonarqube can not color the proper lines when there are multi-line statements.

It seems the coverage parser only show the first line of multiline coverage info, so many third-party software can not recognize the multiline statements, cause the mistake color info

Can we add a option to fix this problem?

Comments (16)

  1. Ned Batchelder repo owner

    This seems to me to be a bug in the third-party software. The XML file does not say there is a missed line. Does it also color docstrings as missed?

  2. Ijaz Sarwar

    The problem in this case is that coverage.xml only reports that line 18 is covered and nothing else. If you do coverage html even then line 19 to 23 would be shown as Not Executable.

    I have the exact same problem and it is not related to 3rd party: UncoveredLinesIssue.png

    Any suggestions here?

  3. Ned Batchelder repo owner

    @isarwar why is it a problem that the lines are marked as not executable? They don't subtract from your coverage measurement, and they aren't indicated as something you need to write tests for.

  4. Ijaz Sarwar

    thanks for the fast response! In my case coverage.xml does not say anything about line 58 to 71. All it says is line 57 is covered. When I annotate line 57 to 71 comes up with !(meaning not covered). So if I have my own program (in PHP) that takes these two as input then I would declare 1 line covered out of 15 (which is incorrect).

    Also I don't understand why coverage.xml does not have information about all the lines?

  5. Ned Batchelder repo owner

    @isarwar Sorry, I can't reproduce what you are describing. Can you provide a reproducible test case? Annotate won't make a line as not executed if the other reports don't mark it as not executed.

  6. Ijaz Sarwar

    ahh my bad. I could not reproduce that exact thing. But I am still seeing the difference between what annotation is reporting and what xml is reporting.

    1) Multi line comments are marked as executed. why? 2) Line 57 to 71 is all executed in annotation (which is fine) but in xml only line 57 is mentioned and rest are missing.


  7. Ned Batchelder repo owner

    The annotate algorithm does better on lines 58-71, but as you note, it also shows false positives on lines 89, and 103-112.

    What is the real problem here? Do you use "annotate"? If I had to choose between the way html works and the way annotate works, I'd choose html. I can make annotate the same as html. I'm not sure there's a way to get it to be perfect (that is, mark lines 58-71, but not 103-112.

  8. Eric Kever

    I believe the primary problem here is that if I were to feed the XML into a tool such as SonarQube, it would yield incorrect coverage results because the XML is only showing the first line of a multi-line statement as being covered, when in reality, all of those lines should be considered covered.

  9. Ned Batchelder repo owner

    @suddenlychloe Can you elaborate on the "incorrect result" that SonaqQube would display?

  10. Eric Kever

    Given a 10 line Python script with no empty lines, completely uncovered except for 1 multi-line statement that was 5 lines long, the XML would only state that the first line of it was covered. In reality all 5 lines were covered because they were part of the same statement. Feeding the XML into SonarQube, however, would yield that I have 10% coverage instead of 50%, because the XML only states that the first line of that multi-line statement is covered.

  11. Ned Batchelder repo owner

    @suddenlychloe I see what you mean. I'm not that interested in the difference between a 1-line statement, and a 5-line statement. really measures statement coverage. I'll take a look at the possibilities for counting all 5 of the lines, but I'm not personally unhappy with the way it is now.

  12. Michael Dougherty

    Hi Ned & Eric - I've been following this ticket for a little while now, glad to see there's some discussion on it :)

    It sounds like the issue is that the coverage XML format is line-based, and downstream consumers of it treat it as such. But of course python is statement-based, and a statement can be multiple lines OR you can even have multiple statements per line. In this case the reporting format loses some of fidelity of the underlying data, but IMO that's a decision users have to make when they opt to use the XML format. However, I think I agree with Eric in that I would want my report-consumer (we also use sonarqube!) to correctly show that the entire multi-line statement is covered. The 10-line script with 5-line statement I think is a good example -- I would expect all of the statement (= all of the lines) to be shown as "covered" in whatever view I might have. It doesn't seem to make sense to show the start of a dictionary ({) on one line as covered and to show that the individual entries ('foo': 'bar') as not covered.

    Hope that helps, at least to add another perspective.

  13. Ned Batchelder repo owner

    I agree that in an ideal world, all 5 lines would be marked as executed. But looked at another way, there was really only one choice: to run all five lines, or to run none of them. There weren't 5 opportunities to execute code, there was only one.

    But really, the limiting factor here is the information that Python provides to us. I'd have to research what information it's giving me, but I'm guessing it only reports on the first line of the statement.

  14. Log in to comment