Issue #170 new

Show who tests what

andrea_crotti
created an issue

I was just using the awesome HTML report to see my test coverage and I had the following thought.

Wouldn't it be nice to be able to see easily what parts of the test suites are actually testing my code?

I guess that these information is collected while doing the annotation, right?

In this way we could actually see if the tests are actually good very easily, which is specially important when working with other people code.

Comments (15)

  1. Ned Batchelder repo owner

    This is a very interesting idea, one that figleaf pioneered with "sections". Right now we don't collect this information. The trace function would have to be modified to walk up the stack to identify "the test", then that information would have to be stored somehow. Then the reporting would have to be changed to somehow display the information.

    That's three significant problems, but only three! Do you have ideas how to do them?

  2. andrea_crotti reporter
    • changed status to open

    Well I need to dive more into the internals to suggest something that makes sense, however I see that the stats struct is this:

    struct { unsigned int calls; unsigned int lines; unsigned int returns; unsigned int exceptions; unsigned int others; unsigned int new_files; unsigned int missed_returns; unsigned int stack_reallocs; unsigned int errors; } stats;

    which probably doesn't help much my idea, because I think we would need to associate every *line* to a list of lines that are testing it.

    So for example

    silly_module.py def silly_func(): foobar()

    silly_test.py: assert silly_func()

    silly_test2.py: assert silly_func()

    I should have that

    silly_func:0 = [silly_test.py:0, silly_test2:0] silly_func:1 = [silly_test.py:0, silly_test2:0]

    I'm afraid that it would be a awful lot of information to store if the project gets really big though.

    For the reporting I imagine just to add a clickable button near every line that opens up a page that collects the different tests that run that line, with some context around.

    That should probably be the easier part, even if I'm not really a good web-developer at the moment..

  3. Ned Batchelder repo owner

    I don't think we need to collect all the *lines* that test product lines, we need to collect the *tests* that test product lines, which reduces the data collection a bit, but it will still be a challenge.

  4. andrea_crotti reporter

    For the tests you mean the code object of the test function?

    In that case I agree, because it should keep track of original file / line where it's defined if I remember correctly.

    Anyway another possible usecase which of this feature is checking if unit-tests are really unit tests.

    If I see for example that module a.py is tested by test_a.py but also from test_z.py which has almost nothing to do with a.py, then there is something wrong, and a system to visualize it would be nice..

  5. Thomas Güttler

    I guess you need this data structure to implement this:

    I use django ORM since it is what I know best, but SQLAlchemy might be a better solution for this. Storage to sqlite would be enough.

    class Line(models.Model):
        file_name=models.CharField()
        line_number=models.IntegerField()
    
    class StackFrame(models.Model):
        executed_line=models.ForeignKey(Line)
        lines_of_stack=models.ManyToManyField(Line)
    

    This structure needs to be filled with every line that gets executed by coverage.

    A HTML Report could be created by this data.

    I guess this is really slow... but who cares? For me it would be enough to run this once very week in a cron/batch job.

  6. Thomas Güttler

    About the ORM: Linus Torvald said sometimes: good programmers care about data structures. That's why I would implement this first. And the db structure in this case is quite easy. The implementation could use pure sqlite without ORM.

    Yes, the execution time would increase a lot. But I don't think an alternative to sqlite would be much faster.

    And: This is not intended to be run every time. We can optimize later.

  7. Log in to comment