Change Test Instance Status to Test List Instance Status level

Issue #524 open
Randle Taylor created an issue

I don’t think I’ve encountered anyone who sets test instance statuses differently within a given test list instance during manual review (it can obviously happen due to autoreview). In many ways moving the “Status” of QC data to the TestListInstance makes more sense than having it on individual test instance basis. People typically want to review the entire test list instance at the same time.

This would be a major breaking change but I think it should be considered seriously. Some points:

  • Moving Status to TestListInstance level would likely simplify some code
  • Some queries would become more efficient
  • The TLI.all_reviewed column could be eliminated.
  • AutoReview rules would move from the Test level to the TestList or UnitTestCollection level. The latter would allow setting different AutoReview rules for the same Test List on different units.
  • Migration edge cases might be tricky:

    • For TLI’s where all TI's have the same status, setting the new TLI Status would be easy but if a TLI has TI’s with different statuses you would need to decide which status to apply to the TLI (for example if some TI’s still required review, the TLI should also require review).
    • Migrating the AutoReviewRules would need to happen at the same time and would have similar issues; if not all tests within a test list have the same ARR’s set, how do you decide what the AutoReviewRule should be? Could possibly just declare bankruptcy on that front and have people reconfigure ARR completely.
  • Bulk review would be simpler

  • More to come as I think of them…

Comments (9)

  1. Marc Chamberland

    At first glance and without having thought about it too much, I think I’m on board with this.

  2. Randle Taylor reporter

    Thanks for chiming in @Marc Chamberland I was going to ask your opinion but you beat me to the punch!

  3. Matt B

    Sounds like a good idea in principle and we definitely mark things as “done” by setting the entire TL to approved. All items flag up to us for review once done as we purposely include a single test which will not auto approve. It would be nice if there was a flag on a TL that would always require review even when everything within set tolerances. (Some things we like to look at even if all in tolerance for various reasons).

  4. Randle Taylor reporter

    @Matt B Can you clarify why you use AutoReview at all if you do this “All items flag up to us for review once done as we purposely include a single test which will not auto approve.”

    “It would be nice if there was a flag on a TL that would always require review even when everything within set tolerances.” If you don’t set AutoReview rules on a Test/TestList then it will always show up in the unreviewed queue. Maybe I’m misunderstanding what you mean though.

  5. Matt B

    I think its because the autoreview rules are at the test level, whereas we have some testlists which we are happy to be auto reviewed but some always need a second check and easiest way for us to do this was to have a test we can add to these which forced it to flag up for review. Testlists which don't need a second check then do not have this test and so autoreview if no issues with results.

    I think originally we didn't have the test which would not autoreview but then it was decided that some results always needed reviewing so needed to add a way of flagging these up.

    It may be that there is a nicer way of doing this…?

  6. Randle Taylor reporter

    @Matt B Can’t you just disable Auto Review on one or all of tests in the test lists that you always want to review manually?

  7. Matt B

    Sorry, I think we already do what you are suggesting. only the tests which we do not want to auto review are set that way. But we are using them to force a testlist to flagup for review, not the individual result of that test. I'm all for moving autoreview from the test to the testlist level 🙂

  8. Log in to comment