Non-deterministic test failure

Issue #82 resolved
Harlan Lieberman-Berg created an issue

The test in passlib/tests/test_utils.py, lines 125-129, improperly rely on random information always matching the test assertions. Approximately 0.5% of the time, the test will fail due to the randomly generated set not containing one of the three possibilities.

A detailed analysis is available at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844233. Many thanks to the Debian Reproducible Build Team for discovering this.

Comments (4)

  1. Eli Collins repo owner
    • changed status to open

    Sigh. It's a dirty testing habit I'm a bit ashamed of, trying to fix them where I find them.

    That function in particular does NOT need actual randomness, I'll rework it to run over a bunch of fixed seeds.

    Release of passlib 1.7 is imminent, fix will be included there.

    Thanks for reporting this!

  2. Eli Collins repo owner

    bugfix: passlib.tests: fix non-deterministic getrandbytes() test

    To resolve issue #82, getrandbytes() UT reworked to use fixed set
    of random seed values. This removes chance of errant false positives,
    but makes sure it still gets checked across a few different "random" inputs.

    → <<cset 1ad782b09224>>

  3. Eli Collins repo owner

    Passlib 1.7 has been released, which should include a fix for this. Please let me know if any other non-determinism shows up.

    Subsequent to that last commit, I made an audit to see if I could find any others; and did a refactor (rev 8b31419) tying all uses of an RNG in the test suite to use the $PYTHONHASHSEED value, which should help make any remaining non-determinism at least be reproducible :)

  4. Log in to comment