Issues

Issue #161 new

Setting PYTHONHASHSEED can crash multiprocessing tests on Windows

Marc Schlaich
created an issue

See http://bugs.python.org/issue20954. I'm not sure how (and if) this can be handled gracefully in tox.

My first idea was setting PYTHONHASHSEED=random per default (I already prepared a PR for that). But then I noticed that this will change the current behavior completely: with PYTHONHASHSEED=random every env will get its own hash seed and you won't get a useful output on run (because there will be PYTHONHASHSEED='random' in the log. So you won't be able to fully reproduce test runs which was the main intention of pull request #79 I guess.

A less intrusive option would be (on Windows): if a test run fails, scan the output for a MemoryError in subprocess. If it matches point to this bug and give the suggestion to run tox with --hashseed random. But I don't think how you feel in general about handling upstream bugs (please note, this Python bug will never be fixed in 2.6 and 3.3).

cc Chris Jerdonek

Comments (4)

  1. holger krekel repo owner

    Thanks Marc for the good report. So if tox sets PYTHONHASHSEED to an int, tests involving multiprocessing will fail on py27, py33 currently and probably in the future, right? Parsing "MemoryError" etc. seems not desirable to me. I'd rather change the default to "random" so people need to read up in the docs on how to change it (or look at "-h" output etc.). At least through the logged value of "random" for PYTHONHASHSEED they get a clue that tox does/can do something with it. What do you think Chris Jerdonek ?

  2. Marc Schlaich reporter

    and probably in the future, right?

    Yes, at least on py33 because it is on security maintenance only. I guess py27 (and py34) will be fixed in the future.

    I'd rather change the default to "random"

    OK, I send a PR for this change.

  3. Chris Jerdonek

    Thanks for the CC, Marc Schlaich and holger krekel . I'm not sure I understand the underlying Python bug precisely. But hmm, it would be a shame to change the behavior on all systems going forward because of a bug affecting only Windows and when the test suites use subprocess. At the least, could the default be changed to random only for the affected systems?

    Also, are we sure there isn't a work-around to pass an explicit value for the affected systems? Like I said, I don't understand the scope of the bug exactly. But for example, one of the original reports says, "sys.flags.hash_randomization has a value of 2147483647 and it tries to build a string with such a length." What if the value was restricted to something small for the affected versions (e.g. something less than 20, say)? Would it work in that case, then? Or maybe it could randomly choose between random and something less than 20 to guarantee that the full range of seed values is possible.

  4. Marc Schlaich reporter

    I'm not sure I understand the underlying Python bug precisely.

    Basically, multiprocessing is creating the string '-R' * int(PYTHONHASHSEED) on starting a subprocess.

    What if the value was restricted to something small for the affected versions

    Yes, that's actually the right way to fix this. Don't know why I didn't see this before =) PR is following.

  5. Log in to comment