testFileRead fails using python 3.x

Create issue
Issue #3 resolved
Hans-Christoph Steiner created an issue

I'm in the process of packaging biplist for Debian. Building it for upload, I run into this problem with the tests:

FAIL: testFileRead (test_valid.TestValidPlistFile)
Traceback (most recent call last):
  File "/tmp/buildd/python-biplist-0.6/tests/test_valid.py", line 26, in testFileRead
  File "/tmp/buildd/python-biplist-0.6/tests/test_valid.py", line 15, in validateSimpleBinaryRoot
    self.assertEquals(root[six.b('dateItem')], datetime.datetime(2010, 8, 19, 22, 27, 30, 385449), "dates not equal" )
AssertionError: datetime.datetime(2010, 8, 19, 22, 27, 30, 385448) != datetime.datetime(2010, 8, 19, 22, 27, 30, 385449) : dates not equal

Here are the relevant package versions:

Setting up libpython3.3-minimal:amd64 (3.3.2-7) ...
Setting up libpython3.3-stdlib:amd64 (3.3.2-7) ...
Setting up libunistring0:amd64 (0.9.3-5) ...
Setting up libpython2.7-minimal:amd64 (2.7.5-8) ...
Setting up python2.7-minimal (2.7.5-8) ...
Setting up python3.3-minimal (3.3.2-7) ...
Setting up libpython2.7-stdlib:amd64 (2.7.5-8) ...
Setting up python2.7 (2.7.5-8) ...
Setting up python-minimal (2.7.5-5) ...
Setting up libpython-stdlib:amd64 (2.7.5-5) ...
Setting up python (2.7.5-5) ...
Setting up python3.3 (3.3.2-7) ...
Setting up python3-minimal (3.3.2-17) ...
Setting up libpython3-stdlib:amd64 (3.3.2-17) ...
Setting up python-all (2.7.5-5) ...
Setting up python-pkg-resources (0.6.49-2) ...
Setting up python-coverage (3.7+dfsg.1-2) ...
Setting up python-nose (1.3.0-2) ...
Setting up python-setuptools (0.6.49-2) ...
Setting up python-six (1.4.1-1) ...
Setting up python3 (3.3.2-17) ...
Setting up python3-all (3.3.2-17) ...
Setting up python3-pkg-resources (0.6.49-2) ...
Setting up python3-coverage (3.7+dfsg.1-2) ...
Setting up python3-nose (1.3.0-2) ...
Setting up python3-setuptools (0.6.49-2) ...
Setting up python3-six (1.4.1-1) ...

Attached is the full build log.

Comments (11)

  1. Andrew Wooster repo owner

    Looks like a rounding mismatch in the fractional seconds. I'm not sure why it's happening though, and can't reproduce on my machines.

  2. Hans-Christoph Steiner reporter

    I'd like to run the test suite as part of the packaging, what do you recommend? Maybe make that test optional or less stringent?

  3. Andrew Wooster repo owner

    Optimally I'd like to fix it. I'll be doing some work with Debian this weekend on EC2, and will check to see if I can repro this. If not, I'll probably need more detailed configuration information (kernel version, Debian distro, hardware info, etc). Also, is this repeatable? (eg does it break every time)

  4. Andrew Wooster repo owner

    Just tested this with Debian precise64 on Intel hardware and wasn't able to reproduce. I'll take a patch, but I don't have amd64 hardware to test this on and not a lot of time to look into it.

    I wouldn't expect Intel and AMD hardware to behave differently on the actual floating point numbers, so it's possible there's a behavior difference not in my code.

    The code failure in question reduces to:

    import binascii
    import datetime
    import struct
    data = binascii.unhexlify('41b21de75262acc9')
    unpacked_date = struct.unpack(">d", data)[0]
    date = datetime.datetime.utcfromtimestamp(unpacked_date + 978307200)
    date == datetime.datetime(2010, 8, 19, 22, 27, 30, 385449)

    It would be helpful to know at which point in the above things differ from other platforms.

  5. Andrew Wooster repo owner

    Actually, I remembered another machine I have access to…

    Tested on Gentoo Linux/amd64 AMD Opteron 6276, and got expected output both for the above and for the unit tests.

    Given that, this seems like a software difference somewhere else in the stack.

  6. Hans-Christoph Steiner reporter
    • edited description

    Did you try running the tests with Python 3.x? The tests work fine with Python 2.7 for me. I tried it on two computers (FYI: amd64 is the same thing as x86_64, it is the generic architecture name for AMD and Intel 64-bit CPUs):

    All tests pass on:

    • Debian/unstable/amd64/Intel Core2 with Python 2.7.8 and six 1.7.3
    • Debian/jessie/amd64/Intel server with Python 2.7.8 and six 1.7.3

    The tests fail with the same error on:

    • Debian/unstable/amd64/Intel Core2 with Python 3.4.1 and six 1.7.3
    • Debian/jessie/amd64/Intel server with Python 3.4.1 and six 1.7.3

    Also, running the biplist 0.7 tests on Debian/jessie/amd64/Intel server fails with a syntax error, so my guess is that the tests are not usually run using python3:

    $ python3.4 /home/pd/code/wooster/biplist/tests/test_write.py
      File "/home/pd/code/wooster/biplist/tests/test_write.py", line 185
        self.roundTrip([0x10000000000000000L, pow(2, 64)])
    SyntaxError: invalid syntax

    I've been running the tests under python3 using:

    python3 setup.py test -vvv
  7. Andrew Wooster repo owner

    I added some fixes in master for running tests on Python 3:


    I was only able to test on Python 3.2, at the moment.

    In the tests, as well as running the following, I didn't get the error you're seeing:

    import binascii
    import datetime
    import struct
    data = binascii.unhexlify(bytes("41b21de75262acc9", "utf-8"))
    unpacked_date = struct.unpack(">d", data)[0]
    date = datetime.datetime.utcfromtimestamp(unpacked_date + 978307200)
    date == datetime.datetime(2010, 8, 19, 22, 27, 30, 385449)
  8. Hans-Christoph Steiner reporter

    I ran the test snippet on Ubuntu/precise/64-bit and it works on 3.2 but not 2.7. It also gave the same error on Mac OS X Mavericks using python2.7:

    $ python3.2 /tmp/test.py 
    $ python2.7 /tmp/test.py 
    Traceback (most recent call last):
      File "/tmp/test.py", line 5, in <module>
        data = binascii.unhexlify(bytes("41b21de75262acc9", "utf-8"))
    TypeError: str() takes at most 1 argument (2 given)

    The tests seem to have gotten a lot further, and it looks like you fixed the syntax error in that same commit. But now I'm getting this odd error:

    hans@palatschinken biplist $ python3.2 setup.py test 
    running test
    running egg_info
    writing requirements to biplist.egg-info/requires.txt
    writing biplist.egg-info/PKG-INFO
    writing top-level names to biplist.egg-info/top_level.txt
    writing dependency_links to biplist.egg-info/dependency_links.txt
    reading manifest file 'biplist.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'biplist.egg-info/SOURCES.txt'
    running build_ext
    testEmptyFile (test_invalid.TestInvalidPlistFile) ... ok
    testInvalid (test_invalid.TestInvalidPlistFile) ... ok
    testTooShort (test_invalid.TestInvalidPlistFile) ... ok
    testEmptyUnicodeRoot (test_valid.TestValidPlistFile) ... ok
    testFileRead (test_valid.TestValidPlistFile) ... ok
    testKeyedArchiverPlist (test_valid.TestValidPlistFile)
    Archive is created with class like this: ... ok
    testLargeIntegers (test_valid.TestValidPlistFile) ... ok
    testSmallReal (test_valid.TestValidPlistFile) ... ok
    testUnicodeRoot (test_valid.TestValidPlistFile) ... ok
    testBadKeys (test_write.TestWritePlist) ... ok
    testBoolRoot (test_write.TestWritePlist) ... /code/wooster/biplist/tests/test_write.py:31:
      ResourceWarning: unclosed file <_io.FileIO name=6 mode='rb'>
      (status, output) = run_command(['/usr/bin/plutil', '-lint', name])
    testBools (test_write.TestWritePlist) ... ok
    testBoolsAndIntegersMixed (test_write.TestWritePlist) ... ok
    testComplicated (test_write.TestWritePlist) ... ok
    testConvertToXMLPlistWithData (test_write.TestWritePlist) ... ok
    testDatetime (test_write.TestWritePlist) ... ok
    testDictRoot (test_write.TestWritePlist) ... ok
    testDuplicate (test_write.TestWritePlist) ... ok
    testFloat (test_write.TestWritePlist) ... ok
    testFloatsAndIntegersMixed (test_write.TestWritePlist) ... ok
    testIntBoundaries (test_write.TestWritePlist) ... ok
    testLargeDict (test_write.TestWritePlist) ... ok
    testListRoot (test_write.TestWritePlist) ... ok
    testNone (test_write.TestWritePlist) ... ok
    testSetRoot (test_write.TestWritePlist) ... ok
    testString (test_write.TestWritePlist) ... ok
    testTuple (test_write.TestWritePlist) ... ok
    testUidWrite (test_write.TestWritePlist) ... ok
    testUnicode (test_write.TestWritePlist) ... ok
    testUniques (test_write.TestWritePlist) ... ok
    testWriteData (test_write.TestWritePlist) ... ok
    testWriteToFile (test_write.TestWritePlist) ... /code/wooster/biplist/tests/test_write.py:31:
      ResourceWarning: unclosed file <_io.FileIO name=7 mode='rb'>
      (status, output) = run_command(['/usr/bin/plutil', '-lint', name])
    testXMLPlist (test_write.TestWritePlist) ... ok
    testXMLPlistWithData (test_write.TestWritePlist) ... ok
    Ran 34 tests in 0.223s
    Error in atexit._run_exitfuncs:
    TypeError: 'NoneType' object is not callable
  9. Andrew Wooster repo owner

    Yeah, the bytes() bit doesn't exist in 2.7. You can remove that call and it should work on 2.7, per the snippet further up the comment thread.

    The ResourceWarning is a super-annoying Python 3.2 change. Fixed in master.

    Meanwhile, I was finally able to reproduce with the Homebrew version of Python 3.4 on OS X. Python 2.7 handles the microseconds part of dates by rounding, with C like so:

    fraction = timestamp - (double)timet;
    us = (int)round_to_long(fraction * 1e6);

    whereas Python 3.x after rev 1e9cc1a03365 has rewritten this (in Python), as:

    t, frac = divmod(unpacked_date, 1.0)
    us = int(frac * 1e6)

    The obvious bit being that under Python 2.7.x there was a call to round, and they've decided to move away from that.

    In any case, this and several other Python 3.4 compat fixes are now in master. If you could check them out and see if they work, that'd be great.

    Thanks for your patience!

  10. Log in to comment