testFileRead fails using python 3.x
I'm in the process of packaging biplist for Debian. Building it for upload, I run into this problem with the tests:
======================================================================
FAIL: testFileRead (test_valid.TestValidPlistFile)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/buildd/python-biplist-0.6/tests/test_valid.py", line 26, in testFileRead
self.validateSimpleBinaryRoot(result)
File "/tmp/buildd/python-biplist-0.6/tests/test_valid.py", line 15, in validateSimpleBinaryRoot
self.assertEquals(root[six.b('dateItem')], datetime.datetime(2010, 8, 19, 22, 27, 30, 385449), "dates not equal" )
AssertionError: datetime.datetime(2010, 8, 19, 22, 27, 30, 385448) != datetime.datetime(2010, 8, 19, 22, 27, 30, 385449) : dates not equal
Here are the relevant package versions:
Setting up libpython3.3-minimal:amd64 (3.3.2-7) ...
Setting up libpython3.3-stdlib:amd64 (3.3.2-7) ...
Setting up libunistring0:amd64 (0.9.3-5) ...
Setting up libpython2.7-minimal:amd64 (2.7.5-8) ...
Setting up python2.7-minimal (2.7.5-8) ...
Setting up python3.3-minimal (3.3.2-7) ...
Setting up libpython2.7-stdlib:amd64 (2.7.5-8) ...
Setting up python2.7 (2.7.5-8) ...
Setting up python-minimal (2.7.5-5) ...
Setting up libpython-stdlib:amd64 (2.7.5-5) ...
Setting up python (2.7.5-5) ...
Setting up python3.3 (3.3.2-7) ...
Setting up python3-minimal (3.3.2-17) ...
Setting up libpython3-stdlib:amd64 (3.3.2-17) ...
Setting up python-all (2.7.5-5) ...
Setting up python-pkg-resources (0.6.49-2) ...
Setting up python-coverage (3.7+dfsg.1-2) ...
Setting up python-nose (1.3.0-2) ...
Setting up python-setuptools (0.6.49-2) ...
Setting up python-six (1.4.1-1) ...
Setting up python3 (3.3.2-17) ...
Setting up python3-all (3.3.2-17) ...
Setting up python3-pkg-resources (0.6.49-2) ...
Setting up python3-coverage (3.7+dfsg.1-2) ...
Setting up python3-nose (1.3.0-2) ...
Setting up python3-setuptools (0.6.49-2) ...
Setting up python3-six (1.4.1-1) ...
Attached is the full build log.
Comments (11)
-
repo owner -
reporter I'd like to run the test suite as part of the packaging, what do you recommend? Maybe make that test optional or less stringent?
-
repo owner Optimally I'd like to fix it. I'll be doing some work with Debian this weekend on EC2, and will check to see if I can repro this. If not, I'll probably need more detailed configuration information (kernel version, Debian distro, hardware info, etc). Also, is this repeatable? (eg does it break every time)
-
repo owner Just tested this with Debian precise64 on Intel hardware and wasn't able to reproduce. I'll take a patch, but I don't have amd64 hardware to test this on and not a lot of time to look into it.
I wouldn't expect Intel and AMD hardware to behave differently on the actual floating point numbers, so it's possible there's a behavior difference not in my code.
The code failure in question reduces to:
import binascii import datetime import struct data = binascii.unhexlify('41b21de75262acc9') unpacked_date = struct.unpack(">d", data)[0] date = datetime.datetime.utcfromtimestamp(unpacked_date + 978307200) date == datetime.datetime(2010, 8, 19, 22, 27, 30, 385449)
It would be helpful to know at which point in the above things differ from other platforms.
-
repo owner Actually, I remembered another machine I have access to…
Tested on Gentoo Linux/amd64 AMD Opteron 6276, and got expected output both for the above and for the unit tests.
Given that, this seems like a software difference somewhere else in the stack.
-
reporter - edited description
Did you try running the tests with Python 3.x? The tests work fine with Python 2.7 for me. I tried it on two computers (FYI: amd64 is the same thing as x86_64, it is the generic architecture name for AMD and Intel 64-bit CPUs):
All tests pass on:
- Debian/unstable/amd64/Intel Core2 with Python 2.7.8 and six 1.7.3
- Debian/jessie/amd64/Intel server with Python 2.7.8 and six 1.7.3
The tests fail with the same error on:
- Debian/unstable/amd64/Intel Core2 with Python 3.4.1 and six 1.7.3
- Debian/jessie/amd64/Intel server with Python 3.4.1 and six 1.7.3
Also, running the biplist 0.7 tests on Debian/jessie/amd64/Intel server fails with a syntax error, so my guess is that the tests are not usually run using python3:
$ python3.4 /home/pd/code/wooster/biplist/tests/test_write.py File "/home/pd/code/wooster/biplist/tests/test_write.py", line 185 self.roundTrip([0x10000000000000000L, pow(2, 64)]) ^ SyntaxError: invalid syntax
I've been running the tests under python3 using:
python3 setup.py test -vvv
-
reporter - changed title to testFileRead fails using python 3.x
-
repo owner I added some fixes in master for running tests on Python 3:
https://bitbucket.org/wooster/biplist/commits/4dbbd79c3563d5feb505cd52458610c7abd138a0
I was only able to test on Python 3.2, at the moment.
In the tests, as well as running the following, I didn't get the error you're seeing:
import binascii import datetime import struct data = binascii.unhexlify(bytes("41b21de75262acc9", "utf-8")) unpacked_date = struct.unpack(">d", data)[0] date = datetime.datetime.utcfromtimestamp(unpacked_date + 978307200) date == datetime.datetime(2010, 8, 19, 22, 27, 30, 385449)
-
reporter I ran the test snippet on Ubuntu/precise/64-bit and it works on 3.2 but not 2.7. It also gave the same error on Mac OS X Mavericks using python2.7:
$ python3.2 /tmp/test.py $ python2.7 /tmp/test.py Traceback (most recent call last): File "/tmp/test.py", line 5, in <module> data = binascii.unhexlify(bytes("41b21de75262acc9", "utf-8")) TypeError: str() takes at most 1 argument (2 given)
The tests seem to have gotten a lot further, and it looks like you fixed the syntax error in that same commit. But now I'm getting this odd error:
hans@palatschinken biplist $ python3.2 setup.py test running test running egg_info writing requirements to biplist.egg-info/requires.txt writing biplist.egg-info/PKG-INFO writing top-level names to biplist.egg-info/top_level.txt writing dependency_links to biplist.egg-info/dependency_links.txt reading manifest file 'biplist.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'biplist.egg-info/SOURCES.txt' running build_ext testEmptyFile (test_invalid.TestInvalidPlistFile) ... ok testInvalid (test_invalid.TestInvalidPlistFile) ... ok testTooShort (test_invalid.TestInvalidPlistFile) ... ok testEmptyUnicodeRoot (test_valid.TestValidPlistFile) ... ok testFileRead (test_valid.TestValidPlistFile) ... ok testKeyedArchiverPlist (test_valid.TestValidPlistFile) Archive is created with class like this: ... ok testLargeIntegers (test_valid.TestValidPlistFile) ... ok testSmallReal (test_valid.TestValidPlistFile) ... ok testUnicodeRoot (test_valid.TestValidPlistFile) ... ok testBadKeys (test_write.TestWritePlist) ... ok testBoolRoot (test_write.TestWritePlist) ... /code/wooster/biplist/tests/test_write.py:31: ResourceWarning: unclosed file <_io.FileIO name=6 mode='rb'> (status, output) = run_command(['/usr/bin/plutil', '-lint', name]) ok testBools (test_write.TestWritePlist) ... ok testBoolsAndIntegersMixed (test_write.TestWritePlist) ... ok testComplicated (test_write.TestWritePlist) ... ok testConvertToXMLPlistWithData (test_write.TestWritePlist) ... ok testDatetime (test_write.TestWritePlist) ... ok testDictRoot (test_write.TestWritePlist) ... ok testDuplicate (test_write.TestWritePlist) ... ok testFloat (test_write.TestWritePlist) ... ok testFloatsAndIntegersMixed (test_write.TestWritePlist) ... ok testIntBoundaries (test_write.TestWritePlist) ... ok testLargeDict (test_write.TestWritePlist) ... ok testListRoot (test_write.TestWritePlist) ... ok testNone (test_write.TestWritePlist) ... ok testSetRoot (test_write.TestWritePlist) ... ok testString (test_write.TestWritePlist) ... ok testTuple (test_write.TestWritePlist) ... ok testUidWrite (test_write.TestWritePlist) ... ok testUnicode (test_write.TestWritePlist) ... ok testUniques (test_write.TestWritePlist) ... ok testWriteData (test_write.TestWritePlist) ... ok testWriteToFile (test_write.TestWritePlist) ... /code/wooster/biplist/tests/test_write.py:31: ResourceWarning: unclosed file <_io.FileIO name=7 mode='rb'> (status, output) = run_command(['/usr/bin/plutil', '-lint', name]) ok testXMLPlist (test_write.TestWritePlist) ... ok testXMLPlistWithData (test_write.TestWritePlist) ... ok ---------------------------------------------------------------------- Ran 34 tests in 0.223s OK Error in atexit._run_exitfuncs: TypeError: 'NoneType' object is not callable
-
repo owner - changed status to resolved
Fixes issue
#3wherein Python changed rounding rules for microseconds.→ <<cset bbcd11b4bc33>>
-
repo owner Yeah, the bytes() bit doesn't exist in 2.7. You can remove that call and it should work on 2.7, per the snippet further up the comment thread.
The ResourceWarning is a super-annoying Python 3.2 change. Fixed in master.
Meanwhile, I was finally able to reproduce with the Homebrew version of Python 3.4 on OS X. Python 2.7 handles the microseconds part of dates by rounding, with C like so:
fraction = timestamp - (double)timet; us = (int)round_to_long(fraction * 1e6);
whereas Python 3.x after rev 1e9cc1a03365 has rewritten this (in Python), as:
t, frac = divmod(unpacked_date, 1.0) us = int(frac * 1e6)
The obvious bit being that under Python 2.7.x there was a call to round, and they've decided to move away from that.
In any case, this and several other Python 3.4 compat fixes are now in master. If you could check them out and see if they work, that'd be great.
Thanks for your patience!
- Log in to comment
Looks like a rounding mismatch in the fractional seconds. I'm not sure why it's happening though, and can't reproduce on my machines.