testFileRead fails using python 3.x

[Hans-Christoph Steiner]

I'm in the process of packaging biplist for Debian. Building it for upload, I run into this problem with the tests:

======================================================================
FAIL: testFileRead (test_valid.TestValidPlistFile)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/buildd/python-biplist-0.6/tests/test_valid.py", line 26, in testFileRead
    self.validateSimpleBinaryRoot(result)
  File "/tmp/buildd/python-biplist-0.6/tests/test_valid.py", line 15, in validateSimpleBinaryRoot
    self.assertEquals(root[six.b('dateItem')], datetime.datetime(2010, 8, 19, 22, 27, 30, 385449), "dates not equal" )
AssertionError: datetime.datetime(2010, 8, 19, 22, 27, 30, 385448) != datetime.datetime(2010, 8, 19, 22, 27, 30, 385449) : dates not equal

Here are the relevant package versions:

Setting up libpython3.3-minimal:amd64 (3.3.2-7) ...
Setting up libpython3.3-stdlib:amd64 (3.3.2-7) ...
Setting up libunistring0:amd64 (0.9.3-5) ...
Setting up libpython2.7-minimal:amd64 (2.7.5-8) ...
Setting up python2.7-minimal (2.7.5-8) ...
Setting up python3.3-minimal (3.3.2-7) ...
Setting up libpython2.7-stdlib:amd64 (2.7.5-8) ...
Setting up python2.7 (2.7.5-8) ...
Setting up python-minimal (2.7.5-5) ...
Setting up libpython-stdlib:amd64 (2.7.5-5) ...
Setting up python (2.7.5-5) ...
Setting up python3.3 (3.3.2-7) ...
Setting up python3-minimal (3.3.2-17) ...
Setting up libpython3-stdlib:amd64 (3.3.2-17) ...
Setting up python-all (2.7.5-5) ...
Setting up python-pkg-resources (0.6.49-2) ...
Setting up python-coverage (3.7+dfsg.1-2) ...
Setting up python-nose (1.3.0-2) ...
Setting up python-setuptools (0.6.49-2) ...
Setting up python-six (1.4.1-1) ...
Setting up python3 (3.3.2-17) ...
Setting up python3-all (3.3.2-17) ...
Setting up python3-pkg-resources (0.6.49-2) ...
Setting up python3-coverage (3.7+dfsg.1-2) ...
Setting up python3-nose (1.3.0-2) ...
Setting up python3-setuptools (0.6.49-2) ...
Setting up python3-six (1.4.1-1) ...

Attached is the full build log.

[Andrew Wooster]

Looks like a rounding mismatch in the fractional seconds. I'm not sure why it's happening though, and can't reproduce on my machines.

[Hans-Christoph Steiner]

I'd like to run the test suite as part of the packaging, what do you recommend? Maybe make that test optional or less stringent?

[Andrew Wooster]

Optimally I'd like to fix it. I'll be doing some work with Debian this weekend on EC2, and will check to see if I can repro this. If not, I'll probably need more detailed configuration information (kernel version, Debian distro, hardware info, etc). Also, is this repeatable? (eg does it break every time)

[Andrew Wooster]

Just tested this with Debian precise64 on Intel hardware and wasn't able to reproduce. I'll take a patch, but I don't have amd64 hardware to test this on and not a lot of time to look into it.

I wouldn't expect Intel and AMD hardware to behave differently on the actual floating point numbers, so it's possible there's a behavior difference not in my code.

The code failure in question reduces to:

import binascii
import datetime
import struct

data = binascii.unhexlify('41b21de75262acc9')
unpacked_date = struct.unpack(">d", data)[0]
date = datetime.datetime.utcfromtimestamp(unpacked_date + 978307200)
date == datetime.datetime(2010, 8, 19, 22, 27, 30, 385449)

It would be helpful to know at which point in the above things differ from other platforms.

[Andrew Wooster]

Actually, I remembered another machine I have access to…

Tested on Gentoo Linux/amd64 AMD Opteron 6276, and got expected output both for the above and for the unit tests.

Given that, this seems like a software difference somewhere else in the stack.

[Hans-Christoph Steiner]

Did you try running the tests with Python 3.x? The tests work fine with Python 2.7 for me. I tried it on two computers (FYI: amd64 is the same thing as x86_64, it is the generic architecture name for AMD and Intel 64-bit CPUs):

All tests pass on:

• Debian/unstable/amd64/Intel Core2 with Python 2.7.8 and six 1.7.3
• Debian/jessie/amd64/Intel server with Python 2.7.8 and six 1.7.3

The tests fail with the same error on:

• Debian/unstable/amd64/Intel Core2 with Python 3.4.1 and six 1.7.3
• Debian/jessie/amd64/Intel server with Python 3.4.1 and six 1.7.3

Also, running the biplist 0.7 tests on Debian/jessie/amd64/Intel server fails with a syntax error, so my guess is that the tests are not usually run using python3:

$ python3.4 /home/pd/code/wooster/biplist/tests/test_write.py
  File "/home/pd/code/wooster/biplist/tests/test_write.py", line 185
    self.roundTrip([0x10000000000000000L, pow(2, 64)])
                                       ^
SyntaxError: invalid syntax

I've been running the tests under python3 using:

python3 setup.py test -vvv

[Andrew Wooster]

I added some fixes in master for running tests on Python 3:

https://bitbucket.org/wooster/biplist/commits/4dbbd79c3563d5feb505cd52458610c7abd138a0

I was only able to test on Python 3.2, at the moment.

In the tests, as well as running the following, I didn't get the error you're seeing:

import binascii
import datetime
import struct

data = binascii.unhexlify(bytes("41b21de75262acc9", "utf-8"))
unpacked_date = struct.unpack(">d", data)[0]
date = datetime.datetime.utcfromtimestamp(unpacked_date + 978307200)
date == datetime.datetime(2010, 8, 19, 22, 27, 30, 385449)

[Hans-Christoph Steiner]

I ran the test snippet on Ubuntu/precise/64-bit and it works on 3.2 but not 2.7. It also gave the same error on Mac OS X Mavericks using python2.7:

$ python3.2 /tmp/test.py 
$ python2.7 /tmp/test.py 
Traceback (most recent call last):
  File "/tmp/test.py", line 5, in <module>
    data = binascii.unhexlify(bytes("41b21de75262acc9", "utf-8"))
TypeError: str() takes at most 1 argument (2 given)

The tests seem to have gotten a lot further, and it looks like you fixed the syntax error in that same commit. But now I'm getting this odd error:

hans@palatschinken biplist $ python3.2 setup.py test 
running test
running egg_info
writing requirements to biplist.egg-info/requires.txt
writing biplist.egg-info/PKG-INFO
writing top-level names to biplist.egg-info/top_level.txt
writing dependency_links to biplist.egg-info/dependency_links.txt
reading manifest file 'biplist.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'biplist.egg-info/SOURCES.txt'
running build_ext
testEmptyFile (test_invalid.TestInvalidPlistFile) ... ok
testInvalid (test_invalid.TestInvalidPlistFile) ... ok
testTooShort (test_invalid.TestInvalidPlistFile) ... ok
testEmptyUnicodeRoot (test_valid.TestValidPlistFile) ... ok
testFileRead (test_valid.TestValidPlistFile) ... ok
testKeyedArchiverPlist (test_valid.TestValidPlistFile)
Archive is created with class like this: ... ok
testLargeIntegers (test_valid.TestValidPlistFile) ... ok
testSmallReal (test_valid.TestValidPlistFile) ... ok
testUnicodeRoot (test_valid.TestValidPlistFile) ... ok
testBadKeys (test_write.TestWritePlist) ... ok
testBoolRoot (test_write.TestWritePlist) ... /code/wooster/biplist/tests/test_write.py:31:
  ResourceWarning: unclosed file <_io.FileIO name=6 mode='rb'>
  (status, output) = run_command(['/usr/bin/plutil', '-lint', name])
ok
testBools (test_write.TestWritePlist) ... ok
testBoolsAndIntegersMixed (test_write.TestWritePlist) ... ok
testComplicated (test_write.TestWritePlist) ... ok
testConvertToXMLPlistWithData (test_write.TestWritePlist) ... ok
testDatetime (test_write.TestWritePlist) ... ok
testDictRoot (test_write.TestWritePlist) ... ok
testDuplicate (test_write.TestWritePlist) ... ok
testFloat (test_write.TestWritePlist) ... ok
testFloatsAndIntegersMixed (test_write.TestWritePlist) ... ok
testIntBoundaries (test_write.TestWritePlist) ... ok
testLargeDict (test_write.TestWritePlist) ... ok
testListRoot (test_write.TestWritePlist) ... ok
testNone (test_write.TestWritePlist) ... ok
testSetRoot (test_write.TestWritePlist) ... ok
testString (test_write.TestWritePlist) ... ok
testTuple (test_write.TestWritePlist) ... ok
testUidWrite (test_write.TestWritePlist) ... ok
testUnicode (test_write.TestWritePlist) ... ok
testUniques (test_write.TestWritePlist) ... ok
testWriteData (test_write.TestWritePlist) ... ok
testWriteToFile (test_write.TestWritePlist) ... /code/wooster/biplist/tests/test_write.py:31:
  ResourceWarning: unclosed file <_io.FileIO name=7 mode='rb'>
  (status, output) = run_command(['/usr/bin/plutil', '-lint', name])
ok
testXMLPlist (test_write.TestWritePlist) ... ok
testXMLPlistWithData (test_write.TestWritePlist) ... ok

----------------------------------------------------------------------
Ran 34 tests in 0.223s

OK
Error in atexit._run_exitfuncs:
TypeError: 'NoneType' object is not callable

[Andrew Wooster]

Fixes issue #3 wherein Python changed rounding rules for microseconds.

→ <<cset bbcd11b4bc33>>

[Andrew Wooster]

Yeah, the bytes() bit doesn't exist in 2.7. You can remove that call and it should work on 2.7, per the snippet further up the comment thread.

The ResourceWarning is a super-annoying Python 3.2 change. Fixed in master.

Meanwhile, I was finally able to reproduce with the Homebrew version of Python 3.4 on OS X. Python 2.7 handles the microseconds part of dates by rounding, with C like so:

fraction = timestamp - (double)timet;
us = (int)round_to_long(fraction * 1e6);

whereas Python 3.x after rev 1e9cc1a03365 has rewritten this (in Python), as:

t, frac = divmod(unpacked_date, 1.0)
us = int(frac * 1e6)

The obvious bit being that under Python 2.7.x there was a call to round, and they've decided to move away from that.

In any case, this and several other Python 3.4 compat fixes are now in master. If you could check them out and see if they work, that'd be great.

Thanks for your patience!