1. Ned Batchelder
  2. coverage.py
Issue #211 invalid

"unhashable type: dict" under Python tracer, passes under C tracer

Ned Batchelder
repo owner created an issue

Ok, I got it. This will fail with TypeError: unhashable type: 'dict' when running under coverage not using the extension module. It will pass when the extension module is installed. In either case, it will pass when not running under coverage.

Here is what I have:

# urio@gravytrain ~
$ sudo pip install coverage
Downloading/unpacking coverage
  Downloading coverage-3.5.3.tar.gz (117kB): 117kB downloaded
  Running setup.py egg_info for package coverage

    no previously-included directories found matching 'test'
Installing collected packages: coverage
  Running setup.py install for coverage
    building 'coverage.tracer' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c coverage/tracer.c -o build/temp.linux-x86_64-2.7/coverage/tracer.o
    coverage/tracer.c:3:20: fatal error: Python.h: No such file or directory
    compilation terminated.
    **
    ** Couldn't install with extension module, trying without it...
    ** SystemExit: error: command 'gcc' failed with exit status 1
    **

    no previously-included directories found matching 'test'
    Installing coverage script to /usr/local/bin
Successfully installed coverage
Cleaning up...

# urio@gravytrain ~/test
$ t
.
├── foo.py
└── test_foo.py

# urio@gravytrain ~/test
$ cat foo.py
#!/usr/bin/python

import itertools

def wacky(a, b, c, d):
    # Gather a list of selection criteria.  This is based on all possible
    # combinations of criteria that match.
    criteria = vars()
    criteria_keys = criteria.keys()
    criteria_set = set()
    for number_of_criteria in range(1, len(criteria)+1):
        selection_sets = itertools.combinations(criteria_keys, number_of_criteria)
        for selection_set in selection_sets:
            active_criteria = []
            for criterion in criteria_keys:
                if criterion in selection_set:
                    active_criteria.append(criteria[criterion])
                else:
                    active_criteria.append(None)
            active_criteria = tuple(active_criteria)
            # add unique criteria set tuples to the list of criteria sets
            criteria_set.add(active_criteria)
    # Convert set of criteria tuples into list of criteria dicts
    criteria_set = [dict(zip(criteria_keys, ct_set)) for ct_set in criteria_set]
    return criteria_set


# urio@gravytrain ~/test
$ cat test_foo.py
#!/usr/bin/python

import unittest

import foo

class FooTestCase(unittest.TestCase):
    def test_foo(self):
        criteria = {'a': 'A', 'b': 'B', 'c': 'C', 'd': 'D'}
        foo.wacky(**criteria)


if __name__ == '__main__':
    unittest.main(verbosity=2)



You run it like this:

# urio@gravytrain ~/test
$ ./test_foo.py
test_foo (__main__.FooTestCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK


# urio@gravytrain ~/test
$ coverage run test_foo.py
test_foo (__main__.FooTestCase) ... ERROR

======================================================================
ERROR: test_foo (__main__.FooTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_foo.py", line 10, in test_foo
    foo.wacky(**criteria)
  File "/home/urio/test/foo.py", line 22, in wacky
    criteria_set.add(active_criteria)
TypeError: unhashable type: 'dict'

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

Now, if I reinstall coverage with the extension module built:

# urio@gravytrain ~
$ sudo pip uninstall coverage
...
# urio@gravytrain ~
$ sudo apt-get install python-dev
...
# urio@gravytrain ~
$ sudo pip install coverage
Downloading/unpacking coverage
  Downloading coverage-3.5.3.tar.gz (117kB): 117kB downloaded
  Running setup.py egg_info for package coverage

    no previously-included directories found matching 'test'
Installing collected packages: coverage
  Running setup.py install for coverage
    building 'coverage.tracer' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c coverage/tracer.c -o build/temp.linux-x86_64-2.7/coverage/tracer.o
    gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/coverage/tracer.o -o build/lib.linux-x86_64-2.7/coverage/tracer.so

    no previously-included directories found matching 'test'
    Installing coverage script to /usr/local/bin
Successfully installed coverage
Cleaning up...

And now...

# urio@gravytrain ~/test
$ ./test_foo.py 
test_foo (__main__.FooTestCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

# urio@gravytrain ~/test
$ coverage run test_foo.py
test_foo (__main__.FooTestCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

I'm using

# urio@gravytrain ~
$ python --version
Python 2.7.3

# urio@gravytrain ~
$ coverage --version
Coverage.py, version 3.5.3.  http://nedbatchelder.com/code/coverage

I'm attaching an archive of the files.

Thanks,

On Thu, Nov 8, 2012 at 5:10 PM, Uri Okrent uokrent@gmail.com wrote:

Well, unfortunately this code is part of a proprietary system so I
cant point you at anything.  I'll try and distill a minimal
reproducible case for you.

--
   Uri

Please consider the environment before printing this message.
http://wwf.panda.org/savepaper/

-- Uri

Comments (4)

  1. Ned Batchelder reporter

    This is a result of using vars() in your function. That returns a reference to a dictionary. That dictionary is updated from the local symbol table whenever you call vars(), but also on every statement if there is a trace function in effect. This is a subtle aspect of CPython's implementation, it's there so that debuggers will work properly.

    The result is that your criteria is a,b,c,d under normal execution, but a,b,c,d,criteria under a trace function. You'll see the same differing behaviors if you run your code in a debugger, and try it a) single-stepping, it will fail; or b) letting it run free with "c(ontinue)", it will succeed.

    The odd thing is that this indicates to me that it should fail under any form of coverage measurement, which is just what I observe. I'm not sure why you see different behavior with the Python tracer and the C tracer.

    You can fix your code by changing this line:

    criteria = vars()
    

    to:

    criteria = dict(vars())
    

    This is get you your own dictionary of the variables at the time of the call, rather than just getting a reference to a dictionary which will later be modified underneath you.

    BTW, this is the subject of a closed bug against CPython: http://bugs.python.org/issue7083

  2. Ned Batchelder reporter

    BTW: if you want a much more concise demonstration of the effect:

    import sys
    
    def trace(frame, event, arg):
        return trace
    
    def foo(a):
        v = vars()
        print(v.keys())
    
    foo(1)
    sys.settrace(trace)
    foo(1)
    sys.settrace(None)
    foo(1)
    

    Simply run this with Python, no coverage or debugger needed. This outputs:

    ['a']
    ['a', 'v']
    ['a']
    

    The second line shows that v is visible in the dict because with the trace function, the dict is updated once the v = vars() line is executed, so the referenced dictionary has had v added to it.

  3. Uri Okrent

    Wow, tricky. Another thing to add to the list of esoteric implementation-detail-specific factoids I'm building up on python. Thanks for the find (and for another reminder to treat the "considered harmful" aspects of the language with due respect).

  4. Log in to comment