1. Matt Chaput
  2. whoosh
Issue #257 resolved

Warnings while running test suite on python 3.2

Anonymous created an issue

Here's some output while running the test suite on python 3.2.2 (fwiw on gentoo). Tests still pass - so not a biggie. To be extra clear: This is 3.2 only. 3.1 has a clean pass.

{{{

!python

whoosh/util.py:141: DeprecationWarning: tostring() is deprecated. Use tobytes() instead. }}}

{{{

!python

test_analysis.test_name_field ... whoosh/util.py:150: DeprecationWarning: tostring() is deprecated. Use tobytes() instead. return _varint(i) whoosh/codec/whoosh2.py:970: DeprecationWarning: tostring() is deprecated. Use tobytes() instead. idcode, idstring = minimize_ids(ids, self.stringids, compression) whoosh/codec/whoosh2.py:1010: DeprecationWarning: fromstring() is deprecated. Use frombytes() instead. self.compression) ok }}}

{{{

!python

test_mpwriter.test_basic_multi ... whoosh/filedb/multiproc.py:229: ResourceWarning: unclosed file <_io.BufferedWriter name=11> self._enqueue() whoosh/filedb/multiproc.py:259: ResourceWarning: unclosed file <_io.BufferedWriter name=11> self._enqueue() whoosh/filedb/multiproc.py:318: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_sbluwgkw9pum.vec'> vreader = None whoosh/filedb/multiproc.py:322: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_sbluwgkw9pum.sto'> sfreader = codec.stored_fields_reader(storage, segment) whoosh/filedb/multiproc.py:341: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_sbluwgkw9pum.vps'> vmatcher = vreader.matcher(i, fieldname, vformat) whoosh/filedb/multiproc.py:318: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_vtq8b77ll43z.vec'> vreader = None whoosh/filedb/multiproc.py:322: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_vtq8b77ll43z.sto'> sfreader = codec.stored_fields_reader(storage, segment) whoosh/filedb/multiproc.py:341: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_vtq8b77ll43z.vps'> vmatcher = vreader.matcher(i, fieldname, vformat) whoosh/filedb/multiproc.py:289: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_rmi0buge6ovq.vec'> self._merge_subsegments(results, mergetype, optimize, merge) whoosh/filedb/multiproc.py:289: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_rmi0buge6ovq.sto'> self._merge_subsegments(results, mergetype, optimize, merge) whoosh/filedb/multiproc.py:289: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_rmi0buge6ovq.vps'> self._merge_subsegments(results, mergetype, optimize, merge) whoosh/filedb/multiproc.py:289: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_rmi0buge6ovq.fln'> self._merge_subsegments(results, mergetype, optimize, merge) whoosh/filedb/multiproc.py:289: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_vtq8b77ll43z.fln'> self._merge_subsegments(results, mergetype, optimize, merge) whoosh/filedb/multiproc.py:289: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/3aa5108a/3aa5108a_sbluwgkw9pum.fln'> self._merge_subsegments(results, mergetype, optimize, merge) ok }}}

{{{

!python

test_sorting.test_float_cache ... whoosh/filedb/fieldcache.py:620: ResourceWarning: unclosed file <_io.BufferedReader name='tmp/floatcache/floatcache_7re3vt7e31vn.num.fc'> return None ok }}}

Comments (14)

  1. Thomas Waldmann

    from http://docs.python.org/dev/whatsnew/3.2.html :

    A new warning category, ResourceWarning, has been added. It is emitted when potential issues with resource consumption or cleanup are detected. It is silenced by default in normal release builds but can be enabled through the means provided by the warnings module, or on the command line.

    A ResourceWarning is issued at interpreter shutdown if the gc.garbage list isn’t empty, and if gc.DEBUG_UNCOLLECTABLE is set, all uncollectable objects are printed. This is meant to make the programmer aware that their code contains object finalization issues.

    A ResourceWarning is also issued when a file object is destroyed without having been explicitly closed. While the deallocator for such object ensures it closes the underlying operating system resource (usually, a file descriptor), the delay in deallocating the object could produce various issues, especially under Windows. Here is an example of enabling the warning from the command line:

    $ python -q -Wdefault
    >>> f = open("foo", "wb")
    >>> del f
    __main__ : 1 : ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
    

    -------------

    I'ld like to add that such stuff also often causes "out of file handles" issues under pypy. As running out of resources is a major (if not even critical) issue, I raised "priority" of this issue to "major".

  2. Thomas Waldmann

    johan, can you please tell which whoosh version you used?

    also, how did you get to see the warnings? nosetests usually does not show warnings if there is not also a test failure. I tried misc. nosetests options, but somehow can't find the right one that lets me see those warnings.

  3. Anonymous

    Hey, sorry for the late replies. Timezones and all.

    version: This is Whoosh 2.4.0 on python 3.2.2. nosetests: I'm at 1.1.2. Running nosetests with -v most often gives you verbose mode, but this time around I used setup.py, in style with:

    $ python3.2 setup.py build -b 3.2
    $ PYTHONPATH=3.2/lib python-3.2 setup.py test

    Let me know if you need something else

  4. Thomas Waldmann
    • changed status to open

    Thanks Johan, getting the warnings now with setup.py (nosetests -v did not help).

    About the renaming of the array methods (from/tostring -> from/tobytes):

    This happened in python 3.2 and the from/tobytes methods are not available in older pythons.

    So, our options are:

    • a version dependant wrapper function in compat.py (we could call it frombytes() / tobytes()), that calls from/tobytes for python >= 3.2 and from/tostring for older python.
    • leave it as is (use .fromstring/.tostring methods). they are deprecated since 3.2, but still there in 3.2 and 3.3. for this case, we maybe should silence the DeprecationWarning.

    Matt, what do you think?

    There is a similar issue with cgi.escape (deprecated since 3.2) vs. html.escape.

  5. Thomas Waldmann

    hmm, shouldn't such stuff go into 2.4x branch?

    if all simple fixes are put into the maintenance branch, you just need to merge the maint branch into the default from time to time.

    otoh, you can't simply merge default into maint (that would make maint pointless as only bugfixes shall be there).

  6. Matt Chaput repo owner

    Re: comment 9: The way they do it where I work is to to do general fixes in the mainline and backport them to relevant older versions. I meant to graft the changeset onto the 2.4x branch after checking it into default and seeing whether tests passed on all platforms, but didn't get to it on the weekend (too sunny here ;)

    Thanks very much for the patch!

  7. Thomas Waldmann

    ah. well, if the code of the "older version(s)" is rather similar to the current mainline, the opposite way is quite easier (and you can also run tests there, which is even more important to not break the next bugfix release).

    that way, you can apply multiple fixes and just merge them all at once into mainline.

    doing it the "cherrypicking and backporting"way, you need to deal with each fix individually (export changeset, apply patch, commit).

    BTW, there is a pending pull-request with that merge. This issue can be closed after pulling.

  8. Log in to comment