Error in atexit._run_exitfuncs:

Issue #1007 closed
Bogdan Tabacaru created an issue

Hi everyone,

I discovered an issue when testing an openpyxl 2.5.1 application with setup.py and py.test.

It occurs when using Python 2.7.14.

It does not occur on Python 3.6.3.

I haven't tried other Python versions.

PFA the minimum setup necessary to reproduce the bug.

Let me know if I should provide any other information to fix this bug.

Description

To reproduce the issue, I created a simple program which opens an XLSX file and saves it under a different name. Additionally, I created a test case using unittest.

I am executing the test as explained here:

https://pypi.python.org/pypi/pytest-runner

Trace Dump

python setup.py pytest
running pytest
running egg_info
creating UNKNOWN.egg-info
writing requirements to UNKNOWN.egg-info/requires.txt
writing UNKNOWN.egg-info/PKG-INFO
writing top-level names to UNKNOWN.egg-info/top_level.txt
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
running build_ext
============================================================ test session starts =============================================================
platform linux2 -- Python 2.7.14, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /home/boggers/eclipse-workspace/openpyxl_write_only_atexit_issue, inifile:
collected 1 item                                                                                                                              

test_main.py .

========================================================== 1 passed in 0.18 seconds ==========================================================
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/openpyxl/worksheet/write_only.py", line 31, in _openpyxl_shutdown
    for path in ALL_TEMP_FILES:
TypeError: 'NoneType' object is not iterable
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/local/lib/python2.7/dist-packages/openpyxl/worksheet/write_only.py", line 31, in _openpyxl_shutdown
    for path in ALL_TEMP_FILES:
TypeError: 'NoneType' object is not iterable

My Two Cents

For some reason, the global list ALL_TEMP_FILES is treated as None when running the _openpyxl_shutdown() method. If I simply add and if statement that returns if the list is None (see below), this issue does not occur any more. However, I am not sure about the consequences of this dumb if statement.

@atexit.register
def _openpyxl_shutdown():
    global ALL_TEMP_FILES
    if ALL_TEMP_FILES is None:
        return
    for path in ALL_TEMP_FILES:
        if os.path.exists(path):
            os.remove(path)

On a different note, I also found this open issue. Is it possible that they are related?

https://bitbucket.org/openpyxl/openpyxl/issues/838/write_only-atexit-not-safe

Comments (3)

  1. CharlieC

    I suspect this could be an artefact of the way you invoke the test as much as anything else. python setup.py pytest is not encouraged in general and certainly not how we test pytest.

    If you can create a relevant test for openpxl then you can run it for all relevant versions with tox.

    The bugs are unrelated as you're not using write-only mode.

  2. Jun Omae

    I encounter the same issue while testing python setup.py test with unittest library.

    py27-trac10-pyxl26 create: /home/jun66j5/src/exceldownloadplugin.git/.tox/py27-trac10-pyxl26
    py27-trac10-pyxl26 installdeps: Trac>=1.0,<1.1dev, openpyxl>=2.6.0,<2.7.0dev
    py27-trac10-pyxl26 inst: /home/jun66j5/src/exceldownloadplugin.git/.tox/.tmp/package/1/ExcelDownloadPlugin-0.12.0.6.zip
    py27-trac10-pyxl26 installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,et-xmlfile==1.0.1,ExcelDownloadPlugin==0.12.0.6,Genshi==0.7.1,jdcal==1.4,openpyxl==2.6.0,Trac==1.0.17
    py27-trac10-pyxl26 run-test-pre: PYTHONHASHSEED='1088758177'
    py27-trac10-pyxl26 runtests: commands[0] | python setup.py test -v
    running test
    running egg_info
    writing entry points to ExcelDownloadPlugin.egg-info/entry_points.txt
    writing top-level names to ExcelDownloadPlugin.egg-info/top_level.txt
    writing dependency_links to ExcelDownloadPlugin.egg-info/dependency_links.txt
    writing ExcelDownloadPlugin.egg-info/PKG-INFO
    writing requirements to ExcelDownloadPlugin.egg-info/requires.txt
    reading manifest file 'ExcelDownloadPlugin.egg-info/SOURCES.txt'
    writing manifest file 'ExcelDownloadPlugin.egg-info/SOURCES.txt'
    running build_ext
    test_query (tracexceldownload.tests.ticket.Excel2007TicketTestCase) ... ok
    test_report (tracexceldownload.tests.ticket.Excel2007TicketTestCase) ... ok
    test_ticket (tracexceldownload.tests.ticket.Excel2007TicketTestCase) ... ok
    
    ----------------------------------------------------------------------
    Ran 3 tests in 0.161s
    
    OK
    Error in atexit._run_exitfuncs:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
        func(*targs, **kargs)
      File "/home/jun66j5/src/exceldownloadplugin.git/.tox/py27-trac10-pyxl26/lib/python2.7/site-packages/openpyxl/worksheet/_writer.py", line 31, in _openpyxl_shutdown
        for path in ALL_TEMP_FILES:
    TypeError: 'NoneType' object is not iterable
    Error in sys.exitfunc:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
        func(*targs, **kargs)
      File "/home/jun66j5/src/exceldownloadplugin.git/.tox/py27-trac10-pyxl26/lib/python2.7/site-packages/openpyxl/worksheet/_writer.py", line 31, in _openpyxl_shutdown
        for path in ALL_TEMP_FILES:
    TypeError: 'NoneType' object is not iterable
    
    Error in atexit._run_exitfuncs:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
        func(*targs, **kargs)
      File "/home/jun66j5/src/exceldownloadplugin.git/.tox/py27-trac12-pyxl24/lib/python2.7/site-packages/openpyxl/writer/write_only.py", line 33, in _openpyxl_shutdown
        if os.path.exists(path):
    AttributeError: 'NoneType' object has no attribute 'path'
    

    The issue is caused by the ALL_TEMP_FILES is collected by GC when _openpyxl_shutdown is invoked on Python termination.

    We could pass the ALL_TEMP_FILES and os variables to atexit.register to prevent the GC, rather than @atexit.register decorator.

    diff -r efe2c845c2ec openpyxl/worksheet/_writer.py
    --- a/openpyxl/worksheet/_writer.py     Tue Feb 19 19:21:09 2019 +0100
    +++ b/openpyxl/worksheet/_writer.py     Wed Feb 20 15:50:49 2019 +0900
    @@ -26,12 +26,13 @@
    
     ALL_TEMP_FILES = []
    
    -@atexit.register
    -def _openpyxl_shutdown():
    -    for path in ALL_TEMP_FILES:
    +def _openpyxl_shutdown(temp_files, os):
    +    for path in temp_files:
             if os.path.exists(path):
                 os.remove(path)
    
    +atexit.register(_openpyxl_shutdown, ALL_TEMP_FILES, os)
    +
    
     def create_temporary_file(suffix=''):
         fobj = NamedTemporaryFile(mode='w+', suffix=suffix,
    
  3. Log in to comment