py.test broken with use of multiprocessing.Process

Issue #58 resolved
Bluebird created an issue

The following program clearly highlights the problem:

{{{

!python

import os from multiprocessing import Process

def do_task1(): print 'task1: pid=%d' % (os.getpid())

class TestProcess:

def test_p1( self ):
    print 'test_p1: pid=%d' % os.getpid()
    p = Process( name='p1', target=do_task1 )
    p.start()
    p.join()

}}}

Philippe@pc-philippe /cygdrive/d/work/elc-dev/tmp $ py.test -s

============================= test session starts ============================= python: platform win32 -- Python 2.6.1
test object 1: d:\work\elc-dev\tmp

test_process.py test_p1: pid=3920
============================= test session starts ============================= python: platform win32 -- Python 2.6.1
test object 1: d:\work\elc-dev\tmp

test_process.py test_p1: pid=3344
============================= test session starts ============================= python: platform win32 -- Python 2.6.1
test object 1: d:\work\elc-dev\tmp

test_process.py test_p1: pid=208
============================= test session starts ============================= python: platform win32 -- Python 2.6.1
test object 1: d:\work\elc-dev\tmp

test_process.py test_p1: pid=3364
============================= test session starts ============================= python: platform win32 -- Python 2.6.1
test object 1: d:\work\elc-dev\tmp

test_process.py test_p1: pid=536

...

As you can see, Process() is restarting py.test instead of starting the target function.

This is on windows XP, python 2.6.1 .

Comments (10)

  1. Holger Krekel repo owner
    • changed status to open

    Hum, the executable is "py.test" on windows because it doesn't have the the shell-shebang.

    I looked up the multiprocessing code and it seems it offers specifically for win32

    multiprocessing.set_executable(path_to_python)

    can you try that out and see if it helps? best, holger

  2. Bluebird reporter

    First step, I added a:

    print '%d: %s' % (os.getpid(), sys.executable)
    

    Philippe@pc-philippe /cygdrive/d/work/elc-dev/tmp $ py.test -s

    ======================= test session starts

    python: platform win32 -- Python 2.6.1 test object 1: d:\work\elc-dev\tmp 2176: C:\Python26\python.exe

    test_process.py test_p1: pid=2176

    ======================= test session starts

    python: platform win32 -- Python 2.6.1 test object 1: d:\work\elc-dev\tmp 2836: C:\Python26\python.exe

    test_process.py test_p1: pid=2836

    ======================= test session starts

    python: platform win32 -- Python 2.6.1 test object 1: d:\work\elc-dev\tmp 340: C:\Python26\python.exe

    test_process.py test_p1: pid=340

    ======================= test session starts

    python: platform win32 -- Python 2.6.1 test object 1: d:\work\elc-dev\tmp 3192: C:\Python26\python.exe

    test_process.py test_p1: pid=3192

    ======================= test session starts

    python: platform win32 -- Python 2.6.1 test object 1: d:\work\elc-dev\tmp 2212: C:\Python26\python.exe

    test_process.py test_p1: pid=2212

  3. Bluebird reporter

    So, it seems that executables already has the right path, no ?

    For your information, on windows, py.test.exe is kind of a wrapper for running py.test-script.py

    Philippe@pc-philippe /cygdrive/d/work/elc-dev/tmp $ ls /cygdrive/c/Python26/Scr 
    ipts/py.test*                                                                   
    /cygdrive/c/Python26/Scripts/py.test-script.py*                                 
    /cygdrive/c/Python26/Scripts/py.test.exe*                                       
    
    

    And py.test-script.py is from what I understand another wrapper installed by setuptools packaging, to call py.test :

    Philippe@pc-philippe /cygdrive/d/work/elc-dev/tmp $ cat /cygdrive/c/Python26/Sc 
    ripts/py.test-script.py                                                         
    #!C:\Python26\python.exe                                                        
    # EASY-INSTALL-ENTRY-SCRIPT: 'py==1.0.2','console_scripts','py.test'            
    __requires__ = 'py==1.0.2'                                                      
    import sys                                                                      
    from pkg_resources import load_entry_point                                      
                                                                                    
    sys.exit(                                                                       
       load_entry_point('py==1.0.2', 'console_scripts', 'py.test')()                
    )                                                
    

    For example, in one of my scripts, I detect that py.test is running with the following code :

           elif ( sys.argv[0].endswith('py.test-script.py')                    
    
  4. Bluebird reporter

    I tried with the dos command line to see if Cygwin could have had any side-effect, but no difference.

    From reading the documentation of multiprocessing, I am under the impression that the fork() implementation of multiprocessing on windows is not a fork() at all, but a new process that imports all the modules one by one and that can not share any state with the parent process.

    For example, all the arguments to Process.init are serialized with pickle to be transferred to the subprocess.

    I haven't read the code so this is just guessing.

    But if the if the subprocess imports really all the modules again, could this trigger py.test execution ?

  5. Holger Krekel repo owner
    • changed milestone to 1.1

    i looked a bit more at the multiprocess code and there is indeed some hairy pickling going on. The py lib itself should not have re-import or pickling problems itself. Maybe there is some setuptools interaction i am unaware off? Currently i can only very inconveniently test on windows (need to re-make my VirtualBox installs) so could you maybe try to create a test-script "py-test.py" like this:

    import sys
    import py
    py.cmdline.pytest()
    

    this should pick up your globally installed py lib so that you can run:

    python py-test.py -s
    

    and see what that gives?

  6. Former user Account Deleted

    I've checked out py.trunk, so I'll make some tests with setuptools and with the source version to see if there is any difference.

    I'll also test your script.

  7. Bluebird reporter

    Ok, here the status.

    Your example still triggers the infinte recursion. However, by protecting the command line code with main verification, it works fine :

    import sys
    import py
    
    if __name__ == '__main__':
    	py.cmdline.pytest()
    

    That probably means that the main py.test module is imported also in the subprocess.

    Since there is no fork() support on Windows, I suppose that the new process is created empty and that the fork emulation imports all the modules one by one into the new process.

    Note that when I interrupt the py.test, I get this exception. Not sure if it helps :

    =================================== ERRORS ==================================== 
    __________ ERROR during collection d:\work\elc-dev\tmp\my_py_test.py __________ 
                                                                                    
        import sys                                                                  
        import py                                                                   
                                                                                    
        if 0:                                                                       
            if __name__ == '__main__':                                              
                    py.cmdline.pytest()                                             
        else:                                                                       
    >       py.cmdline.pytest()                                                     
                                                                                    
    my_py_test.py:9:                                                                
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
                                                                                    
        def main():                                                                 
    >       py.test.cmdline.main()                                                  
                                                                                    
    C:\Python26\lib\site-packages\py-1.0.2-py2.6.egg\py\cmdline\pytest.py:5:        
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
                                                                                    
    args = ['-s']                                                                   
                                                                                    
        def main(args=None):                                                        
            warn_about_missing_assertion()                                          
            if args is None:                                                        
                args = py.std.sys.argv[1:]                                          
            config = py.test.config                                                 
            try:                                                                    
    >           config.parse(args)                                                  
                                                                                    
    c:\Python26\lib\site-packages\py-1.0.2-py2.6.egg\py\test\cmdline.py:13:         
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
                                                                                    
    self = <py.__.test.config.Config object at 0x00C14450>, args = ['-s']           
                                                                                    
        def parse(self, args):                                                      
            """ parse cmdline arguments into this config object.                    
                    Note that this can only be called once per testing process.     
                """                                                                 
            assert not hasattr(self, 'args'), (                                     
    >               "can only parse cmdline args at most once per Config object")   
    E       AssertionError: can only parse cmdline args at most once per Config object                                                                              
                                                                                    
    c:\Python26\lib\site-packages\py-1.0.2-py2.6.egg\py\test\config.py:86: AssertionError                                                                           
    !!!!!!!!!!!!!!!!!!!!!!!!!!!!! KEYBOARD INTERRUPT !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 
    c:\Python26\lib\multiprocessing\forking.py:259: KeyboardInterrupt               
    =========================== 1 error in 0.78 seconds =========================== 
                                                                                    
    
    
  8. Holger Krekel repo owner
    • changed status to new

    thanks a lot for your investigations! It seems indeed that setuptools creates command line scripts without ``if name == 'main'`` protection. Nose has a similar issue: http://code.google.com/p/python-nose/issues/detail?id=265 I just installed Distribute on my Windows-installation via "easy_install Distribute" and the problem went away!

    I also just added an FAQ entry and reworked the installation instructions, see

    http://codespeak.net/py/trunk/faq.html#what-s-up-with-multiprocess-on-windows http://codespeak.net/py/trunk/install.html

    If you agree this is enough to close the issue please do so.

    thanks & best, holger

  9. Bluebird reporter

    Please add the following other alternatives to the FAQ:

    Fix setuptools install script yourself:

    In Python\Scripts\py.test-script.py , change

    sys.exit(
       load_entry_point('py==1.0.2', 'console_scripts', 'py.test')()
    )
    

    with :

    if __name__ == '__main__':
    	sys.exit(
    	   load_entry_point('py==1.0.2', 'console_scripts', 'py.test')()
    	)
    

    You will probably need to repeat this step each time you upgrade py.test through setuptools

    Create a local py.test script for your project

    Create the following script, name it py.test and use it to invoke py.test for your project.

    import sys
    import py
    
    if __name__ == '__main__':
    	py.cmdline.pytest()
    
  10. Log in to comment