added support for checking files in parallel using multiprocessing

#82 Declined
  1. Michal Nowikowski

This a patch that introduces parallel jobs made by @dahlajn (pull request #55) and then improved according to review comments.

Comments (30)

  1. Sylvain Thénault

    we need a review to ensure that checkers don't have a kind of global state (some have) and decide how to handle that.

  2. Claudiu Popa

    Another problem is that this feature doesn't work in Windows yet (probably due to differences between Unix and Windows regarding process creation). There's a lot of pickling errors which appears when passing -j to pylint.

  3. Claudiu Popa

    Looks good to me. I'm waiting either for Torsten or Sylvain to review it and then we can merge it.

  4. Michal Nowikowski author

    One more question: I added some test to test/ before. But this file has been removed. Where should I place this test?

  5. Michal Nowikowski author

    I just caught up default branch (fixed some test). How is going the review?

    1. Claudiu Popa

      Hi, Michal.

      I don't have that much time right now, but if I don't get back in one week, please ping me. This will be part of 1.4, though, which will be launched soon.

    1. Michal Nowikowski author

      It looks that the problem is connected with Python 2.7 and its multiprocessing bug.


      When I run the command (using -m unittest as in tox.ini) then it fails:

      (pyve) C:\Users\gof\pylint>python -Wi -m unittest discover -s test -p
      .......Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "c:\Python27\Lib\multiprocessing\", line 380, in main
        File "c:\Python27\Lib\multiprocessing\", line 488, in prepare
          assert main_name not in sys.modules, main_name
      AssertionError: __main__
      ERROR: test_parallel_execution (test_self.RunTC)
      Traceback (most recent call last):
        File "C:\Users\gof\pylint\test\", line 131, in test_parallel_execution
          self._runtest(['-j 2', 'pylint.lint', 'pylint.utils'], code=29)
        File "C:\Users\gof\pylint\test\", line 66, in _runtest
          Run(args, reporter=reporter)
        File "C:\Users\gof\pylint\", line 1177, in __init__
        File "C:\Users\gof\pylint\", line 636, in check
        File "C:\Users\gof\pylint\", line 643, in _parallel_check
          manager = multiprocessing.Manager()  # pylint: disable=no-member
        File "c:\Python27\Lib\multiprocessing\", line 99, in Manager
        File "c:\Python27\Lib\multiprocessing\", line 528, in start
          self._address = reader.recv()
      Ran 10 tests in 12.651s
      FAILED (errors=1)

      When I run it using nose it passes:

      (pyve) C:\Users\gof\pylint>..\pyve\Scripts\nosetests.exe test\
      Ran 10 tests in 35.777s

      The same with py.test:

      (pyve) C:\Users\gof\pylint>..\pyve\Scripts\py.test.exe test\
      ============================= test session starts =============================
      platform win32 -- Python 2.7.8 -- py-1.4.24 -- pytest-2.6.2
      collected 10 items
      test/ ..........
      ========================= 10 passed in 42.90 seconds ==========================

      I do not know how to work-around -m unittest. I would switch from -m unittest to py.test. It is much more robust than plain -m unittest and a bit better than nosetests.

      1. Michal Nowikowski author

        How is going the review? What we are doing with this failure? Should we switch test runner?

        1. Claudiu Popa

          Hey, Michal, sorry for the delay.

          I'm having other problems, using Python 3.4, but yours should be fixed as well.

          First of all, test_self.test_all fails with:

          FAIL: test_all (test_self.RunTC)
          Make pylint check itself.
          Traceback (most recent call last)
            File "C:\Python34\lib\unittest\", line 58, in testPartExecutor
            File "C:\Python34\lib\unittest\", line 577, in run
            File "C:\Python34\Lib\site-packages\pylint-1.3.0-py3.4.egg\pylint\test\", line 96, in test_all
              self._runtest(['pylint.lint'], reporter=MultiReporter(reporters))
            File "C:\Python34\Lib\site-packages\pylint-1.3.0-py3.4.egg\pylint\test\", line 77, in _runtest
              self.assertEqual(ex.code, code, msg)
            File "C:\Python34\lib\unittest\", line 797, in assertEqual
              assertion_func(first, second, msg=msg)
            File "C:\Python34\lib\unittest\", line 790, in _baseAssertEqual
              raise self.failureException(msg)
          AssertionError: 30 != 28 : expected output status 28, got 30. Below pylint output: 

          Second, test_all.test_parallel_execution blocks forever and must be stopped after some time. I'll try to investigate if I can find some time.

          1. Michal Nowikowski author

            This 30 != 28 is the same problem as you mentioned at the beginning of this thread i.e. it is connected with -m unittest. The second case with blocking I didn't encounter. How can I reproduce it?

            1. Claudiu Popa

              This 30 != 28 problem seems to be related to something else, I'm running the test manually and I see the same error. Probably your patch adds a couple of warnings, which results in a different status code. Regarding the second problem, I don't know how to reproduce it, I tested this on a Windows 8 machine.

                1. Claudiu Popa

                  So, using py27 doesn't give that 30 != 28 problem, but test_parallel_execution still blocks.

      2. Claudiu Popa

        Sorry for the spam. :-) So, running individually doesn't block on Python 3, only with -m discover. Regarding a new test runner, I don't like py.test, it's too magical and I'm not sure if it knows how to handle unittest style assertions, please correct me if I am wrong.

        1. Michal Nowikowski author


          I was looking into these problems. One problem is with test test_all and also with my test_parallel_execution in Both of them check pylint.lint module. And while code of this file changes the returned exitcode from pylint also may change. In our case it changed from expected 28 to 30. So this problem should also be present on default branch (but I didn't check it yet). I would change the tested file to something more stable than pylint.lint.

          Still investigating problem with blocking.

          1. Michal Nowikowski author


            I was trying to fix the problem with hung in Py3.4 on Windows. I see that this is a problem with new Message class and its pickling, and maybe with getnewargs method. Unfortunately it happens only from time to time what is very strange. Now I added a check if problem occurs and handle it gracefully so the program does not hang as before.

            1. Michal Nowikowski author

              This is some nightmare. I wanted to debug pickle so I switched to its pure python implementation because by default native one is used. And then this problem does not appear. So it looks that it may be some bug in pickle native implementation. What is so discouraging is that this problem appears only in tests. When parallel check is run from commandline then all is also working :/

  6. Michal Nowikowski author

    OK. I made the communication between processes more robust so that the Pylint does not hung if there are any problems.

    Regarding problems on Windows, there are two of them. One is connected with bug in multiprocessing on Py27, the second one is connected with pickling and multiprocessing on Py3x. Both of them are caused by the way of running tests (not the test and not the problems in jobs code). The first one does not appear if tests are run by nosetests or py.tests (this is due to this bug nature, more here: The second one is connected with some problems with native version of pickle library - if I use pure python version the problem does not appear.

    So I do not have idea how to make this test working on windows. What is important here is that the test is proper and the implementation of jobs in Pylint is ok and works. So to not fail testing process I added skipping this test on windows (I added proper comment to code in this test).

    So I would like to ask you to complete the review of tihs pull-request.

    1. Claudiu Popa

      Thank you for your extraordinary work, Michal. I have some workshops going on this weekend, so I won't be able to finish the review this weekend, but I'll try to finalize your pull request at the start of the week.

  7. Claudiu Popa

    Hey, Michal. Sorry for taking that long. I finally managed to make the tests work on Windows. I'll add a couple of comments where you should apply the new modifications.