Importing module when PyPy is kernel for Jupyter Notebook

Issue #2690 resolved
created an issue

I encountered a problem when pypy is configured as a kernel for a Jupyter Notebook and I tried to import a module that contains calls the display() rountine in the IPython.display package. This problem does not occur when pypy3 is configured as the kernel for the Jupyter Notebook nor when Python2 or Python3 is configured as the kernel. It only occurs when pypy for Python2 is configured as the kernel.

What happens is that when you execute the code to import a module that calls IPython.display.display() routine the kernel hangs and the import never completes.

I have jupyter installed on an ubuntu linux machine using Anaconda with Python 3. I also installed pypy and pypy3 on the ubuntu linux machine and configured them as kernels for the Jupyter notebook. When I open a notebook I am able to switch between the 3 kernels, namely Python3 , PyPy and PyPy3. The problem I am having is that the following import statement fails when PyPy is configured as the kernel for the notebook.

from test import *

executing this line in a notebook cell fails when PyPy is the kernel for the notebook, but it works fine when the notebook is configured to have the PyPy3 kernel or the Python3 kernel. It also works fine when Python2 is the kernel. The contents of the file is the following.

import IPython

from IPython.display import display

a = 1


When I execute the cell containing

from test import *

when PyPY is the kernel the cell execution never completes and the * symbol remains displayed to the left of the cell indefinitely.

If I put the contents of in a notebook cell and execute the cell it works fine on pypy. It's only when I try to import test that the system hangs.

To recreate the problem you need to install Jupyter,

and also install pypy and pypy3.

Next you need to configure pypy as a kernel for the jupyter notebook following these steps.

Step 1: Download PyPy and install pypy on your machine

Step 2: Add the installed path to pypy on your machine to your PATH environment variable

Step 3: Add pip to PyPy from the terminal window

        pypy -m ensurepip

Step 4: pip install notebook and its dependencies to pypy

        pypy -m pip install notebook

Step 5: Add pypy kernel to Jupyter Notebook

        pypy -m ipykernel install --user --name pypy --display-name "Python (PyPy)"

If these steps were successful, then you should see the PyPy kernel as one of the available kernels in the Jupyter Notebook. From the terminal window run

jupyter notebook

which launches the jupyter notebook in a browser window. Then click the "file" button in the notebook and you should see "Python (PyPy)" as a kernel that you can use. Select PyPy to create a PyPy notebook. Then in the cell of the notebook type.

import sys sys.version

and execute the cell. It should indicate that it is using PyPy.

To execute a cell with python code in a jupyter notebook you click on the cell and then use the key sequence (Shift+Enter) to execute the python code in the cell.

With pypy as the kernel try executing the contents of the code in a notebook cell.

import IPython

from IPython.display import display

a = 1


Then execute this code in the cell (Shift+Enter) key sequence.

Next try importing in a notebook cell.

from test import *

and you will see that it hangs when pypy for Python 2 is the kernel. It works fine for all other kernels.

To change the kernel in a Jupyter Notebook you can use the menu in an open notebook. (Kernel -> Change Kernel ) . You can also restart the kernel using the Kernel menu item in the open notebook.

Comments (6)

  1. mattip

    can reproduce. By adding pip install rpdb to PyPy2, then calling import rpdb;rpdb.set_trace() just before the line display(a), I can dive into the hanging function call. So far I have gotten it down to a hanging call to site-packages/ipykernel/ which is a call to self._flush_streams(), still digging.

  2. mattip

    (I didn't say that to connect to the rpdb, I used nc localhost 4444 in another terminal window after the pypy kernel hit the breakpoint) here is where it hangs

    > pypy2-v5.9.0-linux64/site-packages/ipykernel/
    -> self.pub_thread.schedule(self._flush)
    (Pdb) l
    314             send will happen in the background thread
    315             """
    316             if self.pub_thread.thread.is_alive():
    317                 # wait for flush to actually get through:
    318  ->             self.pub_thread.schedule(self._flush)
    319                 evt = threading.Event()
    320                 self.pub_thread.schedule(evt.set)
    321                 evt.wait()
    322             else:
    323                 self._flush()
    (Pdb) n
    > pypy2-v5.9.0-linux64/site-packages/ipykernel/
    -> evt = threading.Event()
    (Pdb) n
    > pypy2-v5.9.0-linux64/site-packages/ipykernel/
    -> self.pub_thread.schedule(evt.set)
    (Pdb) n
    > pypy2-v5.9.0-linux64/site-packages/ipykernel/
    -> evt.wait()
    (Pdb) n

    It seems to be some interaction with display() and the "normal" iostream handling. Maybe the IPython people could shed some light on the problem?

  3. Min RK

    Here's a test setup that demonstrates the issue without zmq or tornado. It seems to be a general issue of waiting for events that will be set by other threads during import. As I tweak the example, I can reproduce similar issues in various CPython and PyPy versions, so I think this is probably something we should try to work around on the IPython side.

    I don't believe we are violating Python's stated import threading conditions because we aren't spawning a new thread during the same import that waits for it, but we are waiting for an event from a thread here.

    I've opened a PR to ipykernel that avoids waiting for any events during import in this case.

  4. Log in to comment