Issue with numpy and auto completion feature

[Former user]

Hello ITOM Dev Team,

I'm currently having an issue with the auto completion feature and numpy. Whenever it tries to do the auto completion e.g. when typing "np." it takes a few seconds and crashes ITOM. I also tried it in debug mode, where it throws an exception "Unknown exception in jediCompletionRequestEnqueued. Please report this bug." I didn't test every package, but right now it looks like only numpy is causing this problem.

Some info:

I use a self compiled ITOM (4.1.0) - VS 2019 - Qt 5.14.2 - numpy 1.19.3 - jedi 0.17.2

Thank you in advance and best regards from ITO ;),

Florian Rothermel

[M. Gronle]

Hi Flo,

this problem is somehow known, but there is still no perfect fix for this. The problem is that the Python package jedi requires a lot of recursions for the introspection of some packages, especially numpy, scipy or pandas. If the number of recursions is too high an uncaughtable StackOverflow Exception occurs, which will mostly crash the application and sometimes the application recovers from it.

The maximum recursion limit can be limited in Python. Per default, jedi set a recursion limit of 3000. We already limited this in the build or install folder of itom: itom/itom-packages/itomJediLib.py: maxreclimit. Please try to reduce this value to for instance 600 or 800 and check if the crash still occurs for your system (restart itom after having changed the value). This will help us to find a suitable maximum recursion limit. Of course, the smaller this limit is, the less deep Jedi can analyze complex packages for auto completions, help etc.

Currently, the reduced maximum recursion limit has an impact to all Python codes, executed with that itom session. However, I think about locally reducing this value only for Jedi calls.

If you would try out your “limit value”, this would be very great.

I also would like to inform @Johann Krauter about this crash, since he had similar findings.

Cheers

Marc

[[e-mail]]

Hey Marc,

thank you, this did the trick. I found my limit value to be 918 (it does crash for 919). I hope this helps.

Best regards,

Flo

[M. Gronle]

Hi Flo,

thanks for these findings. By means of a test script with an infinite recursion, it can be checked how many recursions can be made in Python until a stack overflow error occurs. I tried this out and have seen, that if the script is executed in pure Python much more recursions are possible than within itom. Therefore I am currently checking if it would be worth to increase the maximum stack size of the main qitom application. In firsts tests the maximum Python recursion depth can be highly increased. I only have to find out if what are the drawbacks about increasing the stack size different from the standard size. If itom is compiled with gcc (Linux), the current stack depth is much bigger than with Windows / Visual Studio.

I will check this and probably solve the problem by increasing this stack buffer size.

Cheers

Marc

[M. Gronle]

Hi Flo,

I did now some improvements to locally change the maximum stack size for the Python thread and a worker thread, that usually executes Jedi requests. Due to this I increased now the maximum allowed recursion depth for Jedi calls to 1800, which should be far below the stack overflow limit of Python in itom with the increased stack size. Could you get the latest updates from the Master branch of itom and build itom again and see if auto completions work now without problems?

Thank you.

Marc

[[e-mail]]

Hi Marc,

I tried your updates and it works just fine now, thank you!

Best regards,

Flo

[M. Gronle]

Fixed by commits c1ef80f, 55706b5 and aea2bb3