100% CPU + segfault after a while

Nginx frontend + PECL ev based backend PHP WebSocket server. (Running on Ubuntu LTS 18.04.)

In general, the backend server works totally fine and handles thousands of simultaneous connections without issue. However, every so often the backend server will suddenly jump to 100% CPU and stay there. The server itself continues to run just fine even though it starts maxing out a CPU core. Restarting Nginx causes all connections to drop to the backend server. At the same time, the backend server segfaults. The OS then auto-restarts the background server service.

I've written many non-PECL ev services over the years and they run without any issues other than they don't scale very well for handling thousands of simultaneous connections.

I realize this is an extremely vague bug report. It is hard to replicate (I don't even know the circumstances under which it can happen and it only occasionally happens) and I don't know if it is something buried somewhere in PHP itself, PECL ev, libev, or a problem with the OS. Segfaulting indicates a lower layer than userland. The fact that the server continues to function even after it goes to 100% CPU and only segfaults when Nginx goes away makes the problem even more bizarre. But maybe that combination will provide a hint to you as to a possible cause.

Comments (1)