High CPU Usage / Hang on System Time Change
Seem to have found a reproducable bug on an embedded arm project that we are working on that uses apscheduler for performing interval based logging operations.
The scheduler works exceedingly well, except under one abmornal during testing.
If during boot, the hardware clock for some reason resets due to bad battery, etc, the system time will be starting at 0 past epoch upon initial program run, when the apscheduler is configured and started with a interval based job ranging from 5 - 60 second intervals.
Once the user corrects the time, or the NTP daemon corrects the time, the apscheduler task hangs indefinately, and the CPU usage shoots to the moon.
We have reproduced this with a little as a basic python program with just the scheduler loaded and running a simple 5 second interval job printing a single line of text. We set the date to an arbitrary time in the past (epoch + 0) , start the program, and using a separate terminal, or via NTP, adjust the time to the current time / date. Immediately the program hangs, and CPU usage goes up.
I am going to start digging for the reason, but any help would be appreciated.