Issue #35 wontfix

High CPU Usage / Hang on System Time Change

Robert Walter
created an issue

Seem to have found a reproducable bug on an embedded arm project that we are working on that uses apscheduler for performing interval based logging operations.

The scheduler works exceedingly well, except under one abmornal during testing.

If during boot, the hardware clock for some reason resets due to bad battery, etc, the system time will be starting at 0 past epoch upon initial program run, when the apscheduler is configured and started with a interval based job ranging from 5 - 60 second intervals.

Once the user corrects the time, or the NTP daemon corrects the time, the apscheduler task hangs indefinately, and the CPU usage shoots to the moon.

We have reproduced this with a little as a basic python program with just the scheduler loaded and running a simple 5 second interval job printing a single line of text. We set the date to an arbitrary time in the past (epoch + 0) , start the program, and using a separate terminal, or via NTP, adjust the time to the current time / date. Immediately the program hangs, and CPU usage goes up.

I am going to start digging for the reason, but any help would be appreciated.

Thanks !

Comments (3)

  1. Robert Walter reporter

    I am starting to think that the get_run_times() is where the the problem lies. Albeit not a problem, it is just doing what programmed, but from the looks of it, a time warp from Epoch + 0 all the way to the present, on a 5 second interval, creates a loop length of 210 million itterations. Hence the CPU usage / hang.

    Starting to consider a workaround, or any guidance would be appreciated.

  2. Alex Grönholm repo owner

    This is not the first time someone has mentioned this, but it was only a potential problem then -- nobody had actually run into it yet. This is a hard problem in computing. Trouble is, I'm not sure how the scheduler should react to changes to system time. I think it already handles daylight savings time somewhat gracefully, but how can it correctly schedule tasks if it can't rely on system time?

    As for get_run_times, the event system was specified so that each missed run would generate one misfire event. If the system clock goes to 0, there would be a lot of those events! I wonder if there's a better way around this?

  3. Log in to comment