history plot requests 720 hours of data

[dd1]

history plot links in bug 286 do not load, mhttpd is using 100% CPU, everything times out. This is because javascript history plot is requesting 720 hours of data instead of only data between the A and B timestamps (about 2 hours in the first URL). K.O.

[Stefan Ritt]

This is a current limitation of the history code. The code supports adding old data to the left side, and new data to the very right side, but nothing “in the middle”. So if you request A to B, then scroll to the right, history needs to be added after B, which is simply not implemented. A quick fix would be to prohibit any scrolling if you start with a A-to-B window, but maybe that’s not what people will like. To implement the “loading after B”, it would require quite some work. I remember I started that when I implemented the new history, but then I gave up after some time because it was terribly slow. Adding array values at the beginning of an array in JS is much slower than adding it at the end. For a handful of recent values that works, but for long arrays that again can take minutes.

[dd1]

there is no scrolling in this case, I am just trying to open the test URLs from bug 286 to confirm the problem was fixed. the first URL should load A-to-B data, that spans 2 hours (unless I miscalculated). Instead load times out and in the debugger I see mhttpd doing an hs_read() of 720 hours (and that takes forever). I will adjust the debug levels in hs_read() to make it easier to see, but if you print hs_read() start time and end time, they are not same as A and B in the URL. K.O.

‌

[Stefan Ritt]

You didn’t get my point above: What you observer is correct, this is the way the code is implemented right now. If you request A-B, it reads A-today. Reading only A-B needs lots of additional work. Not for the reading, but for the internal array handling. Please read again by comment above. This also happens if you DO NOT DO ANY SCROLLING.

[dd1]

hmm… this is very bad. to look at 1 hour of history 2 years ago, we have to load 2 years worth of data?

but I am pretty sure this is not what I see. the history read request end time is not the current time. I will reconfirm this.

[Stefan Ritt]

Ok, I will look again into this.

[Stefan Ritt]

Ok, I looked at it and now I remember what we have.

The history code is not made for having “holes” in it. A “hole” would be if you have a 1 hour data 2 years ago, then there would be a hole of 2 years between that 1h period and today. The reason for that is too complicated to be described here, but to “fix” it would take me several days of solid work. So what the code does is to request all history for the last 2 years. I understand that this is not good, but that’s what we have at the moment. In order not to create dead systems, the code limits a single request to maximum 1 month. So what happens if you request a period 2 years ago is you will see 24 requests of one month each. This was probably what confused you. I added some debug output in the history code and tried a 1 day period some 1 year ago, and this is what I got (I executed the code on Apr 6 2022):

Initial read:Mon Mar 07 2022 09:30:54 GMT+0100 (Central European Standard Time) to Wed Apr 06 2022 10:30:54 GMT+0200 (Central European Summer Time)
loadOldData:Sat Feb 05 2022 09:30:54 GMT+0100 (Central European Standard Time) to Mon Mar 07 2022 09:30:54 GMT+0100 (Central European Standard Time)
loadOldData:Thu Jan 06 2022 09:30:54 GMT+0100 (Central European Standard Time) to Sat Feb 05 2022 09:30:54 GMT+0100 (Central European Standard Time)
loadOldData:Tue Dec 07 2021 09:30:54 GMT+0100 (Central European Standard Time) to Thu Jan 06 2022 09:30:54 GMT+0100 (Central European Standard Time)
loadOldData:Sun Nov 07 2021 09:30:54 GMT+0100 (Central European Standard Time) to Tue Dec 07 2021 09:30:54 GMT+0100 (Central European Standard Time)
loadOldData:Fri Oct 08 2021 10:30:54 GMT+0200 (Central European Summer Time) to Sun Nov 07 2021 09:30:54 GMT+0100 (Central European Standard Time)
loadOldData:Wed Sep 08 2021 10:30:54 GMT+0200 (Central European Summer Time) to Fri Oct 08 2021 10:30:54 GMT+0200 (Central European Summer Time)
loadOldData:Mon Aug 09 2021 10:30:54 GMT+0200 (Central European Summer Time) to Wed Sep 08 2021 10:30:54 GMT+0200 (Central European Summer Time)
loadOldData:Sat Jul 10 2021 10:30:54 GMT+0200 (Central European Summer Time) to Mon Aug 09 2021 10:30:54 GMT+0200 (Central European Summer Time)
loadOldData:Thu Jun 10 2021 10:30:54 GMT+0200 (Central European Summer Time) to Sat Jul 10 2021 10:30:54 GMT+0200 (Central European Summer Time)
loadOldData:Tue May 11 2021 10:30:54 GMT+0200 (Central European Summer Time) to Thu Jun 10 2021 10:30:54 GMT+0200 (Central European Summer Time)
loadOldData:Sun Apr 11 2021 10:30:54 GMT+0200 (Central European Summer Time) to Tue May 11 2021 10:30:54 GMT+0200 (Central European Summer Time)
loadOldData:Fri Mar 12 2021 09:30:54 GMT+0100 (Central European Standard Time) to Sun Apr 11 2021 10:30:54 GMT+0200 (Central European Summer Time)
loadOldData:Wed Feb 10 2021 09:30:54 GMT+0100 (Central European Standard Time) to Fri Mar 12 2021 09:30:54 GMT+0100 (Central European Standard Time)
loadOldData:Mon Jan 11 2021 09:30:54 GMT+0100 (Central European Standard Time) to Wed Feb 10 2021 09:30:54 GMT+0100 (Central European Standard Time)
loadOldData:Thu Dec 31 2020 00:00:00 GMT+0100 (Central European Standard Time) to Mon Jan 11 2021 09:30:54 GMT+0100 (Central European Standard Time)

I’m happy to put the request “only load A to B” on my todo list, but I won’t have several days of time right now to implement that.