history display stuck in "updating data"

[dd1]

there is a failure mode in the history display page, it can enter the “updating data” state and get stuck there.

as I understand, the page will only exit the “updating data” state when an rpc reply is received from mhttpd, if this reply never arrives (there is a network error, there is an unrelated or related mhttpd crash), the page is stuck, the only way to exit is to reload the page (losing one’s place in history time and space).

I have now seen this failure mode often enough that I think we should do something about it. Maybe have a timeout? Maybe add a “cancel load of new data” button?

Also to avoid very long history delays, I still think we should limit the amount of data we ask from mhttpd. For example, if I look at a history plot on a 1 year time scale and accidentally push the “-” (zoom out) button, I think currently the web page will ask mhttpd to read 1 year worth of data. I think we should limit this to 1 month, 1 week, 1 day, whatever.

This may indirectly help with the problem of “stuck in ‘updating data’”

K.O.

[Stefan Ritt]

We discussed RPC handling several times, and you promised to do a retry on a timeout:

https://bitbucket.org/tmidas/midas/issues/209/get-rid-of-mjsonrpc-dialogs-put-it-to-the

If you implement that, this error should be fixed as well.

[Stefan Ritt]

For the long history delays, we should check if there is a bug or if it’s only the slow reading on the mhttpd side. If it’s the latter, then there is nothing we can do. Just don’t press the “-” button. If you shoot yourself in the foot, it hurts, nothing we can do. In the MEG experiment I can easily go back one year without problem, but that’s maybe it a fast machine with lots of memory, so probably the whole year of data can be put in the disk cache (which anyhow is a SSD). Well, to be hones, I could go back, now it’s not possible any more. Going over a certain point, I receive a “abort” from HsFileSchema::read_data:

#2  0x00000000004fbd45 in HsFileSchema::read_data (this=0x7fffe457b870, start_time=1580703094, 

end_time=1584392142, num_var=5, var_schema_index=0x7fffd8000a50, var_index=<optimized out>, 

debug=0, buffer=0x7fffd8001b60) at /home/meg/midas/src/history_schema.cxx:2206

Looking at the code, I read “FIXME!!!”. Any idea?

[Stefan Ritt]

I have some suspicion there: The old history page used decimation for reading lots of history data, like every 10th point, every 100th point etc. This way of course you can overlook some spike. Now I’m reading all points, which might be the reason why things are slower. To verify this, can you please put some debug printout in your history read functions to see how long they take?

[dd1]

I think we did implement the json rpc for binned data. This returns the mean, the rms, the smallest value and the biggest value for each bin. The old history plots using raster “gd” graphics were too coarse and did not show the smallest and biggest value, so the spikes were lost. The new canvas graphics use the monitor native resolution, so we can plot them and the spikes will not disappear. K.O.

‌

[Stefan Ritt]

Yes but I do not use the binned data right now. The reason is that it’s a nightmare to manage. If you open a window, you have unbinned data. Then you press the minus button, get some more unbinned data, then you switch to binned data. Now a single array of x/y value does not do the job any more. You need to maintain a hybrid of binned and un-binned data. Even worse if one zooms in to the binned data, you have to revert to unbinned data at some point, discard the binned data etc. So that’s why I didn’t bother implementing it. I thought 8 seconds for 3 years of data is not bad for unbinned data, but I did not expect that on your system it’s so much slower.

[dd1]

(I fixed the crash yesterday - it was caused by the renaming of midas data types)

[dd1]

with the latest improvements, I do not see the history pages getting stuck in “updating data”, closing this bug. K.O.

[dd1]

problem no longer happens. K.O.