slow update of history plots

[dd1]

Switching from “last 10 min” to “last 7 days” takes a very long time. The old history plot does this pretty much instantaneously, so the problem is not slow access to history files. The new history plots sits there for a long time with a message “updating data…”. (if I go to a different tab, i.e. to type this report, when I return to the history tab, I get a blank page, this is issue 202). I think it would be good if the “updating data…” message included some additional information of the progress of loading the data.

https://daq16.triumf.ca/?cmd=History&group=PWB&panel=http_time

K.O.

[Stefan Ritt]

Strange. If I try on MEG, even from home, it takes about ~1 sec. That means the browser part is fast (MacBook Pro), and it must be related to the slow access of the history files. I learned that the available memory on the backend machine is crucial. If 7 days of history fit in the hard disk cache, things are quick. But if you a low on memory and the OS reduces the disk cache, you have to read directly from the hard disk which takes time. Our new machines all have SSD disks to mitigate this effect.

[dd1]

Slow read of history files is not the problem. The old history plots are generated instantaneously. K.O.

[dd1]

See browser page consume 100% while sitting in “updating data…”. K.O.

[dd1]

js profiler shows 100% CPU use in receiveData(). K.O.

[dd1]

In receiveData(), added console.log() and I see my nVars is 10, my nData is 198000 for each variable. the function is making progress over the variables, but not very quickly. K.O.

[dd1]

The problem is with array.unshift(), it appears to be very slow. If I replace it with array.push(), “last 7 days” loads instantaneously (the plot is wrong, of source). I read this: https://stackoverflow.com/questions/12250697/time-complexity-of-unshift-vs-push-in-javascript

Maybe we can rewrite receiveData() to avoid using unshift()? Maybe put the new data into a temporary array, then prepend this array in front of the real array (1 unshift operation bs 198000 unshift operations). K.O.

[dd1]

This code also seems to be fast (not sure if it is correct, the plot looks roughly right)

var tt      var tt = new Array(nData); 
            var vv = new Array(nData); 
            for (let j = 0; j < nData; j++) { 
               let v = array[i--]; 
               let t = array[i--]; 
               if (this.data[index].time.length === 0) { 
                  this.data[index].time.push(t); 
                  this.data[index].value.push(v); 
               } 
               if (t < this.data[index].time[0]) { 
                 //this.data[index].time.unshift(t); 
                 //this.data[index].value.unshift(v); 
                 tt[nData-j-1] = t; 
                 vv[nData-j-1] = v; 
                 //this.data[index].time.push(t); 
                 //this.data[index].value.push(v); 
               } 
            } 
            this.data[index].time = tt.concat(this.data[index].time); 
            this.data[index].value = vv.concat(this.data[index].value); 

‌

[Stefan Ritt]

Thanks for pointing me to unshift(). Indeed this is the CPU hog. I took your code and slightly modified and committed it. Give it a try and close the issue if you’re satisfied.

[dd1]

“last 7 days” now loads reasonably quickly. good enough for now. some day I will look to see if the delay is in javascript or in data transfer from daq16.triumf.ca to my home. K.O.

[dd1]

fixed midas-2019-09-e