FILE history non-monotonic time
Issue #350
resolved
FILE history reader complains about non-monotonically increasing time in history files. this already happened 2-3 times in Stefan’s experiments. need some way to automatically repair or recover from this. example bad file in email from Stefan, subject “corrupt history file”. K.O.
Comments (3)
-
reporter
-
reporter
bug was induced by anomaly in MEG history files:
daqubuntu:midas$ ls -l *.dat -rw------- 1 olchansk users 986332 Aug 16 14:35 mhf_1449770978_20151210_hv.dat -rw------- 1 olchansk users 3057516 Aug 16 14:35 mhf_1650463772_20220420_environment.dat -rw------- 1 olchansk users 1828352 Aug 16 14:35 mhf_1654869958_20220610_environment.dat -rw------- 1 olchansk users 2176 Aug 16 14:35 mhf_1655714089_20220620_environment.dat -rw-r--r-- 1 olchansk users 4459520 Aug 16 14:35 mhf_1655714180_20220620_environment.dat -rw------- 1 olchansk users 2176 Aug 16 14:35 mhf_1657735410_20220713_environment.dat -rw------- 1 olchansk users 2176 Aug 16 14:35 mhf_1657735644_20220713_environment.dat -rw------- 1 olchansk users 2688 Aug 16 14:35 mhf_1657736152_20220713_environment.dat -rw------- 1 olchansk users 962304 Aug 16 14:35 mhf_1657736713_20220713_environment.dat daqubuntu:midas$some files have:
daqubuntu:midas$ more mhf_1657735410_20220713_environment.dat version: 2.0 event_name: Environment time: 1657735410 tag: /DWORD 1 4 /timestamp tag: FLOAT 1 4 Air conditioner temp. tag: FLOAT 1 4 Rack 0 temp. tag: FLOAT 1 4 Rack 1 temp. tag: FLOAT 1 4 Rack 2 temp. tag: FLOAT 1 4 Rack 3 temp. tag: FLOAT 1 4 Computer temp. tag: FLOAT 1 4 BTS Supply temp. tag: FLOAT 1 4 Laser temp. tag: FLOAT 1 4 Temp inside hut tag: FLOAT 1 4 Temp outside hut tag: FLOAT 1 4 Humidity hut tag: FLOAT 1 4 Input_11 tag: FLOAT 1 4 Input_12 tag: FLOAT 1 4 Input_13 tag: FLOAT 1 4 Input_14 tag: FLOAT 1 4 Input_15 tag: FLOAT 1 4 Input_16 tag: FLOAT 1 4 Input_17 tag: FLOAT 1 4 Pump-WLD1 tag: FLOAT 1 4 Pump-WLD2 tag: FLOAT 1 4 Pump-WLD3 tag: FLOAT 1 4 NC Magnet-WLD1 tag: FLOAT 1 4 NC Magnet-WLD2 tag: FLOAT 1 4 NC Magnet-WLD3 tag: FLOAT 1 4 Pump-T1 tag: FLOAT 1 4 Pump-T2 tag: FLOAT 1 4 Pump-T3 tag: FLOAT 1 4 Pump-T4 tag: FLOAT 1 4 NC Magent-T1 tag: FLOAT 1 4 NC Magent-T2 tag: FLOAT 1 4 NC Magent-T3 record_size: 128 data_offset: 2048while other files have:
daqubuntu:midas$ more mhf_1655714180_20220620_environment.dat version: 2.0 event_name: Environment time: 1655714180 tag: /DWORD 1 4 /timestamp tag: FLOAT 1 4 Air conditioner temp. tag: FLOAT 1 4 Rack 0 temp. tag: FLOAT 1 4 Rack 1 temp. tag: FLOAT 1 4 Rack 2 temp. tag: FLOAT 1 4 Rack 3 temp. tag: FLOAT 1 4 Computer temp. tag: FLOAT 1 4 BTS Supply temp. tag: FLOAT 1 4 Laser temp. tag: FLOAT 1 4 Temp inside hut tag: FLOAT 1 4 Temp outside hut tag: FLOAT 1 4 Humidity hut tag: FLOAT 1 4 Valve tag: FLOAT 1 4 Air Temp tag: FLOAT 1 4 Req. Air Temp tag: FLOAT 1 4 Hut Temp tag: FLOAT 1 4 Req Hut Temp tag: FLOAT 1 4 LM35 Temp tag: FLOAT 1 4 Status tag: FLOAT 1 4 Pump-WLD1 tag: FLOAT 1 4 Pump-WLD2 tag: FLOAT 1 4 Pump-WLD3 tag: FLOAT 1 4 NC Magnet-WLD1 tag: FLOAT 1 4 NC Magnet-WLD2 tag: FLOAT 1 4 NC Magnet-WLD3 tag: FLOAT 1 4 Pump-T1 tag: FLOAT 1 4 Pump-T2 tag: FLOAT 1 4 Pump-T3 tag: FLOAT 1 4 Pump-T4 tag: FLOAT 1 4 NC Magent-T1 tag: FLOAT 1 4 NC Magent-T2 tag: FLOAT 1 4 NC Magent-T3 record_size: 128 data_offset: 2048observe how some variables, i.e. “Hut Temp” and “Req Hut Temp” are sometimes present, sometimes absent (replaced by “
Input_11” & co) -
reporter
- changed status to resolved
fixed. K.O.
- Log in to comment
“Detected duplicate or non-monotonous data" when reading midas history fixed commit 9893f85ebe33e96cc63f501a0f89e1f8932c894d.
Problem was in the history reader: it was not reading history files in chronological order and so was tripping protections against non-monotonous data returned to user.
Fix was to reverse the order of two loops. Correct code has the loop over data files (schemas) first, then for each file, loop over variables. Data files (schemas) are always in chronological order, and result is always in chronological order. Old incorrect code looped of variables and chronological order of data files was not enforced.
Additional protections have been installed: timestamps inside the data file must be consistent with the file header (range between time in the current header and time in the header of next data file), and have to monotonically increase inside the file.