history data inconsistent vs data padding

[dd1]

It looks like the midas history system has problems with data padding:

• log_system_history() in mlogger writes “/history/links” history, it calls hs_write_event() with a data buffer that does not have any padding, data size is the sum of individual link items.
• log_history() in mlogger writes “/equipment/xxx/variables” history, it calls hs_write_event() with a data buffer filled by db_get_record(), where each data item is padded per c++ padding rules for struct data members. data size includes the struct size padding (see bug #273).

Obviously the two data formats are not compatible (one has padded data, the other has unpadded data).

When reading from history:

• mhdump.cxx assumes padded data, not clear if padding rules are same as db_get_record()
• history.cxx (“MIDAS” history) seems to assume unpadded data, hs_read() computes the offset by adding size of all tags, without padding.
• history_schema.cxx (MySQL, Sqlite and “FILE” history) assumes unpadded data.

So the best I can tell, mlogger writes padded data, all readers assume unpadded data. How did all this ever work?!? I will need to debug it some more.

K.O.

[dd1]

In “per-variable-history” mode, each variable is written into it’s own history event and there is no padding of data - all history events are individual variables or arrays.

In this mode, reading and writing is consistent and there is no problem.

Most likely I was using “per-variable-history” mode when writing history_schema.cxx and did not notice that mlogger can give me padded data (which would not be handled correctly).

K.O.

[dd1]

For “/history/links” there is no equivalent “per-variable-history” mode, so either add it or change the data size from value computed by db_get_record_size() (which includes padding) to total_size (no padding).

This will break mhdump (size with padding and padded data), but fix bug #194.

K.O.

‌

[dd1]

The older (no longer used) ODBC (MYSQL) history only made sense in “per-variable-history” mode, where this problem does not exist.

Perhaps that is why I did not notice this problem when I wrote the ODBC (Mysql) history.

K.O.

[dd1]

It looks to me that this problem is not easy to fix, but we have two workarounds:

• always use per-variable history (then there is no padding)
• watch for the “data size mismatch” errors in mlogger and rearrange the data to avoid any padding.

The best I can tell, in all cases where there is a data mismatch caused by padding, history_schema will output cm_msg(MERROR) reporting a data size mismatch and we will not accidentally write incorrect history data.

In the old “MIDAS” history code, there is no message about size mismatch, but correct (padded) data will be written into the .hst history file, and we can fix the reading code later if needed. (again, this setup has worked for years, I am not sure why we did not see this problem before).

K.O.

‌