log_history_periodic() doesn't account for run being stopped / program being stopped

[Ben Smith]

I have many equipments that update history variables periodically during a run. Previously the history DB would only get updated if the frontend was running and a run was in progress. However, it seems that the new log_history_periodic() business with minimum and maximum logging periods does not account for either of these factors, and the history DB is being "updated" with stale values even if the frontend is stopped. This is significantly bloating the history DB with useless/misleading data.

For all of my frontends, the old logic of only using the hotlink made total sense - when the FE updates the variables, the new values get written to the DB. Even if the values haven’t really changed during the run, the FE still re-writes the values and the hotlink gets triggered. We never get any big gaps in the history during a run, as we’re always updating every 10s or so.

For this very common use case, the log_history_periodic() is redundant during a run (as it’s already a periodic equipment) and broken the rest of the time.

I’m not sure what the best solution is. Perhaps max_period could be set to a very large number (instead of Common/Period) if the equipment is already periodic? If that doesn’t work, then I feel like the whole log_history_periodic() concept may need to be revisited.

[dd1]

I reverted this change to mlogger for midas-2020-12 as I did not have time to understand what it is doing. I have Stefan’s explanation of his intent, but have been busy with the frontend code to digest it. while testing the frontend code, I confirm the current mlogger is broken - it is writing history data for frontends that are not running. K.O.

[dd1]

cannot do next release of midas until this is resolved. K.O.

[Stefan Ritt]

I had a look at things and confirm it’s broken. But before I do any change, let’s agree on a scheme everybody agrees on. My idea now is to:

• Create a separate branch “historyPeriod” not to screw up running experiments.
• Drop “max_period” (which is derived from “Common/Period”) completely. Currently, if “Period” is 100 (in [ms]), this means that the logger logs a history record once every second, flooding the history database. I agree with Ben that a record should only be logged if the ODB value changes. It’s the responsibility of the frontend when to update the ODB values.
• Keep “min_period” and keep it derived from “Common/Log history”. If “Log history” = 0, do not log any history, if “Log history” = 1, log history whenever there is a change in the ODB. If “Log history = n” (n > 1), then log history when there is a change in the ODB, but not more often than “n” seconds. Like if n=60, log at most once record per minute. This is to avoid “noisy” equipment which fluctuates a lot, but does not really change. Most people will have “Log history = 1”, so the behaviour is close to what we had before. Let me remind you that the maximum logging rate before was 10 seconds, and we could not see changes faster than 10 seconds. That’s why I started the whole business of reworking the history.
• Announce the change behavior in the Forum before we merge the new branch.

Please comment to this before this weekend. If I hear no comments, I will start implementing this next week.

Best,
Stefan

‌

[Ben Smith]

That sounds like a good solution to me. “Common/Log history” is a signed int, so we could also consider 0 = no logging, positive = log at most once per N seconds, negative = log whenever there is a change (no limit). That way 1 isn’t treated as a special value. I don’t have a strong preference either way.

[Stefan Ritt]

I’m working now to implement his as dicussed.

[Stefan Ritt]

I have now time to revisit that issue. Ben proposes a negative number to write history when it changes. I think that’s not a good idea. Even if the ‘log history’ is positive, data should be logged only if it changes in the ODB, so a negative value is not necessary. So the final behaviour as I implemented it today is:

• Common/Log history = 0 means no history logging
• Common/Log history = 1 means log whenever the value changed in the ODB
• Common/Log history = N means log whenever the value changed in the ODB and the previous write was more than N seconds ago

So most experiments should be happy with 0 or 1. Only experiments which have fluctuating values due to noisy sensors might benefit from a value larger than 1 to limit the history logging. Anyhow this is not the preferred way to limit history logging. This should be done by the front-end limiting the updates to the ODB. Most of the midas slow control drivers have a “threshold” value. Only if the input changes by more then the threshold are written to the ODB. This allows a per-channel “dead band” and not a per-event limit on history logging as ‘log history’ would do.

‌

[Stefan Ritt]

Please everybody have a look at the current situation and close this case if everything is fine.

[Thomas Lindner]

Stefan: I looked at your latest proposal and it seems reasonable to me. I agree with going ahead with this proposal.

[Stefan Ritt]

The scheme has been implemented last October and seems to run fine at PSI since then. If you can confirm that, we can close this issue.