slow "FILE" history loading from ZFS
in the agmini system, we see very slow run start times - 30 sec or more - where mlogger takes that much time to initialize the history system. this is caused by reading the headers of all FILE-type history files. When these files are in the ZFS cache or when they are on SSD storage, run start is almost instantaneous. But reading the headers from HDD storage takes a long time. In the agmini system there are about 5000 history files.
to fix this, I propose to add a cache file for history file headers.
K.O.
Comments (2)
-
reporter -
reporter commit 20a923226196f3cb7e63caa98d4a746f53293625 implements incremental loading of FILE history schema. I see mlogger start time is much reduced - in agmini we read 100 history files instead of all the existing 9000 files.
“ls -ltr” on the history directory is still there, nothing I can do about it…
also disable the watchdog while we go reading history files, it can take an arbitrary amount of time.
K.O.
- Log in to comment
slightly better understanding of ZFS cache.
Normal data access by the FILE history will not cause history data to migrate to the MFU cache. But if we setup a cron job (or a thread) to keep accessing history files, it will migrate to the MFU cache and everything is fast afterwards. not sure how often we need to access the data to prevent it’s expiration.
Of course mlogger writing bulk data to ZFS storage will case ZFS cache churn and will expire useful data…
Other solution is to read fewer history files:
K.O.