state file update blocks on frozen filesystem

[Tildeslash]

When the filesystem where monit writes the statefile freezes, monit blocks on state file update (write) and stops monitoring the system until the filesystem is r/w again.

We should change the statefile machinery to be non-blocking with write timeout, so monit can continue.

[Tildeslash]

tested with non-blocking I/O, doesn't solve the problem ... even if the file is opened as nonblocking or switched to non-blocking using fcntl(), if the filesystem where monit state file is placed is frozen, the write() will block.

To freeze:

fsfreeze -f /<fs_with_monit_state_file>

To unfreeze:

fsfreeze -u /<fs_with_monit_state_file>

We can probably switch the statefile update to standalone thread, which won't stop the testing if it gets blocked by frozen filesystem.

Workaround: place the statefile to a filesystem which cannot freeze (use "set statefile <path>").