Monitoring failure in linux daemon update

[SzV]

There was a security update to sshd, the binary is deleted and recreated during the package update. Monit sent this in the email: permission test failed for /usr/lib/openssh/sftp-server -- current permission is 0000

monit monitor sshd didn't help, the only solution is service monit restart.

What could I do to make monit handle such an update by itself?

[Tildeslash]

which monit version it is? (monit -V)

can you send a monit log to [e-mail]?

[SzV]

version 5.10

sent.

[SzV]

Start looking at [CET Jan 15 06:39:34] debug : 'sshd_bin' file exists

[Tildeslash]

Please can you post the configuration of the sftp_bin service? Which action do you use if the checksum failed?

It seems that the monitoring was disabled after the checksum failure (most probably by unmonitor or stop action), which resets the file informations, so all tests after the one which disabled the monitoring will work with zero values ... we'll fix.

[SzV]

 check process sshd with pidfile /var/run/sshd.pid
   group system
   group sshd
   start program = "/etc/init.d/ssh start"
   stop  program = "/etc/init.d/ssh stop"
   if failed host localhost port 22 with proto ssh then restart
   if 5 restarts with 5 cycles then timeout
   depend on sshd_bin
   depend on sftp_bin
   depend on sshd_rc
   depend on sshd_rsa_key
   depend on sshd_dsa_key

 check file sshd_bin with path /usr/sbin/sshd
   group sshd
   include /etc/monit/templates/rootbin

 check file sftp_bin with path /usr/lib/openssh/sftp-server
   group sshd
   include /etc/monit/templates/rootbin

 check file sshd_rsa_key with path /etc/ssh/ssh_host_rsa_key
   group sshd
   include /etc/monit/templates/rootstrict

 check file sshd_dsa_key with path /etc/ssh/ssh_host_dsa_key
   group sshd
   include /etc/monit/templates/rootstrict

 check file sshd_rc with path /etc/ssh/sshd_config
   group sshd
   include /etc/monit/templates/rootrc

[SzV]

I suspect monit holds on to the old - deleted - inode on the filesystem.

[Tildeslash]

Please can you post content of /etc/monit/templates/rootbin? (that file most probably contains the checksum test and action on failure.

The problem is not related to old binary's inode, it is really internal stuff (reset of service data after unmonitor/stop action class) - we have reproduced the problem with unmonitor action, just need to be sure which action you use so we can test the same.

[SzV]

 if failed checksum       then unmonitor
 if failed permission 755 then unmonitor
 if failed uid root       then unmonitor
 if failed gid root       then unmonitor

[Tildeslash]

thanks for data, working on fix

[SzV]

Thank you for your work.

[Tildeslash]

Fix Issue #315: If some file, filesystem, directory or fifo test triggered unmonitor, stop or restart, the tests which followed the one which performed that action may fail as the collected data were reset to zero during unmonitor/stop/restart action. The status also showed wrong values for uid, gid, permission and timestamp during that cycle.

→ <<cset fb5f579000f6>>

[SzV]

Thank you! I've just packaged monit-5.16.deb for testing.

[SzV]

@tildeslash OK, It works now. Is there a command to issue after a service update when the checksum changes? The only way I know of is restarting monit.

[Tildeslash]

You can use "if changed checksum", instead of "if failed checksum":

if changed checksum then alert

It will notify you whenever the checksum changed, no need for monit reload.

[SzV]

OK. I don't want it to automatically accept changed checksums.

Is there a way to manually accept changed checksums?

[Tildeslash]

If you use the "if failed checksum", the value is initialised on Monit start/reload only - currently the only way is to reload monit.

[SzV]

Thank you! It is very nice of you that you tell it.

[SzV]

Please consider a solution for package upgrades. Thanks again.

[SzV]

My solution is to stop monit during packages upgrades.

    cat > /etc/apt/apt.conf.d/05monit <<"EOF"
DPkg::Pre-Invoke { "[ -x /usr/bin/monit ] && /etc/init.d/monit stop" };
DPkg::Post-Invoke { "[ -x /usr/bin/monit ] && /etc/init.d/monit start" };
EOF