“CLF Probes” blocks Googlebot:
example.com-access_log:188.8.131.52 - - [16/Aug/2020:15:57:36 +0300] "GET /index.php?option=com_content&view=section&layout=blog&id=12&Itemid=81 HTTP/1.1" 404 794 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.92 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" (0.047)
Here is how we can verify an IP is from Googlebot:
I can modify /usr/local/libexec/sshg-fw-ipfw to not block Googlebot, but then in my system logs file I will still see these entries:
Blocking "184.108.40.206/32" for 240 secs (3 attacks in 1712 secs, after 2 abuses over 2785 secs.)
Also this file will be replaced in the next sshguard upgrade.
I think the best solution is to add a feature to sshguard, add an option to sshguard.conf ( dynamicwhitelist = /usr/local/etc/sshguard.whitelist.sh ) and then before a block it executes:
The shell script we will get the IP, does it stuff and if it returns 0 then sshguard blocks the IP and if it returns 1 to not block it.