I'm putting this on hold until upstream fixes removing rules in a predictable fashion. It’s too easy to delete the wrong rule when trying to guess the correct handle to delete on someone else's system.
the solution is to use the correct function of nft. firstly, just like iptables, separate chains are supported so that one can be assigned to a firewall generator process like sshguard. secondly, for cases like this, instead of adding individual rules which is very inefficient at evaluation time, a set should be used, ipset or nft set.
edit: it seems I was misled by the (very?) out-of-date documentation. the point is that nft (now?) supports sets.