One of the ways that locking might fail is that there is a DF conflict between two refs (e.g., you want to lock "foo/bar" but "foo" already exists). In this case, we return an error, but there is no way for the caller to know the specific problem.
This patch sets errno to ENOTDIR, which is the most sensible code. It's what we would see if the refs were stored purely in the filesystem (but these days we must check the namespace manually due to packed refs).
Signed-off-by: Jeff King <firstname.lastname@example.org> Signed-off-by: Junio C Hamano <email@example.com>