I am sorry about this. We had a bug that allowed you to lose access to a repository and we were not removing the following relationship. This is a bug, it should not be allowed to happen. This is why there is no way for you to unfollow a repository you can not see.
We are going to run a script that will fix up all these broken relationships, this will fix the problem for you.