Don't show icon indicating I can remove myself from having perms on my own repo

Currently, in the admin page, I see little "-" signs next to my name for permissions, even though I cannot un-permission myself.