The C lexer (and possibly others) creates an empty (Token.Text, u'') token in order to move from the 'root' state to the 'statement' state. Instead of this workaround, we can allow states to specify a default state transition to do.
Updated based on review comments
Conflict resolved. I'd love to get this in soon. I can see at lease 146 lines where I would apply this.
Sure, me too. I need a little time to test it... in the meantime could you add a test that confirms multiple states, like default('a', 'b') works?
I didn't throw the exception due to the check in _process_token
Great -- I merged to avoid future conflicts while I figure out whether regexlint needs changes now.