Allow Tokens with Multiple Parents

Issue #1466 new
Glen Fletcher
created an issue

The reason for this is best explained by example, lets say we have a lexer which treats single and double quote string differently so "Hello" end up being Token.String.Double, while 'Hello' is Token.String.Single, then both types of string support some form of escape sequences which should also be a child of Token.String.Escape.

Currently we have the option of using: Token.String.Single.Escape Token.String.Double.Escape or Token.String.Escape.Single Token.String.Escape.Double

However this means that either

DoubleQuotedEscapeSequence in Token.String.Double


DoubleQuotedEscapeSequence in Token.String.Escape

will fail.

Solutions replace parent with a tuple parents, and use subtypes to check for contains parent can be conditionally mapper to parents[0] given the length is exactly 1.

This way We could create Token.String.(Double, Escape) as Token.String.DoubleEscape

Consideration for Styles

Token.String.(Double, Escape) would take Token.String.Double style, then apply integrate changes since last common parent from Token.String.Escape, finally looking at Token.String.DoubleEscape

For child creation add the function child to the TokenType i.e. Token.String.child("Double", "Escape") = Token.String.(Double, Escape) if the token isn't defined it should create it other wise return it, but not create a matching attribute unless it only has a single argument.

You also need a module level function child which would be invoked as child("Token.String.DoubleEscape", Token.String.Double, Token.String.Escape) which would create a multi parented token, and then build the appropriate attributes to define it

Comments (1)

  1. Glen Fletcher reporter

    Here a way of creating such Tokens, this method also makes each token a type, which can be instanced with a value, (Note I wrote this for a separate lexer as part of a compiler, and modified this to match the behavior expected by pygments) it could also be implemented with token types as objects like pygments currently dose, on the other hand having tokens be a type could be useful for future extensions.

    Note this method relies on pythons MRO for multiple inheritance, but this would be manually implemented quite easily

  2. Log in to comment