List priority - Doesn't test everything

Issue #35 new
created an issue

I've recently added a small class in my code - SimpleVar -which makes it:

class IntLiteral(str):
    grammar = number

class StringLiteral(str):
    grammar = string_re

literal = [IntLiteral, StringLiteral]

class SimpleVar(str):
    grammar = word

class LetInExp():
    grammar = K("let"), endl,\
            indent(attr("decs", maybe_some(dec))), endl,\
            K("in"), endl,\
            indent(attr("exps", maybe_some(exp, endl))), endl,\

class FunctionCall():
    call_parameter = [literal]
    call_parameters = optional(csl(call_parameter))

    grammar = name(), '(', attr("parms", call_parameters), ')'


class Code(List):
    grammar = exp


So, in the exp list, we have [LetInExp, SimpleVar, [IntLiteral, StringLiteral]] According to the documentation, they are tested in sequence.

A list instance which is not derived from pypeg2.Concat represents different options. They're tested in their sequence. The first option which parses is chosen, the others are not tested any more.

When I remove exp.append(SimpleVar) the expression let in end is correctly parsed.

When I don't, however, I get the following error: SyntaxError: expecting Keyword('end')

Shouldn't the parser try to go for the LetInExp before the SimpleVar ?

Comments (5)

  1. kino reporter

    Furthermore, I feel like it doesn't explore all the options.

    I've also got this.

    class FunctionCall():
        call_parameter = [literal, SimpleVar]
        call_parameters = optional(csl(call_parameter))
        grammar = name(), '(', attr("parms", call_parameters), ')'

    Let's say I'm trying to parse the following code: print(a)

    If FunctionCall is the first element of the list, it'll parse. If it isn't, I'll get a Syntax Error, missing keyword let for example.

    Why is that?

  2. John Kemp

    I appear to also have this issue. I have a list of options, including one that I know will match the string I'm parsing. However, the checks will fail 3 or 4 items into the list and throw an error of the form

    SyntaxError: expecting {whatever, corresponding to a rule halfway down the list}

    and never reach the one that does match. If I reduce the number of options in the list to only a few then it'll match successfully. Note that the error given isn't the one for no options matching, it's the one for only having a single rule that didn't match. If it truly was failing to match any items in the list then the error should have been

    SyntaxError: expecting one of [{whatever options}]

  3. John Kemp

    On further testing, I believe there's something else at play. Take the following test program:

    from __future__ import unicode_literals
    import pypeg2 as peg
    import re
    class String(peg.RegEx):
        grammar = re.compile(r'"[^"]*"')
    class Number(peg.RegEx):
        grammar = re.compile(r'[-+]?[0-9]*\.?[0-9]*f?')
    class Name(peg.RegEx):
        grammar = re.compile(r'\w+([\-/_]\w+)*')
    print peg.parse("SomeWords2MoreText", [String, Number, Name])

    Expected behaviour is that this will match a Name. However, it fails with:

    SyntaxError: expecting match on "[^"]*"

    But if Number is removed from the options then it will run successfully. Any thoughts on why a non-matching option would cause i) the remainder of the list to be ignored and ii) an error to be raised as though the previous option in the list was the only one supplied?

  4. Log in to comment