Parsertools is a library meant to aid in construction of hand-written parsers. The library is based on the paper "Top Down Operator Precedence" written by Vaughan Pratt. Though author hasn't read the paper but only various texts available free online that explains the technique. (Simple Top-Down Parsing in Python)
Overall structure of the library is formed by 'Token' and 'SourceLine' classes. Every piece in the library processes them in some way. SourceLine contains token line position, which allows user to pinpoint any error in the source file if necessary. Token represents an essential datatype both a tokenizer and grammar must handle.
- Token instantiation example for lazy people:
- token = Token(name='eof', value=None, sourceline=None, column=0)
tokenize and layout are mostly useless pieces and they can be thought as somewhat badly written examples about providing the parser with input to process from a source file. Their source code contains better documentation about their behavior.
Major help the parsertools library provides are the Grammar and Stream, which are meant to be used together. Grammar abstracts rules for parsing operator precedence and expressions. Pass a grammar and token generator for Stream, you get lookahead stream back.