Add support for partial tree parsing
With the additional rules for matching invalid syntax, parsing is becoming too slow to be used in a real time (keystroke) context. Keystroke parsing is important for supporting context aware completion. E.g. knowing the user just typed an INCLUDEONCE statement, and therefore knowing to only offer completions of files, not other symbols. Also, offering Type aware completion - User adds a new variable, and tried to set its value. Currently completions of all types would be shown, whereas if the type of the variable was known (without having to save and reparse the entire file) archie could prioritize completions that match that variable's type.
The solution is to be something like this: 1. When no tree exists, do a full lex/parse. 2. Upon creating a new tree, walk the tree (or use XPATH) to create a model of the 'chunks' of code in the file - a chunk is just a subtree that, if it were in a file by itself, could be properly parsed into a tree in isolation. The goal is to support detaching a chunk from and replaced with a different tree, instead of reparsing the entire file for every keystroke. If chunks are quite small this could significantly improve the speed of parsing, particularly for huge if statements. 3. Archie must be able to detect when a 'chunk' had changed, based on editbus messages, reparse that chunk of code, and reattach the new subtree. 4. The model will need to be aware of the values of HASH-OFF/ON, LCC, OICC and CICC at the start of each chunk so it can inform the lexer. 5. If chunk boundaries are edited, multiple chunks, or the entire file may need to be rebuilt.
Q. What about the symbol table? I think this step is quick enough to support the entire tree being reprocessed on keystroke.