Parsing fractions of a file, starting in a certain state and handle incremental changes.

Issue #1109 new
created an issue

Hi all,

Is it currently possible to parse a fraction of a file, giving a certain start state, and returning the end state?

This is very useful for an editor, where highlighting the whole file on every single key press is not desired.

If not already possible, I could try to implement this myself. (When I have time). Any pointers are appreciated.


Comments (5)

  1. Tim Hatch

    Yes, this has been on my to-do list for a while, but without a tracking bug. The main complication comes from the multiline matches (like Python docstrings) so the continuation cookie needs to include both state information, but also a match position from which to resume.

    The version I've been kicking around looks something like:

    START_COOKIE = [[], 0, 0]
    def partial_highlight(text, cookie, desired_length):
      stack, desired_start, actual_start = cookie
      left = desired_length + desired_start - actual_start
      for ttype, value in lexer.get_tokens(text[startpos:]):
        # output here
        left -= len(value)
        if left < 1: break
      # construct cookie

    I doubt I'll get to it before PyCon, but may work on this during the sprints. Will you be around?

  2. Peter Brittain

    I've been looking at something like this recently and started putting a prototype together. Things are getting interesting with nested and delegating lexers, so I think I need more than just one stack and I'm currently using a linked list of LexerContext objects (returned as a 4th variable from get_tokens_unprocessed) to handle this.

    Would this be of interest if I can get all the tests running?

  3. Log in to comment