Improve speed of text highlight.

Issue #35 resolved
Marcin Kuzminski
created an issue

I'm using whoosh as an indexer in my app, it scanns over a mercurial repositories. When marking search results using highlight module sometime in can kill my server. For example building an index on few dictionary files and searching one word from this index makes my app freze for 2-4 minutes off 100% cpu usage to highlight the one word from content.

This is how i do it it's not a complicated analyzer or formatter. {{{


analyzer = RegexTokenizer(expression=r"\w+") | LowercaseFilter() formatter = HtmlFormatter('span', between='\n<span class="break">...</span>\n')

how the parts are splitted within the same text part

fragmenter = SimpleFragmenter(200)

fragmenter = ContextFragmenter(search_items)

for res in results: d = {} d.update(res) hl = highlight(escape(res['content']), search_items, analyzer=analyzer, fragmenter=fragmenter, formatter=formatter, top=5)


It would be awesome if it could be improved.

Comments (11)

  1. Matt Chaput repo owner

    On my first try, I can't reproduce your problem. This is what I tried:

    from whoosh import analysis, highlight
    analyzer = analysis.RegexTokenizer(expression=r"\w+") | analysis.LowercaseFilter()
    formatter = highlight.HtmlFormatter('span', between='\n<span class="break">...</span>\n') 
    fragmenter = highlight.SimpleFragmenter(200)
    #fragmenter = ContextFragmenter(search_items)
    f = open("context.txt", "rb")
    text ="utf8")
    from whoosh.util import now
    from cgi import escape
    t = now()
    text = escape(text)
    hl = highlight.highlight(text, ["abandon", "rocket"],
                             analyzer, fragmenter, formatter, top=5)
    print repr(hl)
    print now() - t

    This prints:

    u'Unfortunately, Klietz was eventually forced to <span class="match term0">abandon</span> his work.  The\ncompany that originally owned the rights to Screenplay, Gambit, was\nsubsumed into a larger company, Interplay.  Interplay later filed for'

    This isn't especially fast, but it's definitely not a freeze. Is there something else I should try?

  2. Marcin Kuzminski reporter

    I don't know much about the internals, but I think it's just to much text to parse, maybe if there where info how to efficient chunk the content just for highlighting it would help allot.

  3. Matt Chaput repo owner

    Yes, tokenizing a 3.5 MB file like that, including all the codes, is just going to be slow in Python. Changing the regular expression to only parse the words at the start of each line at least reduces the time to 3s. What is it you want to do? Were you just using this file as a test?

  4. Marcin Kuzminski reporter

    My use scenario is a full text search in source codes, and I have to use "heavy" tokenizers, since i would like to search on imports "from whoosh.example import something" which are divided by '.' in python. Other programming languages are totally different so finding such light tokenizer would be difficult.

    The problem in my case is that when searching for a specific pattern i accidentally hit a large file, the highlight freezes the application.

    The perfect solution for me would be if whoosh could chunk the content for interesting pieces only, similar to SimpleFragmenter, and then highlighting would be super fast. I don't know if such solution is even possible. I image that many of codes would have log files people like to search in my app and those can be big.

  5. Matt Chaput repo owner

    OK, I get it now. This is definitely possible, by storing character positions in the index, but it will increase the size of the index and requires using lower-level APIs. I'll work up an example and attach it.

  6. Matt Chaput repo owner

    Well, I had to fix up the implementation of spans, but this should be a start...

    This solution involves storing the characters at which each term occurs in each document as part of the index. Then when you go to search for a query, you have to use the low-level Matcher API to get the whoosh.spans.Span objects representing matches in the document. The spans will give you the start and end character of each occurance in the text, so you can pull out a chunk around each occurance and highlight/display that.

    # Instantiate a new field type that's like TEXT but uses the Characters
    # format, which stores the start and end character of each term in the
    # index
    charfield = fields.FieldType(format=formats.Characters(analyzer),
                                 scorable=True, stored=True)
    # Use the custom field specification in the schema
    schema = fields.Schema(title=TEXT(stored=True), content=charfield)
    # Index the collection with this schema
    # ...
    # Now, to search for my_query.
    # Use the lower-level Matcher API instead of a Results object
    # to get the matching spans.
    searcher = my_index.searcher()
    matcher = my_query.matcher(searcher)
    # Loop over the matching documents
    while matcher.is_active():
        # The document number of the current match
        docnum =
        # How to get the stored fields for the document number
        d = searcher.stored_fields(docnum)
        content = d["content"]
        # Loop over all the places in the document that matched
        # the query
        for span in matcher.spans():
            print "  Match from %d to %d" % (span.startchar, span.endchar)
            print "  ", content[max(0, span.startchar-40) : span.endchar+40]
        # Move to the next matching document

    Of course this is not an ideal solution. It's low-level, and for a complex query (e.g. OR with lots of words) you'll get spans all over the place. The Spans class has a couple of simple methods to merge spans ( and Span.merge()), but it would be nice to integrate spans with highlighting so the highlighting system could find clusters of spans and use them to pick the best "chunks" to highlight. But hopefully this is enough to help you.

  7. Marcin Kuzminski reporter

    Wow, I'm impressed by Your work. It worked out very good, first test that i run showed chunked out and highlighted content in fraction of seconds. Build index with those changes is only slightly larger.

    Works perfect !

    Keep on the good work, whoosh is for me the #1 library i discovered.


  8. Log in to comment