add_document use list/tuple field value will not call analyzer when indexing

Issue #442 invalid
fyrestone NA
created an issue

I have to use

writer.add_document(name=u' '.join(string_list))

instead of

writer.add_document(name=string_list)

to workaround the problem. The tokens function in formats.py:121 not call analyzer when value isinstance of list/tuple:

def tokens(value, analyzer, kwargs):
    if isinstance(value, (tuple, list)):
        gen = entoken(value, **kwargs) # entoken not call analyzer, but search query always call analyzer, so some values can not be searched.
    else:
        gen = analyzer(value, **kwargs)
    return unstopped(gen)

Comments (3)

  1. Matt Chaput repo owner

    Yes, sorry, that's expected behavior. People wanted the ability to submit a list of raw tokens instead of having a string tokenized. Maybe I should have done it differently but it's too late now ;) Your workaround (name=u' '.join(...)) is the correct thing to do in this case.

  2. Log in to comment