location filter uses lowercase only and ignores locations with dashes

Create issue
Issue #35 resolved
Pascal Lesage created an issue

Two cases were found:

1) Filtering on location using uppercase (as they are found in the activity) returns onthing (i.e. an empty list). Changing to lowercase works, even though in reality e.g. 'DE' != 'de'

2) Filtering on locations with dashes (as they are found in the activities) returns nothing (i.e. an empty list). Changing to lowercase in this case doesn't change anything, e.g. 'ca-qc', CA-QC', 'caqc', CAQC' all return nothing.

Comments (7)

  1. Chris Mutel repo owner

    CA-QC is a separate issue, as - is a special character in the search engine. Fixing this will be complicated, and is a bit too much off the path for Brightway2 development, at least for me. You can always do the following:

    [x for x in Database("foo").search("bar", filter={'location': "ca"}) if x['location'] == 'QC-CA']

    I will make a post to the Whoosh mailing list about the first case; note that the default analyzer is case-insensitive, but this is maybe applied inconsistently?

  2. Chris Mutel repo owner

    Reply from Matt Chaput (whoosh author):

    When you use the query parser, it runs the analyzer on the text and builds the query objects from the output.
    If you construct query objects directly, you are giving them the *exact* text to search for.

    Our search filter uses a query object:

    And([Term(k, v) for k, v in filter.items()])

    However, it is not yet clear to me how to work with the Whoosh machinery to make this case-insensitive.

  3. Bernhard Steubing

    also the "mask" option does only work with lowercase. Knowing this limitation, couldn't you simply convert all user input to location, mask, etc. to lowercase? Just an idea.

  4. Chris Mutel repo owner

    Note that the fix doesn't help terms that have a dash, these have to be filtered outside the Whoosh search index.

  5. Log in to comment