whoosh / docs / source / releases / 1_0.rst

Whoosh 1.x release notes

Whoosh 1.0 is a major milestone release with vastly improved performance and several useful new features.

The index format of this version is not compatibile with indexes created by previous versions of Whoosh. You will need to reindex your data to use this version.

Whoosh 1.3

Whoosh 1.3 adds a more efficient DATETIME field based on the new tiered NUMERIC field, and the DateParserPlugin. See :doc:`../dates`.

Whoosh 1.2

Whoosh 1.2 adds tiered indexing for NUMERIC fields, resulting in much faster range queries on numeric fields.

New features

Orders of magnitude faster searches for common terms. Whoosh now uses optimizations similar to those in Xapian to skip reading low-scoring postings.

Faster indexing and ability to use multiple processors (via multiprocessing module) to speed up indexing.

Flexible Schema: you can now add and remove fields in an index with the :meth:`whoosh.writing.IndexWriter.add_field` and :meth:`whoosh.writing.IndexWriter.remove_field` methods.

New hand-written query parser based on plug-ins. Less brittle, more robust, more flexible, and easier to fix/improve than the old pyparsing-based parser.

On-disk formats now use 64-bit disk pointers allowing files larger than 4 GB.

New :class:`whoosh.searching.Facets` class efficiently sorts results into facets based on any criteria that can be expressed as queries, for example tags or price ranges.

New :class:`whoosh.writing.BatchWriter` class automatically batches up individual add_document and/or delete_document calls until a certain number of calls or a certain amount of time passes, then commits them all at once.

New :class:`whoosh.analysis.BiWordFilter` lets you create bi-word indexed fields a possible alternative to phrase searching.

Fixed bug where files could be deleted before a reader could open them in threaded situations.

New :class:`whoosh.analysis.NgramFilter` filter, :class:`whoosh.analysis.NgramWordAnalyzer` analyzer, and :class:`whoosh.fields.NGRAMWORDS` field type allow producing n-grams from tokenized text.

Errors in query parsing now raise a specific whoosh.qparse.QueryParserError exception instead of a generic exception.

Previously, the query string * was optimized to a :class:`whoosh.query.Every` query which matched every document. Now the Every query only matches documents that actually have an indexed term from the given field, to better match the intuitive sense of what a query string like tag:* should do.

New :meth:`whoosh.searching.Searcher.key_terms_from_text` method lets you extract key words from arbitrary text instead of documents in the index.

Previously the :meth:`whoosh.searching.Searcher.key_terms` and :meth:`whoosh.searching.Results.key_terms` methods required that the given field store term vectors. They now also work if the given field is stored instead. They will analyze the stored string into a term vector on-the-fly. The field must still be indexed.

User API changes

The default for the limit keyword argument to :meth:`whoosh.searching.Searcher.search` is now 10. To return all results in a single Results object, use limit=None.

The Index object no longer represents a snapshot of the index at the time the object was instantiated. Instead it always represents the index in the abstract. Searcher and IndexReader objects obtained from the Index object still represent the index as it was at the time they were created.

Because the Index object no longer represents the index at a specific version, several methods such as up_to_date and refresh were removed from its interface. The Searcher object now has :meth:`~whoosh.searching.Searcher.last_modified`, :meth:`~whoosh.searching.Searcher.up_to_date`, and :meth:`~whoosh.searching.Searcher.refresh` methods similar to those that used to be on Index.

The document deletion and field add/remove methods on the Index object now create a writer behind the scenes to accomplish each call. This means they write to the index immediately, so you don't need to call commit on the Index. Also, it will be much faster if you need to call them multiple times to create your own writer instead:

# Don't do this
for id in my_list_of_ids_to_delete:
    myindex.delete_by_term("id", id)
myindex.commit()

# Instead do this
writer = myindex.writer()
for id in my_list_of_ids_to_delete:
    writer.delete_by_term("id", id)
writer.commit()

The postlimit argument to Index.writer() has been changed to postlimitmb and is now expressed in megabytes instead of bytes:

writer = myindex.writer(postlimitmb=128)

Instead of having to import whoosh.filedb.filewriting.NO_MERGE or whoosh.filedb.filewriting.OPTIMIZE to use as arguments to commit(), you can now simply do the following:

# Do not merge segments
writer.commit(merge=False)

# or

# Merge all segments
writer.commit(optimize=True)

The whoosh.postings module is gone. The whoosh.matching module contains classes for posting list readers.

Whoosh no longer maps field names to numbers for internal use or writing to disk. Any low-level method that accepted field numbers now accept field names instead.

Custom Weighting implementations that use the final() method must now set the use_final attribute to True:

from whoosh.scoring import BM25F

class MyWeighting(BM25F):
        use_final = True

        def final(searcher, docnum, score):
                return score + docnum * 10

This disables the new optimizations, forcing Whoosh to score every matching document.

:class:`whoosh.writing.AsyncWriter` now takes an :class:`whoosh.index.Index` object as its first argument, not a callable. Also, the keyword arguments to pass to the index's writer() method should now be passed as a dictionary using the writerargs keyword argument.

Whoosh now stores per-document field length using an approximation rather than exactly. For low numbers the approximation is perfectly accurate, while high numbers will be approximated less accurately.

The doc_field_length method on searchers and readers now takes a second argument representing the default to return if the given document and field do not have a length (i.e. the field is not scored or the field was not provided for the given document).

The :class:`whoosh.analysis.StopFilter` now has a maxsize argument as well as a minsize argument to its initializer. Analyzers that use the StopFilter have the maxsize argument in their initializers now also.

The interface of :class:`whoosh.writing.AsyncWriter` has changed.

Misc

  • Because the file backend now writes 64-bit disk pointers and field names instead of numbers, the size of an index on disk will grow compared to previous versions.
  • Unit tests should no longer leave directories and files behind.
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.