storage api: create / destroy storage

Create issue
Issue #277 resolved
Thomas Waldmann created an issue

there should be a way to create (starting from "nothing") and completely destroy (resulting in "nothing") a storage for whoosh without having storage-specific code in the application.

e.g. for FileStorage, one needs to first mkdir(index_dir) in the application, before creating a FileStorage object.

the FileStorage.clean method is not helpful here (although it looks a bit like it was intended for this), because one can not create a FileStorage object for a non-existing index_dir as it raises an exception in init then.

also, clean() is a bit unclean (hah!), because it somehow is a bit mixing up creation (see mkdir) and (partial) destruction (removal of all index files).

there is no way to completely destroy a FileStorage though, no rmdir, no rmtree, so one can not get completely rid of a storage to create a fresh one at the same place.

note: for gae (DatastoreStorage), clean() does nothing at all.

See there for some troubles this is creating on the application level: (up to line 375)

Comments (8)

  1. Matt Chaput repo owner

    I think this makes a lot of sense, to make it easier to swap storage implementations as you said. Adding .create() and .destroy() methods to Storage (with associated implementations) seems straightforward. The only thing that might require a little thought is how this would affect the create_in helper function.

  2. Matt Chaput repo owner

    Note that the arguments to open and create storage objects will still be implementation specific (e.g. FileStorage.create("dirname", create_if_missing=True) vs. DatastoreStorage.create()) so you'll still have a problem with swapping storage types, but at least it will reduce the amount of storage-specific code in the application. I'd be happy to hear any thoughts you have on this.

  3. Thomas Waldmann reporter

    just had a look at the changeset:

       Storage implementations should be written so that calling create() a
       second time on the same storage

    that phrase looks incomplete.

        def close(self):

    I don't think calling close should ever call destroy() internally. But luckily, you already removed that in a later changeset.

        def __enter__(self):
            return self

    One usually would rather expect some open() call inside here. Maybe it is just a matter of names and what is calling what, needs more thinking.

  4. Log in to comment