Issue #10 new

Creating Indexes need much improvement

pyinstaller
created an issue

So to be so harsh, but I spend 3 hours trying to avoid the need to create a new class for each index. All indexes are sharing the same logic.

What I tried to aachief: Create many indexes like in your tut2 (add obj to index if key exists) at runtime without creating a new class for each index, since all indexes are sharing the same logic.

What I tried:

I tried many, many different approaches, but all failed. I even tried generating code on the fly. At last I can out with this one:

class BaseIndex(HashIndex):

def __init__(self, path, name, _key, _key_format, *args, **kwargs):
    self._key = _key
    self._key_format = _key_format
    kwargs['key_format'] = _key_format
    super(BaseIndex, self).__init__(path, name, *args, **kwargs)

[...] idx = BaseIndex(db.path, name, key, key_format) db.add_index(idx)

This works fine for creating the index, but opening the database later fails: File "lib/python2.7/site-packages/CodernityDB/database.py", line 203, in _read_index_single ind_obj = globals()_class TypeError: init() takes at least 5 arguments (3 given)

Please implement a decent way to create indexes!

Comments (2)

  1. codernity repo owner

    First of all, you have to understand how Indexes in CodernityDB are processed, and how it works, please look there: http://labs.codernity.com/codernitydb/database_indexes.html#how-an-index-code-is-processed-by-codernitydb

    If it's not clear enough I will explain it below in points:

    1. You create Index object in your code (code part controlled by you)
    2. Database is doing a code lookup for it using inspect module: https://bitbucket.org/codernity/codernitydb/src/75acda609c31bf00d93000b18bd29bb9e8df3ecb/CodernityDB/database.py?at=default#cl-161
    3. That code is saved to a disk into indexes directory.
    4. Then index is read from that file: https://bitbucket.org/codernity/codernitydb/src/75acda609c31bf00d93000b18bd29bb9e8df3ecb/CodernityDB/database.py?at=default#cl-185
    5. Please look at line https://bitbucket.org/codernity/codernitydb/src/75acda609c31bf00d93000b18bd29bb9e8df3ecb/CodernityDB/database.py?at=default#cl-203, there Index object is created again.

    That method allows us to open your index code without having it in the scope (you don't have to add index code every time when you want to use CodernityDB). But it has some disadvantages also. The most obvious one is that what you experienced. You just can't add more custom arguments to init to your custom index code, because it will crash.

    What you can do instead is to do similar mechanism that we have for Tree based index: https://bitbucket.org/codernity/codernitydb/src/75acda609c31/CodernityDB/tree_index.py?at=default#cl-154

    As you can see during open_index we read the properties from saved (it's first saved https://bitbucket.org/codernity/codernitydb/src/75acda609c31/CodernityDB/tree_index.py?at=default#cl-128)

    That's one of methods that you can try. What might work in your case is also to create indexes names in smart way, let's say my_index__x then during that open_index you can detect it's name (self.name) and parse it and fetch from that name x value (you said that your indexes are mostly for "if x in data").

    Another method is to use our indexcreator: http://labs.codernity.com/codernitydb/database_indexes.html#easier-way-of-creating-indexes It should fit your needs if you use plain hash or plain tree index. For your custom class you will need to play a bit with parser class to make it accept your custom class name. That will be probably one of the best methods for you, because it's quite powerful and not that hard to extend.

    Is it more clear now to you ?

    // and it's for sure not a bug.

    -- Warm Regards Jedrzej Nowak

  2. Log in to comment