support Redis via HGET

Issue #81 new
jvanasco
created an issue

Brainstorming as an idea for feedback on how to best pull off -- it would be great to support Redis HGET [http://redis.io/commands/hget]

HGET allows one to shard keys into hashes. If you have a large amount of numeric keys, this can significantly drop the memory imprint of the redis database by partitioning into buckets ( here's an example blogpost http://instagram-engineering.tumblr.com/post/12202313862/storing-hundreds-of-millions-of-simple-key-value )

Some initial thoughts:

• RedisBackend could just handle hget/hset operations if the key were a tuple (not string) • A key_manger itself could be used to generate the hget/hset tuples • A variant of key_mangler assigned to the region could be used to generate a hset/hget tuple

Comments (5)

  1. jvanasco reporter

    different commands are needed by the client/redis to handle hashses, so the backend would still need adjustment.

    for example:

    def get(self, key):
        # handle tuple keys as HGET, string keys are GET
        if isinstance(key, ()):
            value = self.client.hget(key[0], key[1])
        else:
            value = self.client.get(key)
        if value is None:
            return NO_VALUE
        return pickle.loads(value)
    

    there appears to be a supported hash function for each normal k/v function

  2. jvanasco reporter

    I built a version of this working as an alternate backend.

    Do you have any recommendations for how people should release 3rd party plugins? ie, what naming conventions to use for the project, or what to namespace the plugin registration to?

  3. Michael Bayer repo owner

    hi, sorry I've had no time for dogpile things.

    I'm not sure if there are existing dogpile plugins on pypi (checking...) OK as we'd expect, there's everything, dogpile-foobar, foobar-dogpilecache, dogpile_cache_blah, so, welcome to Python :) dogpile-redis-hget? shrugs. is the only change here just that you want to drop in "hget" instead of "get" ? this almost looks more like a recipe rather than a released tool...

  4. jvanasco reporter

    no apologies necessary!

    if you don't have a preferred registry, i'll just leave it as-is... dogpile_backend_redis_advanced

    it's not just a hget/get swap. Redis puts the TTL expiry on the key, not the hash field -- so there was a bit of work getting the compatibility there. There was also some work on adapting set_multi and get_multi to perform the correct operations and return the ordered results. (e.g. set_multi needs to turn a mapping into a set of hash mappings for the relevant keys; get_multi needs to perform get and multiple hgets then combine the results into the right order).

    I also realized it would be much easier for me to maintain most of the absolute abuses of technology in the custom fork I use as a backend plugin than keeping it in sync with dogpile - so I've got this package offering an override to pickle as well (msgpack is way smaller). I will probably integrate the bit where I tear out the CachedValue wrapper and just rebuild it on the fly. We use expiry on Redis, so we can save a ton of space by storing only the payload. On moderately used caches this stuff is meaningless, but when you can drop a 300MB redis imprint to under 60MB it's a noticeable difference.

  5. Log in to comment