Invalidate, Delete, Refresh... using a pattern

Issue #63 new
Diogo Neves
created an issue

Would be great to invalidate, delete and refresh multiple values using a pattern instead of a list of keys.

I guess this would require changes to the API.

It could be as simple as adding a .keys(pattern) method.

At the moment I've extended Redis (the backend I'm using) and overridden the _multi methods to support a pattern string and lists.

I wouldn't mind giving it a try at implementing this, what do you think?

Comments (6)

  1. Michael Bayer repo owner

    how are you generating the keys from this pattern ? if you mean, "filter over all existing keys", that won't work. most backends don't support that. this use case seems more like a recipe to me (we have a recipe section in the docs).

  2. Diogo Neves reporter

    Ok, I wasn't aware of that :S
    I'm extending the RedisBackend directly to add the keys() method and access it directly too at the moment.
    This is probably not the best approach but I'm just prototyping the system.

    Do you have any recommendations? I won't mind updating the documentation ;)

  3. Diogo Neves reporter

    Some code ;)

    class RedisPatternBackend(RedisBackend):
        def __init__(self, arguments):
            super(RedisPatternBackend, self).__init__(arguments)
    
        def keys(self, pattern):
            # TODO: This should play well with the key generator
            return self.client.keys(pattern)
    
  4. Michael Bayer repo owner

    there's a recipe that grabs keys as they are made here: http://dogpilecache.readthedocs.org/en/latest/usage.html#invalidating-a-group-of-related-keys

    things like that. there's no keys(), so let's figure out how to track them as they're made locally.

    of course if you have arbitrarily large numbers of keys, that won't work.

    that's why, overall, plain old time-based invalidation is the easiest and most foolproof system, e.g. http://dogpilecache.readthedocs.org/en/latest/api.html#dogpile.cache.region.CacheRegion.invalidate.

  5. Diogo Neves reporter

    Maybe it's better if I explain the problem ;)

    First, the application will be running in multiple workers and the key creation may be done in a different worker from deletion.

    For most of the time, the cache behaves like a normal timed cache. The problem is, we have these moments when we need to invalidate all the content related to a user (different types of content, from different methods, called by different parts of the system).

    To make it easier, I created a key generator that appends 'uc_{userid}hash' to all user content. Then, when needed, I just delete anything starting with 'uc{userid}'.

    Different workers and parts of the system can cache content to speed up read times, but the invalidation is done in one place.

    The real problem has a few more details, but that's the bulk of it.

    Does it make any sense? Does it help understand how the pattern is useful?

  6. Diogo Neves reporter

    I just realised, re-reading the links you sent, this could be done by keeping a map of user_id -> [hashes associated]... the map itself can be saved in the cache backend ;)

    BOOM!

    Would that be a good recipe?

  7. Log in to comment