option to drop module/function from key_generator functions in util.py

Issue #121 new
jvanasco
created an issue

I tracked down a performance issue to usage of dogpile.cache.region:Region.cache_on_arguments in a few places.

The default key generation is provided key generator functions in dogpile.cache.util, with this relevant code shared by all:

    if namespace is None:
        namespace = '%s:%s' % (fn.__module__, fn.__name__)
    else:
        namespace = '%s:%s|%s' % (fn.__module__, fn.__name__, namespace)

My issue was the "base" of the key('%s:%s' % (fn.__module__, fn.__name__)) was incredibly long in a handful of the most-used places, and the keys themselves were taking up a significant chunk of the memory allocated to Redis. Dropping the key length reclaimed a lot of space.

While a key_mangler would solve this, I needed to keep the keys unmangled. The easiest solution was to just reimplement the stock key generator to just use the namespace argument as the key.

This seems like a relatively useful approach for many people, so I wanted to suggest porting it upstream:

I think the relevant changes would be:

  1. The key_generators have a new kwarg to omit the module+name prefix (like to_str)
  2. cache_on_arguments(_multi) have a new kwarg for the same, which is just passed to the key_generator (again, like to_str)

Comments (4)

  1. Michael Bayer repo owner

    this is...for convenience vs. making your own function_key_generator? the issue of "long keys" is indented to be solved by key_mangler. You can make a key_mangler that just truncates the module/name part, if you dont want to go with making a GUID (I assume that's where your concern wtih mangling is)..

    that is, you can get what you want with your own fn_key_geneartor, you can get what you want with a key mangler . where is this lacking?

  2. Michael Bayer repo owner

    how about this:

    def function_key_generator(namespace, fn, to_str=compat.string_type, namespace_key_template="%(module)s:%(fn_name)s|%(namespace)s", key_template="%(module)s:%(fn_name)s"):
    

    ?

  3. jvanasco reporter

    re: custom key_generator. This is for convenience. I fixed my deployment in 5 minutes with a custom key_generator, but this seems like it's something other people would want/encounter .

    re: mangled guid Yes, having a GUID/hash isn't an option. There are a bunch of tools that dump keys and pull metrics off them.

    re: mangled 'other' this particular deployment (web app) already has a key manglerĀ on all regions to handle 'invalid' values. adding splits/regex to the existing region would not work - dogpile is heavily leveraged on some pages (100-1000 hits). splitting it into another "region" with the same settings but a different key_mangler is an option, but would require a lot of work due to how regions are setup, managed, and tested in this app.

    the idea for a key_template is much better.

  4. Log in to comment