I tracked down a performance issue to usage of
dogpile.cache.region:Region.cache_on_arguments in a few places.
The default key generation is provided key generator functions in dogpile.cache.util, with this relevant code shared by all:
if namespace is None: namespace = '%s:%s' % (fn.__module__, fn.__name__) else: namespace = '%s:%s|%s' % (fn.__module__, fn.__name__, namespace)
My issue was the "base" of the key(
'%s:%s' % (fn.__module__, fn.__name__)) was incredibly long in a handful of the most-used places, and the keys themselves were taking up a significant chunk of the memory allocated to Redis. Dropping the key length reclaimed a lot of space.
key_mangler would solve this, I needed to keep the keys unmangled. The easiest solution was to just reimplement the stock key generator to just use the
namespace argument as the key.
This seems like a relatively useful approach for many people, so I wanted to suggest porting it upstream:
I think the relevant changes would be:
- The key_generators have a new kwarg to omit the module+name prefix (like to_str)
- cache_on_arguments(_multi) have a new kwarg for the same, which is just passed to the key_generator (again, like to_str)