I am caching a large list of unique objects using the RedisBackend and I am seeing memory usage increase each time the cache expires.
This issue appears to be the intended behavior of pickle:
The Pickler instance keeps a memory of each of the lists dumped alive, so that if you later pickle a reference to the same list (or other mutable object) again, it can pickle a reference rather than a copy of the value. This is a feature.
By using the same Pickler instance to dump 10,000 unrelated lists, you simply grow the memo data structure beyond reason. So just don't do this!" - Guido van Rossum http://bugs.python.org/issue229810
So, we saw our memory usage grow because pickle.dumps is storing the new list in its memo each time cache expires.
What would be the best way to solve this?
Since custom serialization is not possible yet ( https://bitbucket.org/zzzeek/dogpile.cache/issues/18/get-serialization-to-be-configurable-with), I was thinking about overriding the RedisBackend and adding pickle.Pickler.clear_memo to the set method.