Additions to included backends?

Issue #29 resolved
Tim Hanus
created an issue


I've created 2 new cache backends for a project I'm working on.

The first is basically the memory backend, but limited in size to a certain number of records. The idea being that for a long running process you don't want a dict to grow completely unbounded. Expirations are least recently used is the first out.

The second is a wrapper that allows you to chain 2+ backends together. My use case is running it in combination with the above class to provide local memory lookup for many different processes running across many different hosts that all share the same memcache + mysql backend. In other words when we pull a value we check: local process memory -> shared memcache cluster -> finally expensive db query.

Another use case may be to bridge two memcache regions (1 local + 1 over wan for instance). Although admittedly, that idea is half baked. :)

I wanted to gauge if there was any interest in either or both of these things. If so, is it better as a pull request into dogpile.cache proper, or a separate project?

Comments (4)

  1. Michael Bayer repo owner

    A fixed-size memory backend is a very useful thing but is very hard to get right in a generalized sense; that is, works as efficiently as is reasonable in all cases. It's really what memcached is designed to do. So if you've built something that does some semblance of as good a job as Memcached does, with all the tests that go with that, then yes that would be pretty useful, but then again that kind of thing is complicated enough that it would be it's own package, and probably wouldn't be immediately specific to dogpile. It could have a dogpile backend included. Otherwise, if it's something that works so far only in kind of a trivial sense, then we're going to have an endless parade of bug/feature requests to make it act more fully like an in-memory Memcached, which is out of scope for Dogpile itself (just the fact that there's a memory backend already has led to this request :) ).

    The second backend you refer to is not too different from the one I'm looking for someone to write for #26, which is the "proxy backend". It allows one to basically customize behavior on top of an existing backend, and it's probably something that could be used to write rules for deciding between two (or more) backends as well. I'd love someone to make a good shot at #26.

  2. Tim Hanus reporter

    Thanks for the response.

    On #1: Fair point :) No, this is definitely not something that is efficient in all cases. This is more something that scratches my particular itch. I'll keep my strange little mutant class to myself. :)

    On #2: Thanks for the pointer to the other issue. I read that before opening this ticket, but I think I see what you mean a little better now. I'll close this and see if what I have can be broadened to address the proxy backend need. I will close this issue and move future questions / pull requests to #26.

    As an aside, thanks for this module. I've been squashing various disparate homebrew caching mechanisms for the last week and it's been glorious.

  3. Log in to comment