.. note:: dogpile.cache is **not released or completed** at this time. Development
is currently in progress and the current code is not yet functional.
-At its core, Dogpile provides a locking interface around a "value creation" function.
+At the time of this writing, popular key/value servers include
+`Memcached <http://memcached.org>`_, `Redis <http://redis.io/>`_, and `Riak <http://wiki.basho.com/>`_.
+While these tools all have different usage focuses, they all have in common that the storage model
+is based on the retrieval of a value based on a key; as such, they are all potentially
+suitable for caching, particularly Memcached which is first and foremost designed for
-The interface supports several levels of usage, starting from
-one that is very rudimentary, then providing more intricate
-usage patterns to deal with certain scenarios. The documentation here will attempt to
-provide examples that use successively more and more of these features, as
-we approach how a fully featured caching system might be constructed around
-Note that when using the `dogpile.cache <http://bitbucket.org/zzzeek/dogpile.cache>`_
-package, the constructs here provide the internal implementation for that system,
-and users of that system don't need to access these APIs directly (though understanding
-the general patterns is a terrific idea in any case).
-Using the core Dogpile APIs described here directly implies you're building your own
-resource-usage system outside, or in addition to, the one
-`dogpile.cache <http://bitbucket.org/zzzeek/dogpile.cache>`_ provides.
+With a caching system in mind, dogpile.cache provides an interface to a particular Python API
+targeted at that system. In this document, we'll illustrate Memcached usage
+using the `pylibmc <http://pypi.python.org/pypi/pylibmc>`_ backend, which is a high performing
+Python library for Memcached. It can be compared to the `python-memcached <http://pypi.python.org/pypi/python-memcached>`_
+client, which is also an excellent product. Pylibmc is written against Memcached's native API
+so is markedly faster, though might be considered to have rougher edges. The API is actually a bit
+more verbose to allow for correct multithreaded usage.
A dogpile.cache configuration consists of the following components:
-* A *region*, which is an instance of ``CacheRegion``, and defines the configuration
- details for a particular cache backend.
-* A *backend*, which is an instance of ``CacheBackend``, describing how values
+* A *region*, which is an instance of :class:`.CacheRegion`, and defines the configuration
+ details for a particular cache backend. The :class:`.CacheRegion` can be considered
+ the "front end" used by applications.
+* A *backend*, which is an instance of :class:`.CacheBackend`, describing how values
are stored and retrieved from a backend. This interface specifies only
- ``get()``, ``put()`` and ``delete()``.
+ :meth:`~.CacheBackend.get`, :meth:`~.CacheBackend.put` and :meth:`~.CacheBackend.delete`.
+ The actual kind of :class:`.CacheBackend` in use for a particular :class:`.CacheRegion`
+ is determined by the underlying Python API being used to talk to the cache, such
+ as Pylibmc. The :class:`.CacheBackend` is instantiated behind the scenes and
+ not directly accessed by applications under normal circumstances.
* Value generation functions. These are user-defined functions that generate
- new values to be placed in the cache.
+ new values to be placed in the cache. While dogpile.cache offers the usual
+ "put" approach of placing data into the cache, the usual mode of usage is to only instruct
+ it to "get" a value, passing it a *creation function* which will be used to
+ generate a new value if and only if one is needed. This "get-or-create" pattern
+ is the entire key to the "Dogpile" system, which coordinates a single value creation
+ operation among many concurrent get operations for a particular key, eliminating
+ the issue of an expired value being redundantly re-generated by many workers simultaneously.
-The most common caching style in use these days is via memcached, so an example
-of this using the `pylibmc <http://pypi.python.org/pypi/pylibmc>`_ backend looks like::
+dogpile.cache includes a Pylibmc backend. A basic configuration looks like::
from dogpile.cache import make_region
-Above, we create a ``CacheRegion`` using the ``make_region()`` function, then
-apply the backend configuration via the ``configure()`` method, which returns the
+Above, we create a :class:`.CacheRegion` using the :func:`.make_region` function, then
+apply the backend configuration via the :meth:`.CacheRegion.configure` method, which returns the
region. The name of the backend is the only required argument,
in this case ``dogpile.cache.pylibmc``.
-Subsequent arguments then include *expiration_time*, which is the expiration
+The configuration is separated into two sections. Upon construction via :func:`.make_region`,
+the :class:`.CacheRegion` object is available, typically at module
+import time, for usage in decorating functions. Additional configuration details passed to
+:meth:`.CacheRegion.configure` are typically loaded from a configuration file and therefore
+not necessarily available until runtime, hence the two-step configurational process.
+Key arguments passed to :meth:`.CacheRegion.configure` include *expiration_time*, which is the expiration
time passed to the Dogpile lock, and *arguments*, which are arguments used directly
by the backend - in this case we are using arguments that are passed directly
+The :func:`.make_region` function currently calls the :class:`.CacheRegion` constructor directly.
+.. autoclass:: dogpile.cache.region.CacheRegion
+One you have a :class:`.CacheRegion`, the :meth:`.CacheRegion.cache_on_arguments` method can
+be used to decorate functions, but the cache itself can't be used until
+:meth:`.CacheRegion.configure` is called. The interface for that method is as follows:
+.. automethod:: dogpile.cache.region.CacheRegion.configure
+The :class:`.CacheRegion` can also be configured from a dictionary, using the :meth:`.CacheRegion.configure_from_config`
+.. automethod:: dogpile.cache.region.CacheRegion.configure_from_config
+The :class:`.CacheRegion` object is our front-end interface to a cache. It includes
+.. automethod:: dogpile.cache.region.CacheRegion.get
+.. automethod:: dogpile.cache.region.CacheRegion.get_or_create
+.. automethod:: dogpile.cache.region.CacheRegion.put
+.. automethod:: dogpile.cache.region.CacheRegion.delete
+.. automethod:: dogpile.cache.region.CacheRegion.cache_on_arguments
backend in the same way as if it were part of the existing sys.path.
For example, to create a backend called ``DictionaryBackend``, we subclass
from dogpile.cache import CacheBackend, NO_VALUE
pickle or similar can be used on the tuple - the "metadata" portion will always
be a small and easily serializable Python structure.
-The ``make_region()`` function accepts these arguments:
- Optional. A string name for the region. This isn't used internally
- but can be accessed via the ``.name`` parameter, helpful
- for configuring a region from a config file.
- Optional. Plug in a function that will produce a "cache key" given
- a data creation function and arguments. The structure of this function
- should be two levels: given the data creation function, return a new
- function that generates the key based on the given arguments. Such
- def my_key_generator(fn):
- namespace = fn.__name__
- def generate_key(*arg):
- return namespace + "_".join(str(s) for s in arg)
- function_key_generator = my_key_generator
- Optional. Function which will "mangle" the incoming keys. If left
- at ``None``, the backend may provide a default "mangler" function.
- Set to ``False`` to unconditionally disable key mangling.
-One you have a ``CacheRegion``, the ``cache_on_arguments()`` method can
-be used to decorate functions, but the cache itself can't be used until
-``configure()`` is called. That method accepts these arguments:
- Required. This is the name of the ``CacheBackend`` to use, and
- is resolved by loading the class from the ``dogpile.cache`` entrypoint.
- Optional. The expiration time passed to the dogpile system. The ``get_or_create()``
- method as well as the ``cache_on_arguments()`` decorator (note: **not** the
- ``get()`` method) will call upon the value creation function after this
- time period has passed since the last generation.
- Optional. The structure here is passed directly to the constructor
- of the ``CacheBackend`` in use, though is typically a dictionary.
-Configure Region from a Configuration Dictionary
-Call ``configure_from_config()`` instead::
- local_region = make_region()
- memcached_region = make_region()
- # regions are ready to use for function
- # decorators, but not yet for actual caching
- # later, when config is available
- "cache.memcached.arguments.url":"127.0.0.1, 10.0.0.1",
- local_region.configure_from_config(myconfig, "cache.local.")
- memcached_region.configure_from_config(myconfig, "cache.memcached.")
-The ``CacheRegion`` object is our front-end interface to a cache. It includes
- Return a value from the cache, based on the given key. While it's typical
- the key is a string, it's passed through to the underlying backend so can
- be of any type recognized by the backend. If the value is not present, returns the
- token ``NO_VALUE``. ``NO_VALUE`` evaluates to False, but is separate
- from ``None`` to distinguish between a cached value of ``None``.
- Note that the ``expiration_time`` argument is **not** used here - this method
- is a direct line to the backend's behavior.
- Similar to ``get``, will use the given "creation" function to create a new
- value if the value does not exist. This will use the underlying dogpile/
- expiration mechanism to determine when/how the creation function is called.
- Place a new value in the cache under the given key.
- Remove a value from the cache. This operation is idempotent (can be
- called multiple times, or on a non-existent key, safely)
- A function decorator that will cache the return value of the function
- using a key derived from the name of the function, its location within
- the application (i.e. source filename) as well as the arguments
- passed to the function.
- The generation of the key from the function is the big
- controversial thing that was a source of user issues with Beaker. Dogpile
- provides the latest and greatest algorithm used by Beaker, but also
- allows you to use whatever function you want, by specifying it
- to ``make_region()`` using the ``function_key_generator`` argument.