Commits

Jason Moiron committed 416b97f

version bump to 0.3, documentation update to come into line with some changes in 1.3 and the change in operation in ``get_backend``, change the preferred way johnny's cache is configured to using an option in the CACHES configuration rather than providing its own URI

Comments (0)

Files changed (13)

docs/backends.rst

 ~~~~~~~~~
 
 .. automodule:: johnny.backends.memcached
+.. autoclass:: johnny.backends.memcached.MemcachedCache
+.. autoclass:: johnny.backends.memcached.PyLibMCCache
 
 locmem
 ~~~~~~
 
 .. automodule:: johnny.backends.locmem
+.. autoclass:: johnny.backends.locmem.LocMemCache
 
+filebased
+~~~~~~~~~
 
+.. automodule:: johnny.backends.filebased
+.. autoclass:: johnny.backends.filebased.FileBasedCache
+
+
 Johnny Cache is a caching framework for django_ applications.  It works with
 the django caching abstraction, but was developed specifically with the use of
 memcached_ in mind.  Its main feature is a patch on Django's ORM that
-automatically caches all reads in a consistent manner.
+automatically caches all reads in a consistent manner.  It works with Django 
+1.1, 1.2, and 1.3.
 
 .. highlight:: sh
 
 
 A typical ``settings.py`` file configured for ``johnny-cache``::
     
-    # add johnny to installed apps
-    INSTALLED_APPS = ( 
-        # ...
-        'johnny',
-    )
     # add johnny's middleware
     MIDDLEWARE_CLASSES = (
         'johnny.middleware.LocalStoreClearMiddleware',
         # ... 
     )
     # some johnny settings
-    CACHE_BACKEND = 'johnny.backends.memcached://...'
+    CACHES = {
+        'default' : dict(
+            BACKEND = 'johnny.backends.memcached.MemcachedClass',
+            LOCATION = ['127.0.0.1:11211'],
+            JOHNNY_CACHE = True,
+        )
+    }
     JOHNNY_MIDDLEWARE_KEY_PREFIX='jc_myproj'
 
-Django doesn't *actually* require libraries to be 'installed', and since
-Johnny doesn't define any views, urls, or models, the first step isn't a
-requirement, but we like to make it clear what we're using and where on the
-``PYTHONPATH`` it might be.
+*Note*: The above configuration is for Django 1.3, which radically changed
+its cache configuration.  To see a full inspection of options for earlier
+versions of Django please see the `queryset cache <queryset_cache.html>`_
+docs.
 
 The ``MIDDLEWARE_CLASSES`` setting enables two middlewares:  the outer one
 clears a thread-local dict-like cache located at ``johnny.cache.local`` at
 your stack.  The second one enables the main feature of Johnny:  the 
 `queryset cache <queryset_cache.html>`_.
 
-The `custom backend setting <backends.html>`_ enables a thin wrapper around
-Django's ``memcached`` (or ``locmem``) cache class that allows cache times
-of "0", which memcached interprets as "forever" and locmem is patched to 
-see as forever.
+The ``CACHES`` configuration includes a `custom backend <backends.html>`_,
+which allows cache times of "0" to be interpreted as "forever", and marks
+the ``default`` cache backend as the one Johnny will use.
 
 Finally, the project's name is worked into the Johnny key prefix so that if
 other projects are run using the same cache pool, Johnny won't confuse the
 read the `queryset cache documentation <queryset_cache.html>`_ closely to
 see if you are doing anything that might require manual invalidation.
 
+Johnny does not define any views, urls, or models, so we can skip adding it
+to ``INSTALLED_APPS``.
+
 New in this version
 ~~~~~~~~~~~~~~~~~~~
 
-* many, many bugfixes
-* fixes for invalidation on queries that contain subselects in WHERE clauses
-* addition of `TransactionCommittingMiddleware <queryset_cache.html#using-with-transactionmiddleware>`_
-* python 2.4 support
-
-Future Django 1.3 Support
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-``johnny-cache`` is currently *not* compatible with Django 1.3b.  Version 0.2.1
-provides cache classes in the vein of the new cache classes for what will be
-Django 1.3, but johnny's transaction support is not functional in 1.3 yet.
-
-After 1.3 final is released, johnny 0.3 will be released, which will fully
-support 1.1-1.3.  If you need to run with 1.3 or django-trunk in the meantime,
-be mindful that Johnny can potentially cache (forever) reads that are done
-within a failing transaction, and please use it only if you are sure that this
-will not impact your application.
+* Django 1.3 support
 
 In Depth Documentation
 ~~~~~~~~~~~~~~~~~~~~~~

docs/localstore_cache.rst

 just be kept, referred to, or even modified throughout the lifetime of a 
 request, like messaging, media registration, or cached datasets.
 
-By default, the LocalStore Cache *just is*.  It is an instantiated object
-of the ``johnny.localstore.LocalStore`` class located in ``johnny.cache.local``.
-The use of the class comes from the middleware that clears it at the end
+By default, the LocalStore cache is an instantiated copy of
+the ``johnny.localstore.LocalStore`` class located in ``johnny.cache.local``.
+The usefulness of the class comes from the middleware that clears it at the end
 of each request.  Being a module-level object, it is a singleton.
 
+Johnny relies on ``johnny.cache.local`` for its transaction and savepoint
+support, so it is a good idea to enable the middleware to clear it per request
+as not doing so can gradually leak memory as this object has no built-in
+eviction.
+
 .. autoclass:: johnny.localstore.LocalStore
 

docs/queryset_cache.rst

 LRU per slab), you can cache reads forever and the old generations will
 naturally expire out faster than any "live" cache data.
 
-The QuerySet Cache supports Django versions 1.1 and 1.2.
+The QuerySet Cache supports Django versions 1.1, 1.2, and 1.3.
 
 .. autoclass:: johnny.cache.QueryCacheBackend
 
 .. autofunction:: johnny.cache.get_backend
 
+**NOTE**: The usage of ``get_backend`` has changed in Johnny 0.3.  The old
+version returned a class object, but the new version works more as a factory
+that gives you a properly configured QuerySetCache *object* for your Django
+version and current settings.
+
 The main goals of the QuerySet Cache are:
 
 * To cache querysets forever
 On commit, these keys are pushed to the global cache;  on rollback, they are
 discarded.
 
-Using with TransactionMiddleware
---------------------------------
+Using with TransactionMiddleware (Django 1.2 and earlier)
+---------------------------------------------------------
 
 Django ships with a middleware called 
 ``django.middleware.transaction.TransactionMiddleware``, which wraps all
 TransactionMiddleware, you'll never populate the cache with the querysets
 used in those views.
 
-This problem is described in `django ticket #9964`_, but unfortunately fixing
-it isn't straightforward because the "correct" thing to do here is in dispute.
-Starting with version 0.2.1, Johnny includes a middleware called
+This problem is described in `django ticket #9964`_, and has been fixed as
+of Django 1.3.  If you are using a Django version earlier than 1.3 and need
+to use the TransactionMiddleware, Johnny includes a middleware called
 ``johnny.middleware.CommittingTransactionMiddleware``, which is the same as
 the built in version, but always commits transactions on success.  Depending
 on your database, there are still ways to have SELECT statements modify data,
 Savepoints
 ----------
 
-Preliminary savepoint support is included in version 0.1.  More testing is
-needed (and welcomed).  Currently, the only django backend that has support 
-for Savepoints is the PostgreSQL backend (MySQL's InnoDB engine 
+Johnny supports savepoints, and although it has some comprehensive testing
+for savepoints, it is not entirely certain that they behave the same way
+across the two backends that support them.  Savepoints are tested in single
+and multi-db environments, from both inside and outside the transactions.
+
+Currently, of the backends shipped with Django only the PostgreSQL and 
+Oracle backends support savepoints (MySQL's InnoDB engine
 `supports savepoints <http://dev.mysql.com/doc/refman/5.0/en/savepoint.html>`_, 
-but its backend doesn't).  If you use savepoints, please see the 
-:ref:`manual-invalidation` section.
+but the Django MySQL backend doesn't).  If you use savepoints and are
+encountering invalidation issues, please report a bug and see the 
+:ref:`manual-invalidation` section for possible workarounds.
 
 Usage
 ~~~~~
 
 The following settings are available for the QuerySet Cache:
 
+* ``CACHES .. JOHNNY_CACHE``
 * ``DISABLE_QUERYSET_CACHE``
 * ``JOHNNY_MIDDLEWARE_KEY_PREFIX``
 * ``JOHNNY_MIDDLEWARE_SECONDS``
 * ``MAN_IN_BLACKLIST``
-* ``JOHNNY_CACHE_BACKEND``
+
+.. highlight:: python
+
+``CACHES .. JOHNNY_CACHE`` is the preferred way of designating a cache
+as the one used by Johnny.  Generally, it will look something like this::
+
+    CACHES = {
+        # ...
+        'johnny' : {
+            'BACKEND': '...',
+            'JOHNNY_CACHE': True,
+        }
+    }
+
+Johnny *needs* to run on one shared cache pool, so although the behavior is
+defined, a warning will be printed if ``JOHNNY_CACHE`` is found to be ``True``
+in multiple cache definitions.  If ``JOHNNY_CACHE`` is not present, Johnny
+will fall back to the deprecated ``JOHNNY_CACHE_BACKEND`` setting if set,
+and then to the default cache.
 
 ``DISABLE_QUERYSET_CACHE`` will disable the QuerySet cache even if the
-middleware is installed.  This is mostly to make it easy for other modules
-to disable the queryset cache without re-creating the entire middleware
-stack and then removing the QuerySet cache middleware.
+middleware is installed.  This is mostly to make it easy for non-production
+environments to disable the queryset cache without re-creating the entire 
+middleware stack and then removing the QuerySet cache middleware.
 
 ``JOHNNY_MIDDLEWARE_KEY_PREFIX``, default "jc", is to set the prefix for
 Johnny cache.  It's *very important* that if you are running multiple apps
 ``JOHNNY_MIDDLEWARE_SECONDS``, default ``0``, is the period that Johnny
 will cache both its generational keys *and* its query cache results.  Since
 the design goal of Johnny was to be able to maintain a consistent cache at
-all times, the default behavior is to cache everything *forever*.  Note that
-if you are not using one of Johnny's `custom backends <backends.html>`_, the 
-default value of ``0`` will work differently on different backends.
+all times, the default behavior is to cache everything *forever*.  If you are 
+not using one of Johnny's `custom backends <backends.html>`_, the default 
+value of ``0`` will work differently on different backends and might cause 
+Johnny to never cache anything.
 
 ``MAN_IN_BLACKLIST`` is a user defined tuple that contains table names to
 exclude from the QuerySet Cache.  If you have no sense of humor, or want your
 settings file to be understandable, you can use the alias
 ``JOHNNY_TABLE_BLACKLIST``.  We just couldn't resist.
 
+*Deprecated*
+------------
+
+* ``JOHNNY_CACHE_BACKEND``
+
 ``JOHNNY_CACHE_BACKEND`` is a cache backend URI similar to what is used by
-Django by default, but only used for Johnny. This allows you to seperate
-the Cache that is used by Johnny from the caching backend of the rest of your
-site.
+Django by default, but only used for Johnny.   In Django 1.2 and earlier, it
+was impossible to define multiple cache backends for Django's core caching
+framework, and this was used to allow separation between the cache that is
+used by Johnny and the caching backend for the rest of your app.
+
+In Django 1.3, this can also take the name of a configured cache, but it is
+recommended to use the ``JOHNNY_CACHE`` cache setting instead.
 
 Signals
 ~~~~~~~

johnny/backends/__init__.py

 # -*- coding: utf-8 -*-
 
 """
-Johnny provides two backends, both of which are subclassed versions of
-django builtins that cache "forever" when passed a 0 timeout. These
-are essentially the same as mmalone's inspiring ``django-caching``
-application's `cache backend monkey-patch
-<http://github.com/mmalone/django-caching/blob/master/app/cache.py>`_.
+Johnny provides a number of backends, all of which are subclassed versions of
+django builtins that cache "forever" when passed a 0 timeout. These are
+essentially the same as mmalone's inspiring ``django-caching`` application's
+`cache backend monkey-patch`_.
+
+The way Django interprets cache backend URIs has changed during its history.
+Please consult the specific `cache documentation`_ for your version of Django
+for the exact usage you should use.
+
+    .. _`cache backend monkey-patch`: http://github.com/mmalone/django-caching/blob/master/app/cache.py
+    .. _`cache documentation`: http://docs.djangoproject.com/en/dev/topics/cache
+
+Example usage for various Django versions::
+
+    CACHE_BACKEND="johnny.backends.memcached//..." # django <= 1.2
+
+    CACHE_BACKEND="johnny.backends.memcached.MemcachedCache//..." # django > 1.2
+
+    CACHES = {
+        'default' : {
+            'BACKEND': 'johnny.backends.memcached.PyLibMCCache',
+            'LOCATION': '...',
+        }
+    }
+
+**Important Note**:  The ``locmem`` and ``filebased`` caches are NOT recommended
+for setups in which there is more than one server using Johnny; invalidation
+will break with potentially disasterous results if the cache Johnny uses is not
+shared amongst all machines writing to the database.
 """
 
 __all__ = ['memcached', 'locmem', 'filebased']

johnny/backends/filebased.py

 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
-"""Infinite file-based caching.  Caches forever when passed timeout of 0.
-
-To use, change your ``CACHE_BACKEND`` setting to something like this::
-
-    CACHE_BACKEND="johnny.backends.filebased:///var/tmp/my_cache_directory"
-"""
+"""Infinite file-based caching.  Caches forever when passed timeout of 0."""
 
 from django.core.cache.backends import filebased
 from django.utils.encoding import smart_str
             timeout = sys.maxint
         return super(CacheClass, self).set(key, value, timeout)
 
-if django.VERSION[:2] < (1, 3):
+if django.VERSION[:2] > (1, 2):
     class FileBasedCache(CacheClass):
+        """File based cache named according to Django >= 1.3 conventions."""
         pass

johnny/backends/locmem.py

 This actually doesn't cache "forever", just for a very long time.  On
 32 bit systems, it will cache for 68 years, quite a bit longer than any
 computer will last.  On a 64 bit machine, your cache will expire about
-285 billion years after the Sun goes red-giant and destroys Earth.
-
-To use, change your ``CACHE_BACKEND`` setting to something like this::
-
-    CACHE_BACKEND="johnny.backends.locmem://.."
-"""
+285 billion years after the Sun goes red-giant and destroys Earth."""
 
 from django.core.cache.backends import locmem
 from django.utils.encoding import smart_str
 
 if django.VERSION[:2] > (1, 2):
     class LocMemCache(CacheClass):
+        """Locmem cache interpreting 0 as "a very long time", named according
+        to the Django 1.3 conventions."""
         pass

johnny/backends/memcached.py

 # -*- coding: utf-8 -*-
 
 """Infinite caching memcached class.  Caches forever when passed a timeout
-of 0. To use, change your ``CACHE_BACKEND`` setting to something like this::
-
-    CACHE_BACKEND="johnny.backends.memcached://.."
+of 0.  For Django >= 1.3, this module also provides ``MemcachedCache`` and
+``PyLibMCCache``, which use the backends of their respective analogs in
+django's default backend modules.
 """
 
 from django.core.cache.backends import memcached
 
 if django.VERSION[:2] > (1, 2):
     class MemcachedCache(memcached.MemcachedCache):
+        """Infinitely Caching version of django's MemcachedCache backend."""
         def _get_memcache_timeout(self, timeout=None):
             if timeout == 0: return 0 #2591999
             return super(MemcachedCache, self)._get_memcache_timeout(timeout)
 
     class PyLibMCCache(memcached.PyLibMCCache):
+        """PyLibMCCache version that interprets 0 to mean, roughly, 30 days.
+        This is because `pylibmc interprets 0 to mean literally zero seconds
+        <http://sendapatch.se/projects/pylibmc/misc.html#differences-from-python-memcached>`_
+        rather than "infinity" as memcached itself does.  The maximum timeout
+        memcached allows before treating the timeout as a timestamp is just
+        under 30 days."""
         def _get_memcache_timeout(self, timeout=None):
             # pylibmc doesn't like our definition of 0
             if timeout == 0: return 2591999
     # up with is to pre-create a blacklist set and use intersect.
     return bool(settings.BLACKLIST.intersection(tables))
 
-def get_backend():
-    """Get's a QueryCacheBackend class for the current version of django."""
+def get_backend(cache_backend=None, keyhandler=None, keygen=None):
+    """Get's a QueryCacheBackend object for the given options and current
+    version of django.  If no arguments are given, and a QCB has been
+    created previously, ``get_backend`` returns that.  Otherwise,
+    ``get_backend`` will return the default backend."""
     import django
+    cls = None
     if django.VERSION[:2] == (1, 1):
-        return QueryCacheBackend11
+        cls = QueryCacheBackend11
     if django.VERSION[:2] in ((1, 2), (1, 3)):
-        return QueryCacheBackend
-    raise ImproperlyConfigured("QueryCacheMiddleware cannot patch your version of django.")
+        cls = QueryCacheBackend
+    if cls is None:
+        raise ImproperlyConfigured("Johnny doesn't work on this version of django.")
+    return cls(cache_backend=cache_backend, keyhandler=keyhandler, keygen=keygen)
 
 def invalidate(*tables, **kwargs):
     """Invalidate the current generation for one or more tables.  The arguments
     can be either strings representing database table names or models.  Pass in
-    kwarg 'using' to set the database."""
-    backend = get_backend()()
+    kwarg ``using`` to set the database."""
+    backend = get_backend()
     db = kwargs.get('using', 'default')
     def resolve(x):
         if isinstance(x, basestring):
         if keyhandler: self.kh_class = keyhandler
         if keygen: self.kg_class = keygen
         if not cache_backend and not hasattr(self, 'cache_backend'):
-            from django.core.cache import cache as cache_backend
+            cache_backend = settings._get_backend()
 
         if not keygen and not hasattr(self, 'kg_class'):
             self.kg_class = KeyGen

johnny/middleware.py

 from johnny import cache, settings
 from django.core import signals
 
-if settings.CACHE_BACKEND:
-    cache_backend = get_cache(settings.CACHE_BACKEND)
-
-    # Some caches -- python-memcached in particular -- need to do a cleanup at the
-    # end of a request cycle. If the cache provides a close() method, wire it up
-    # here.
-    if hasattr(cache_backend, 'close'):
-        signals.request_finished.connect(cache_backend.close)
-else:
-    from django.core.cache import cache as cache_backend
-
 class QueryCacheMiddleware(object):
     """This middleware class monkey-patches django's ORM to maintain
     generational info on each table (model) and to automatically cache all
             # when we install, lets refresh the blacklist, just in case johnny
             # was loaded before the setting exists somehow...
             cache.blacklist = settings.BLACKLIST
-            self.query_cache_backend = cache.get_backend()(cache_backend)
+            self.query_cache_backend = cache.get_backend()
             self.query_cache_backend.patch()
             self.installed = True
 

johnny/settings.py

 
 MIDDLEWARE_SECONDS = getattr(settings, 'JOHNNY_MIDDLEWARE_SECONDS', 0)
 
-CACHE_BACKEND = getattr(settings, 'JOHNNY_CACHE_BACKEND', 
+CACHE_BACKEND = getattr(settings, 'JOHNNY_CACHE_BACKEND',
                 getattr(settings, 'CACHE_BACKEND', None))
+
+CACHES = getattr(settings, 'CACHES', {})
+
+def _get_backend():
+    """Returns the actual django cache object johnny is configured to use.
+    This relies on the settings only;  the actual active cache can
+    theoretically be changed at runtime."""
+    from django.core.cache import get_cache, cache
+    enabled = [n for n,c in sorted(CACHES.items()) if c.get('JOHNNY_CACHE', False)]
+    if len(enabled) > 1:
+        from warnings import warn
+        warn("Multiple caches configured for johnny-cache; using %s." % enabled[0])
+    if enabled:
+        return get_cache(enabled[0])
+    if CACHE_BACKEND:
+        backend = get_cache(CACHE_BACKEND)
+        if backend not in CACHES:
+            from django.core import signals
+            # Some caches -- python-memcached in particular -- need to do a cleanup at the
+            # end of a request cycle. If the cache provides a close() method, wire it up
+            # here.
+            if hasattr(cache_backend, 'close'):
+                signals.request_finished.connect(cache_backend.close)
+        return backend
+    return cache
+
     #CACHES = { 'default' : { 'BACKEND': 'johnny.backends.locmem.LocMemCache' }}
     CACHES = {
         'default' : {
-            'BACKEND': 'johnny.backends.memcached.MemcachedCache',
+            'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
+            #'BACKEND': 'johnny.backends.memcached.MemcachedCache',
             'LOCATION': ['localhost:11211'],
+            'JOHNNY_CACHE': True,
         }
     }
 
 from setuptools import setup
 #from distutils.core import setup
 
-version = '0.2.1'
+version = '0.3.0'
 
 setup(name='johnny-cache',
       version=version,