Overview

Python extension for libmemcache

Retired

cmemcache is retired as of 2009/12/21 due to lack of maintenance and more viable replacements. You can find the alternatives on the memcached website.

I have tried python-libmemcached briefly in my cache compare benchmark and it scores a little bit better than cmemcache on most operations, and on the rndmulti 10% faster. However, I have not run it in production so I can only endorse it in the sense that it is fast and in active development. If somebody has experience with python-libmemcached handling of faulty and changing memcached servers, cmemcache often crashed in these situations, I would very much like to hear about it.

Many thanks for using and contributing to cmemcache!

Description

cmemcache is about 1.7 times faster than python-memcache with short key names (8 characters), faster with larger key names (I get about 2x for 100 character keys). Using get_multi is faster still, almost 2x for 2 8-character keys. See cachecmp.py for profiling logic.

Motivation

This extension was created after doing some timings on a simple session caching scheme where I noticed that python-memcache was 'only' three times as fast as PostgreSQL. I expected it to be faster than that, since PostgreSQL does a lot more than memcached. It was suggested that it was perhaps python-memcache, which for instance gets the initial part of the message from memcached byte for byte. After discovering libmemcache it seemed pretty straight forward to create an extension on top of that.

cmemcache has 2 clients: StringClient and Client. StringClient is the extension that only supports python strings for values. The api was copied from python-memcache, except that it excepts strings only. Client is a module that implements caching of arbitrary python objects using Pickle on top of the StringClient. This code is a copy/paste from python-memcache. Most of the test code in test.py is run on the cmemcache Client and the python-memcache Client to make sure that they are interchangeable. Although I have not tested this but it should be possible to mix cmemcache and python-memcache clients in a running system as well, since they use the same constants for encoding object types.

The speed difference would be less if the memcached protocol would be changed to precede the reply header with its size. A client could then read the header size, read the full header (in one read), parse header to get the data size, and read data (this is already done in one read).

Dependencies

Download

Download no longer available (used to be hosted on http://gijsbert.org/cmemcache).

To do

  • use mc_req_add_ref to avoid copy of key
  • add performance test to test.py

Known Bugs

cmemcache:

  • Aborting libmemcache errors are not converted into python exceptions.

libmemcache:

  • set_servers with the wrong port number causes a segfault (in libmemcache). See commented out testing code in test.py.

  • mc_err_filter_add() broken. This results in warnings on add() and replace().

    This is fixed by applying the patch from the download section.

  • libmemcache-1.4.0.rc2 is not compatible with memcached 1.2.1, this results in get_stats returning no stats.

  • start memcached, create Client, kill memcached, start memcached, do a get() and python process exits, with messages like:

    [ERROR@1170236923.979369] mcm_buf_read():361: read(2) failed: Operation now in progress:
    server unexpectedly closed connection
    

    libmemcache does an exit on severe errors. It is possible to install an error handler that could change the severity to not exit and then the python extension could throw a python exception. However, I do not know if libmemcache will recover correctly.

    I do not have time at the moment to try and fix this.

    Reported by Mark and Philip, 31/01/2007.

Changes

Versions:

0.96

Change Client._set() str(ing) logic to pass unicode strings through the pickle code. Found and patched by Armin Ronacher. Fixed Client() object memory leak, found and patched by Dan Helfman. Updated some docs, fixed build on MacOS.

0.95

Fixed expire time internal type (was int, must be time_t). To be on the save side the expire time is also clamped to 2**31-1 to follow the memcache protocol spec. Old code caused problems when using sys.maxint on 64-bit machines. Reported and patched by Simon Law.
0.94

Added missing debuglog() implementation, copy/paste error from memcache.py. get() now returns None on all error cases. Reported by Simon Law.

Fixed some memory leaks in get_multi().

Fixed get_multi() return values for Client. All results would be returned as strings, instead of the proper types. Reported and fixed by Alfred J Fazio.

cmemcache.py uses debuglog() (memcache.py remnant) but debuglog() is not even defined, doh! Reported by Simon Law. His patch uses python module logging, but to avoid dependencies I added the same stderr logger as used in memcache.py. One can override the cmemcache.log variable to install another log function.

0.93

Fixed memory leak caused by not Py_DECREF of key and val objects when calling PyDict_SetItem(). Reported and fixed by Alfred J Fazio.

Allocated my own context and moved from 'mc_*' api to 'mcm_*' api to use. Install error handler to change the 'cont' state of the memcache_err_ctxt object to 'y' to try not to abort on fatal errors. This is an attempt to not crash python on fatal errors from libmemcache, but libmemcache needs considerable changes to make it work. I am not too sure about my libmemcache so I have not released them yet. Code is compatible with libmemcache-1.4.0 though.

0.92
Changed return values for set, add, and replace to be the same as memcache.py, ie nonzero on success. Reported by Marek Majkowski.
0.91
Remove @staticmethod from _convert method to make it python 2.3 compatible.
0.90
Initial version.

Author

Gijsbert de Haan <gijsbert.de.haan@gmail.com>