1. Nephics
  2. Untitled project
  3. beanstalkt

Commits

Jacob Söndergaard  committed 484605c

Initial commit

  • Participants
  • Branches default
  • Tags v0.1.0

Comments (0)

Files changed (7)

File README.md

View file
  • Ignore whitespace
+# beanstalktc: An async beanstalkd client for Tornado
+
+## About beanstalkd
+
+[Beanstalk](http://kr.github.com/beanstalkd/) is a simple, fast work queue.
+
+Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously.
+
+This module contains a beanstalkd client for [Tornado](http://www.tornadoweb.org) implemented using the asynchronous (non-blocking) [IOStream](http://www.tornadoweb.org/documentation/iostream.html) socket wrapper.
+
+The code and documentation is licensed under the Apache Licence, Version 2.0 ([http://www.apache.org/licenses/LICENSE-2.0.html]()).
+
+## Example client usage
+
+This simple example involves the most basic operations of putting a job in the queue, reserving and deleting it (the code are in `demo.py`):
+
+    import tornado.ioloop
+    import beanstalktc
+
+    def show(msg, value, cb):
+      print msg % value
+      cb()
+    
+    def stop():
+      client.close(ioloop.stop)
+    
+    def put():
+      client.put('A job to work on', callback=lambda s: show(
+          'Queued a job with jid %d', s, reserve))
+    
+    def reserve():
+      client.reserve(callback=lambda s: show(
+          'Reserved job %s', s, lambda: delete(s['jid'])))
+    
+    def delete(jid):
+      client.delete(jid, callback=lambda: show(
+          'Deleted job with jid %d', jid, stop))
+
+    client = beanstalktc.Client()
+    client.connect(put)
+    ioloop = tornado.ioloop.IOLoop.instance()
+    ioloop.start()
+
+Executing the script (`python demo.py`) with beanstalkd running produces the following output:
+
+    Queued a job with jid 1
+    Reserved job {'body': 'A job to work on', 'jid': 1, 'reserved': True}
+    Deleted job with jid 1
+
+Where `jid` is the job id that beanstalkd has given the job when putting in on the queue.
+
+## The job lifecycle
+
+A job in beanstalk gets created by a client with the put command. During its life it can be in one of four states:
+
+**`ready`**  
+The job waits in the ready queue until a worker sends the "reserve" command.  
+
+**`reserved`**  
+The job is reserved for a worker. The worker will execute the job, when it is finished the worker will send a "delete" command, removing the job from the queue.  
+
+**`delayed`**  
+The job is waiting a requested amount of time before it will be transitioned to the `ready` state.  
+
+**`buried`**  
+The job is in a FIFO linked list that will not be touched by the server until a client kicks them with the "kick" command  
+
+
+## Tubes
+
+The system has one or more tubes. Each tube consists of a ready queue and a delay queue. Each job spends its entire life in one tube. Consumers can show interest in tubes by sending the `watch` command; they can show disinterest by sending the `ignore` command. This set of interesting tubes is said to be a consumer’s `watch` list. When a client reserves a job, it may come from any of the tubes in its watch list.
+
+When a client connects, its watch list is initially just the tube named `default`. If it submits jobs without having sent a use command, they will live in the tube named `default`.
+
+Tubes are created on demand whenever they are referenced. If a tube is empty (that is, it contains no `ready`, `delayed`, or `buried` jobs) and no client refers to it, it will be deleted.
+
+## Reference for the client module
+
+The complete spec for the beanstalkd protocol is available in the repository.
+
+**`beanstalktc.Client(host='localhost', port=11300, connect_timeout=socket.getdefaulttimeout(), io_loop=None)`**  
+Creates a client object with methods for all beanstalkd commands as of version 1.8. The methods are described in the following.
+
+### Connection methods
+
+**`connect(callback=None)`**  
+Establish the client's connection to beanstalkd. Calls back when connection has been established.
+
+**`close(callback=None)`**  
+Close the client's connection to beanstalkd. Calls back when connection has been closed.
+
+### Producer methods
+
+**`put(body, priority=DEFAULT_PRIORITY, delay=0, ttr=120, callback=None)`**  
+This method is for any process that wants to insert a job (body, a string) into the current tube. The job can be delayed a number of seconds, before it is put in the ready queue, default is no delay. The job is assigned a Time To Run (tar, in seconds), the minimum is 1 sec., default ttr=120 sec. Calls back with job id when inserted.
+
+**`use(name, callback=None)`**  
+This method is for producers. Subsequent put commands will put jobs into the tube specified by this command. If no use command has been issued, jobs will be put into the tube named `default`. Calls back with the name of the tube now being used.
+
+### Worker methods
+
+**`reserve(timeout=None, callback=None)`**  
+Calls back with a newly-reserved job. If no job is available to be reserved, beanstalkd will wait to send a response until one becomes available. Once a job is reserved for the client, the client has limited time to run (TTR) the job before the job times out. When the job times out, the server will put the job back into the ready queue. Both the TTR and the actual time left can be found in response to the `stats-job` command.
+  
+**`delete(jid, callback=None)`**  
+Removes a job from the server entirely. It is normally used by the client when the job has successfully run to completion. A client can delete jobs that it has `reserved`, `ready` jobs, `delayed` jobs, and jobs that are `buried`.
+
+**`release(jid, priority=DEFAULT_PRIORITY, delay=0, callback=None)`**  
+Puts a reserved job back into the ready queue (and marks its state as ready) to be run by any client. It is normally used when the job fails because of a transitory error.
+
+**`bury(jid, priority=DEFAULT_PRIORITY, callback=None)`**  
+The `bury` command puts a job into the "buried" state. Buried jobs are put into a FIFO linked list and will not be touched by the server again until a client kicks them with the `kick` command.
+
+**`touch(jid, callback=None)`**  
+The `touch` command allows a worker to request more time to work on a job. This is useful for jobs that potentially take a long time, but you still want the benefits of a TTR pulling a job away from an unresponsive worker. A worker may periodically tell the server that it’s still alive and processing a job (e.g. it may do this on `DEADLINE_SOON`).
+
+**`watch(name, callback=None)`**  
+The `watch` command adds the named tube to the watch list for the current connection. A reserve command will take a job from any of the tubes in the watch list. For each new connection, the watch list initially consists of one tube, named `default`.
+
+**`ignore(name, callback=None)`**  
+The `ignore` command is for consumers. It removes the named tube from the watch list for the current connection.
+
+## Other commands
+
+**`peek(jid, callback=None)`**  
+**`peek_ready(callback=None)`**  
+**`peek_delayed(callback=None)`**  
+**`peek_buried(callback=None)`**  
+The `peek` commands let the client inspect a job in the system. There are four variations. All but the first operate only on the currently used tube.
+
+**`kick(bound=1, callback=None)`**  
+The `kick` command applies only to the currently used tube. It moves jobs into the ready queue. If there are any buried jobs, it will only kick buried jobs.
+
+**`kick_job(jid, callback=None)`**  
+The `kick_job` command is a variant of kick that operates with a single job identified by its job id. If the given job id exists and is in a buried or delayed state, it will be moved to the ready queue of the the same tube where it currently belongs.
+
+**`stats_job(jid, callback=None)`**  
+The `stats_job` command gives statistical information about the specified job if it exists. The callback gets a Python `dict` containing these keys:
+
+* `id` is the job id (mid)
+* `tube` is the name of the tube that contains this job
+* `state` is `ready`, `delayed`, `reserved` or `buried`
+* `pri` is the priority value set by the `put`, `release`, or `bury` commands.
+* `age` is the time in seconds since the `put` command that created this job.
+* `time-left` is the number of seconds left until the server puts this job into the ready queue. This number is only meaningful if the job is reserved or delayed. If the job is reserved and this amount of time elapses before its state changes, it is considered to have timed out.
+* `file` is the number of the earliest bin log file containing this job. If `-b` flag wasn’t used, this will be 0.
+* `reserves` is the number of times this job has been reserved.
+* `timeouts` is the number of times this job has timed out during a reservation.
+* `releases` is the number of times a client has released this job from a reservation.
+* `buries` is the number of times this job has been buried.
+* `kicks` is the number of times this job has been kicked.
+
+**`stats_tube(name, callback=None)`**  
+The stats-tube command gives statistical information about the specified tube if it exists. The callback gets a Python `dict` containing these keys:
+
+* `name` is the tube’s name.
+* `current-jobs-urgent` is the number of ready jobs with priority < 1024 in this tube.
+* `current-jobs-ready` is the number of jobs in the ready queue in this tube.
+* `current-jobs-reserved` is the number of jobs reserved by all clients in this tube.
+* `current-jobs-delayed` is the number of delayed jobs in this tube.
+* `current-jobs-buried` is the number of buried jobs in this tube.
+* `total-jobs` is the cumulative count of jobs created in this tube in the current beanstalkd process.
+* `current-using` is the number of open connections that are currently using this tube.
+* `current-waiting` is the number of open connections that have issued a reserve command while watching this tube but not yet received a response.
+* `current-watching` is the number of open connections that are currently watching this tube.
+* `pause` is the number of seconds the tube has been paused for.
+* `cmd-delete` is the cumulative number of delete commands for this tube
+* `cmd-pause-tube` is the cumulative number of pause-tube commands for this tube.
+* `pause-time-left` is the number of seconds until the tube is un-paused.
+
+Entries described as "cumulative" are reset when the beanstalkd process starts; they are not stored on disk with the `-b` flag.
+
+
+**`stats(callback=None)`**  
+The stats command gives statistical information about the system as a whole. The callback gets a Python `dict` containing these keys:
+
+* `current-jobs-urgent` is the number of ready jobs with priority < 1024.
+* `current-jobs-ready` is the number of jobs in the ready queue.
+* `current-jobs-reserved` is the number of jobs reserved by all clients.
+* `current-jobs-delayed` is the number of delayed jobs.
+* `current-jobs-buried` is the number of buried jobs.
+* `cmd-put` is the cumulative number of put commands.
+* `cmd-peek` is the cumulative number of peek commands.
+* `cmd-peek-ready` is the cumulative number of peek-ready commands.
+* `cmd-peek-delayed` is the cumulative number of peek-delayed commands.
+* `cmd-peek-buried` is the cumulative number of peek-buried commands.
+* `cmd-reserve` is the cumulative number of reserve commands.
+* `cmd-use` is the cumulative number of use commands.
+* `cmd-watch` is the cumulative number of watch commands.
+* `cmd-ignore` is the cumulative number of ignore commands.
+* `cmd-delete` is the cumulative number of delete commands.
+* `cmd-release` is the cumulative number of release commands.
+* `cmd-bury` is the cumulative number of bury commands.
+* `cmd-kick`` is the cumulative number of kick commands.
+* `cmd-stats` is the cumulative number of stats commands.
+* `cmd-stats-job` is the cumulative number of stats-job commands.
+* `cmd-stats-tube` is the cumulative number of stats-tube commands.
+* `cmd-list-tubes` is the cumulative number of list-tubes commands.
+* `cmd-list-tube-used` is the cumulative number of list-tube-used commands.
+* `cmd-list-tubes-watched` is the cumulative number of list-tubes-watched commands.
+* `cmd-pause-tube` is the cumulative number of pause-tube commands
+* `job-timeouts` is the cumulative count of times a job has timed out.
+* `total-jobs` is the cumulative count of jobs created.
+* `max-job-size` is the maximum number of bytes in a job.
+* `current-tubes` is the number of currently-existing tubes.
+* `current-connections` is the number of currently open connections.
+* `current-producers` is the number of open connections that have each issued at least one put command.
+* `current-workers` is the number of open connections that have each issued at least one reserve command.
+* `current-waiting` is the number of open connections that have issued a reserve command but not yet received a response.
+* `total-connections` is the cumulative count of connections.
+* `pid is` the process id of the server.
+* `version` is the version string of the server.
+* `rusage-utime` is the cumulative user CPU time of this process in seconds and microseconds.
+* `rusage-stime` is the cumulative system CPU time of this process in seconds and microseconds.
+* `uptime` is the number of seconds since this server process started running.
+* `binlog-oldest-index` is the index of the oldest bin log file needed to store the current jobs
+* `binlog-current-index` is the index of the current bin log file being written to. If bin log is not active this value will be 0
+* `binlog-max-size` is the maximum size in bytes a bin log file is allowed to get before a new bin log file is opened
+* `binlog-records-written` is the cumulative number of records written to the bin log
+* `binlog-records-migrated` is the cumulative number of records written as part of compaction
+
+Entries described as "cumulative" are reset when the beanstalkd process starts; they are not stored on disk with the `-b` flag.
+
+**`list_tubes(callback=None)`**  
+The `list_tubes` command calls back with a list of all existing tubes.
+
+**`list_tube_used(callback=None)`**  
+The `list_tube_used` command calls back with the name of the tube currently being used by the client.
+
+**`list_tubes_watched(self, callback=None)`**  
+The `list_tubes_watched` command calls back with a list of tubes currently being watched by the client.
+
+**`pause_tube(name, delay, callback=None)`**  
+The `pause_tube` command can delay any new job being reserved for a given time.
+
+## Implementation notes
+
+Tests are contained in `btc_test.py` and all tests cases can be run by `python btc_test.py`. 
+
+The beanstalkd protocol uses YAML for communicating the various stats and lists. The client has a crude YAML parser, suitable only for parsing simple lists and dicts, which eliminates the dependency of a YAML parser.

File beanstalktc/__init__.py

View file
  • Ignore whitespace
+from beanstalktc import (Client, BeanstalktcException, UnexpectedResponse,
+        CommandFailed, Buried, DeadlineSoon)

File beanstalktc/beanstalktc.py

View file
  • Ignore whitespace
+#!/usr/bin/env python
+"""beanstalktc - An async beanstalkd client for Tornado"""
+
+__license__ = '''
+Copyright (C) 2012 Nephics AB
+
+Parts of the code adopted from the beanstalkc project are:
+    Copyright (C) 2008-2012 Andreas Bolka
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+'''
+
+__version__ = '0.1.0'
+
+import socket
+
+from tornado.ioloop import IOLoop
+from tornado.iostream import IOStream
+
+
+DEFAULT_PRIORITY = 2 ** 31
+DEFAULT_TTR = 120  # Time (in seconds) To Run a job, min. 1 sec.
+
+
+class BeanstalktcException(Exception): pass
+class UnexpectedResponse(BeanstalktcException): pass
+class CommandFailed(BeanstalktcException): pass
+class Buried(BeanstalktcException): pass
+class DeadlineSoon(BeanstalktcException): pass
+
+
+class Client(object):
+
+    def __init__(self, host='localhost', port=11300,
+                 connect_timeout=socket.getdefaulttimeout(), io_loop=None):
+        self._connect_timeout = connect_timeout
+        self.host = host
+        self.port = port
+        self.io_loop = io_loop or IOLoop.instance()
+        self._socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM,
+                socket.IPPROTO_TCP)
+        self._stream = IOStream(self._socket, io_loop=self.io_loop)
+
+    def connect(self, callback=None):
+        """Connect to beanstalkd server."""
+        self._stream.connect((self.host, self.port), callback)
+
+    def close(self, callback=None):
+        """Close connection to server."""
+        self._stream.set_close_callback(callback)
+        self._stream.write('quit\r\n', self._stream.close)
+
+    def closed(self):
+        '''Returns true if the connection has been closed.'''
+        return self._stream.closed()
+
+    def _interact(self, command, callback, expected_ok=[], expected_err=[]):
+        # write command to socket stream
+        self._stream.write(command,
+                # when command is written: read line from socket stream
+                lambda: self._stream.read_until('\r\n',
+                # when a line has been read: return status and results
+                lambda resp: self._parse_response(resp, command, expected_ok,
+                expected_err, callback)))
+
+    def _parse_response(self, resp, command, expected_ok, expected_err,
+            callback):
+        spl = resp.split()
+        status, results = spl[0], spl[1:]
+        if status in expected_ok:
+            callback(results)
+            return
+        cmd = command.split('\r\n')[0]
+        if status == 'BURIED':
+            raise Buried(cmd, status, int(results[0])
+                    if results and results[0].isdigit() else results)
+        elif status in expected_err:
+            raise CommandFailed(cmd, status, results)
+        else:
+            raise UnexpectedResponse(cmd, status, results)
+
+    def _interact_success(self, command, callback, expected_ok=[],
+            expected_err=[]):
+        self._interact(command, lambda r: callback and callback(), expected_ok,
+                expected_err)
+
+    def _interact_value(self, command, callback, expected_ok=[],
+            expected_err=[]):
+        # callback with a string or integer value
+        self._interact(command, lambda r: callback and callback(int(r[0])
+                if r[0].isdigit() else r[0]), expected_ok, expected_err)
+
+    def _interact_peek(self, command, callback):
+        # callback with a job or None
+        try:
+            self._interact(command,
+                lambda r: self._read_job(int(r[0]), int(r[1]), callback,
+                reserved=False), ['FOUND'], ['NOT_FOUND'])
+        except CommandFailed:
+            callback and callback(None)
+
+    def _read_job(self, jid, size, callback, reserved=True):
+        # read the body including the terminating two bytes of crlf
+        self._stream.read_bytes(size + 2, lambda resp: callback and callback({
+                'jid': jid, 'body': resp[:-2], 'reserved': reserved}))
+
+    def _interact_data(self, command, callback, expected_ok=[],
+            expected_err=[]):
+        self._interact(command,
+                # on success, read data bytes including the terminating two
+                # bytes of crlf, and parse the yaml data
+                lambda r: self._stream.read_bytes(int(r[0]) + 2,
+                lambda data: self._parse_yaml(data, callback)),
+                expected_ok, expected_err)
+
+    def _parse_yaml(self, data, callback):
+        # dirty parsing of yaml data (assume either list or dict)
+        spl = data[:-2].split('\n')[1:-1]
+        if spl[0].startswith('- '):
+            # it is a list
+            callback([s[2:] for s in spl])
+        else:
+            # it is a dict
+            conv = lambda v: ((float(v) if '.' in v else int(v))
+                if v[0].isdigit() or v[-1].isdigit() else v)
+            callback(dict((k, conv(v.strip())) for k, v in
+                    (s.split(':') for s in spl)))
+
+    #
+    #  Producer commands
+    #
+
+    def put(self, body, priority=DEFAULT_PRIORITY, delay=0, ttr=120,
+            callback=None):
+        """Put a job body (a string) into the current tube.
+
+        The job can be delayed a number of seconds, before it is put in the
+        ready queue, default is no delay.
+
+        The job is assigned a Time To Run (ttr, in seconds), the mininum is
+        1 sec., default ttr=120 sec.
+        
+        Calls back with job id when inserted.
+        
+        Raises a Buried or CommandFailed exception if job is buried (the
+        server ran out of memory trying to grow the priority queue data 
+        structure), the body is too big or the server is in draining mode.
+        """
+        self._interact_value('put %d %d %d %d\r\n%s\r\n' %
+                (priority, delay, ttr, len(body), body),
+                callback, ['INSERTED'], ['BURIED', 'JOB_TOO_BIG',
+                'DRAINING'])
+
+    def use(self, name, callback=None):
+        """Use the tube with given name.
+    
+        Calls back with the name of the tube now being used.
+        """
+        self._interact_value('use %s\r\n' % name, callback, ['USING'])
+    
+    #
+    #  Worker commands
+    #
+
+    def reserve(self, timeout=None, callback=None):
+        """Reserve a job from one of the watched tubes, with optional timeout
+        in seconds.
+
+        Calls back with a job dict, with keys jid (job id), body (string), and
+        reserved (boolean). If request times out, call back with None.
+
+        Raises a DeadlineSoon exception if ....
+        """
+        if timeout is not None:
+            command = 'reserve-with-timeout %d\r\n' % timeout
+        else:
+            command = 'reserve\r\n'
+        try:
+            self._interact(command,
+                    lambda r: self._read_job(int(r[0]), int(r[1]), callback),
+                    ['RESERVED'], ['DEADLINE_SOON', 'TIMED_OUT'])
+        except CommandFailed as (_, status, results):
+            if status == 'TIMED_OUT':
+                callback and callback(None)
+            elif status == 'DEADLINE_SOON':
+                raise DeadlineSoon(results)
+
+    def delete(self, jid, callback=None):
+        """Delete a job, by job id.
+    
+        Call back when job is deleted.
+    
+        A CommandFailed exception is raised if job id does not exists.
+        """
+        self._interact_success('delete %d\r\n' % jid, callback, ['DELETED'],
+                ['NOT_FOUND'])
+    
+    def release(self, jid, priority=DEFAULT_PRIORITY, delay=0, callback=None):
+        """Release a reserved job back into the ready queue.
+    
+        Call back when job is released or buried (if the server ran out of
+        memory trying to grow the priority queue data structure).
+    
+        A CommandFailed exception is raised if the job does not exist or is not
+        reserved by the client.
+        """
+        self._interact_success('release %d %d %d\r\n' % (jid, priority, delay),
+                callback, ['RELEASED'], ['BURIED', 'NOT_FOUND'])
+    
+    def bury(self, jid, priority=DEFAULT_PRIORITY, callback=None):
+        """Bury a job, by job id.
+    
+        Call back when job is buried.
+    
+        A CommandFailed exception is raised if the job does not exist or is not
+        reserved by the client.
+        """
+        self._interact_success('bury %d %d\r\n' % (jid, priority), callback,
+                ['BURIED'], ['NOT_FOUND'])
+    
+    def touch(self, jid, callback=None):
+        """Touch a job, by job id, requesting more time to work on a reserved
+        job before it expires.
+    
+        Call back when job is touched.
+    
+        A CommandFailed exception is raised if the job does not exist or is not
+        reserved by the client.
+        """
+        self._interact_success('touch %d\r\n' % jid, callback, ['TOUCHED'],
+                ['NOT_FOUND'])
+
+    def watch(self, name, callback=None):
+        """Watch a given tube.
+    
+        Call back with number of tubes currently in the watch list.
+        """
+        self._interact_value('watch %s\r\n' % name, callback, ['WATCHING'])
+
+    def ignore(self, name, callback=None):
+        """Stop watching a given tube.
+    
+        Call back with number of tubes currently in the watch list.
+    
+        A CommandFailed exception is raised on an attempt to ignore the only
+        tube in the watch list.
+        """
+        self._interact_value('ignore %s\r\n' % name, callback, ['WATCHING'],
+            ['NOT_IGNORED'])
+
+    #
+    #  Other commands
+    #
+
+    def peek(self, jid, callback=None):
+        """Peek at a job.
+    
+        Call back with a job dict or None.
+        """
+        self._interact_peek('peek %d\r\n' % jid, callback)
+    
+    def peek_ready(self, callback=None):
+        """Peek at next ready job in the current tube.
+    
+        Call back with a job dict or None.
+        """
+        self._interact_peek('peek-ready\r\n', callback)
+    
+    def peek_delayed(self, callback=None):
+        """Peek at next delayed job in the current tube.
+    
+        Call back with a job dict or None.
+        """
+        self._interact_peek('peek-delayed\r\n', callback)
+    
+    def peek_buried(self, callback=None):
+        """Peek at next buried job in the current tube.
+    
+        Call back with a job dict or None.
+        """
+        self._interact_peek('peek-buried\r\n', callback)
+    
+    def kick(self, bound=1, callback=None):
+        """Kick at most bound jobs into the ready queue.
+
+        Call back with the number of jobs actually kicked.
+        """
+        self._interact_value('kick %d\r\n' % bound, callback, ['KICKED'])
+
+    def kick_job(self, jid, callback=None):
+        """Kick job with given id into the ready queue.
+        (Requires Beanstalkd version >= 1.8)
+
+        Call back if job is kicked.
+
+        A CommandFailed exception is raised if job does not exists or is not in
+        a kickable state.
+        """
+        self._interact_success('kick-job %d\r\n' % jid, callback, ['KICKED'],
+                ['NOT_FOUND'])
+
+    def stats_job(self, jid, callback=None):
+        """Return a dict of stats about a job, by job id."""
+        self._interact_data('stats-job %d\r\n' % jid, callback,
+                ['OK'], ['NOT_FOUND'])
+
+    def stats_tube(self, name, callback=None):
+        """Return a dict of stats about a given tube.
+    
+        A CommandFailed exception is raised if tube does not exists.
+        """
+        self._interact_data('stats-tube %s\r\n' % name, callback,
+                ['OK'], ['NOT_FOUND'])
+    
+    def stats(self, callback=None):
+        """Call back with a dict of beanstalkd statistics."""
+        return self._interact_data('stats\r\n', callback, ['OK'])
+    
+    def list_tubes(self, callback=None):
+        """Call back with a list of all existing tubes."""
+        self._interact_data('list-tubes\r\n', callback, ['OK'])
+
+    def list_tube_used(self, callback=None):
+        """Call back with a name of tube currently being used."""
+        self._interact_value('list-tube-used\r\n', callback, ['USING'])
+
+    def list_tubes_watched(self, callback=None):
+        """Call back with a list of all tubes being watched."""
+        self._interact_data('list-tubes-watched\r\n', callback, ['OK'])
+
+    def pause_tube(self, name, delay, callback=None):
+        """Pause a tube for a given delay time, in seconds.
+
+        Call back when tube is paused.
+
+        A CommandFailed exception is raised if tube does not exists.
+        """
+        self._interact_success('pause-tube %s %d\r\n' % (name, delay),
+                callback, ['PAUSED'], ['NOT_FOUND'])

File beanstalktc/btc_test.py

View file
  • Ignore whitespace
+import tornado.testing
+
+import beanstalktc
+
+
+class BeanstalkTest(tornado.testing.AsyncTestCase):
+
+    def setUp(self):
+        tornado.testing.AsyncTestCase.setUp(self)
+        self.btc = beanstalktc.Client(io_loop=self.io_loop)
+        self.btc.connect(self.stop)
+        self.wait(timeout=0.1)
+
+    def tearDown(self):
+        self.btc.close(self.stop)
+        self.wait(timeout=0.1)
+        tornado.testing.AsyncTestCase.tearDown(self)
+
+    def test_basics(self):
+        '''Test put-reserve-delete cycle'''
+        # put the job on the queue
+        body = 'test job'
+        self.btc.put(body, callback=self.stop)
+        jid = self.wait()
+        self.assertIsInstance(jid, int)
+        
+        # reserve the job
+        self.btc.reserve(callback=self.stop)
+        job = self.wait()
+        self.assertIsNotNone(job)
+        self.assertEqual(job['jid'], jid)
+        self.assertEqual(job['body'], body)
+        self.assertTrue(job['reserved'])
+        
+        # delete the job
+        self.btc.delete(jid, callback=self.stop)
+        self.wait()
+
+    def test_peek_bury_kick(self):
+        '''Test peeking, burying and kicking'''
+        # put the job on the queue with 1 sec delay
+        body = 'test job'
+        self.btc.put(body, delay=1, callback=self.stop)
+        jid = self.wait()
+
+        def check(job, reserved=False):
+            self.assertIsNotNone(job)
+            self.assertEqual(job['jid'], jid)
+            self.assertEqual(job['body'], body)
+            if reserved:
+                self.assertTrue(job['reserved'])
+            else:
+                self.assertFalse(job['reserved'])
+
+        # peak the next delayed job
+        self.btc.peek_delayed(callback=self.stop)
+        check(self.wait())
+
+        # peak the job
+        self.btc.peek(jid, callback=self.stop)
+        check(self.wait())
+
+        # kick the job to ready
+        self.btc.kick_job(jid, callback=self.stop)
+        try:
+            self.wait()
+        except beanstalktc.UnexpectedResponse as (_, status, __):
+            if status != 'UNKNOWN_COMMAND':
+                raise
+            # kick-job command is not available in Beanstalkd version <= 1.7
+            self.btc.kick(callback=self.stop)
+            self.wait()
+        
+        # peak next ready
+        self.btc.peek_ready(callback=self.stop)
+        check(self.wait())
+
+        # reserve and bury the job
+        self.btc.reserve(callback=self.stop)
+        job = self.wait()
+        check(job, True)
+        self.btc.bury(jid, callback=self.stop)
+        self.wait()
+
+        # peak the next buried job
+        self.btc.peek_buried(callback=self.stop)
+        check(self.wait())
+
+        # kick the job to ready
+        self.btc.kick(callback=self.stop)
+        self.assertEqual(self.wait(), 1)
+        
+        # delete the job
+        self.btc.delete(jid, callback=self.stop)
+        self.wait()
+
+
+if __name__ == '__main__':
+    import sys
+    if len(sys.argv) == 1:
+        sys.argv.append('btc_test')
+    tornado.testing.main()

File beanstalktc/demo.py

View file
  • Ignore whitespace
+import tornado.ioloop
+import beanstalktc
+
+def show(msg, value, cb):
+  print msg % value
+  cb()
+
+def stop():
+  client.close(ioloop.stop)
+
+def put():
+  client.put('A job to work on', callback=lambda s: show(
+      'Queued a job with jid %d', s, reserve))
+
+def reserve():
+  client.reserve(callback=lambda s: show(
+      'Reserved job %s', s, lambda: delete(s['jid'])))
+
+def delete(jid):
+  client.delete(jid, callback=lambda: show(
+      'Deleted job with jid %d', jid, stop))
+
+client = beanstalktc.Client()
+client.connect(put)
+ioloop = tornado.ioloop.IOLoop.instance()
+ioloop.start()

File protocol.txt

View file
  • Ignore whitespace
+= Beanstalk Protocol =
+
+Protocol
+--------
+
+The beanstalk protocol runs over TCP using ASCII encoding. Clients connect,
+send commands and data, wait for responses, and close the connection. For each
+connection, the server processes commands serially in the order in which they
+were received and sends responses in the same order. All integers in the
+protocol are formatted in decimal and (unless otherwise indicated)
+nonnegative.
+
+Names, in this protocol, are ASCII strings. They may contain letters (A-Z and
+a-z), numerals (0-9), hyphen ("-"), plus ("+"), slash ("/"), semicolon (";"),
+dot ("."), dollar-sign ("$"), underscore ("_"), and parentheses ("(" and ")"),
+but they may not begin with a hyphen. They are terminated by white space
+(either a space char or end of line). Each name must be at least one character
+long.
+
+The protocol contains two kinds of data: text lines and unstructured chunks of
+data. Text lines are used for client commands and server responses. Chunks are
+used to transfer job bodies and stats information. Each job body is an opaque
+sequence of bytes. The server never inspects or modifies a job body and always
+sends it back in its original form. It is up to the clients to agree on a
+meaningful interpretation of job bodies.
+
+The client may issue the "quit" command, or simply close the TCP connection
+when it no longer has use for the server. However, beanstalkd performs very
+well with a large number of open connections, so it is usually better for the
+client to keep its connection open and reuse it as much as possible. This also
+avoids the overhead of establishing new TCP connections.
+
+If a client violates the protocol (such as by sending a request that is not
+well-formed or a command that does not exist) or if the server has an error,
+the server will reply with one of the following error messages:
+
+ - "OUT_OF_MEMORY\r\n" The server cannot allocate enough memory for the job.
+   The client should try again later.
+
+ - "INTERNAL_ERROR\r\n" This indicates a bug in the server. It should never
+   happen. If it does happen, please report it at
+   http://groups.google.com/group/beanstalk-talk.
+
+ - "BAD_FORMAT\r\n" The client sent a command line that was not well-formed.
+   This can happen if the line does not end with \r\n, if non-numeric
+   characters occur where an integer is expected, if the wrong number of
+   arguments are present, or if the command line is mal-formed in any other
+   way.
+
+ - "UNKNOWN_COMMAND\r\n" The client sent a command that the server does not
+   know.
+
+These error responses will not be listed in this document for individual
+commands in the following sections, but they are implicitly included in the
+description of all commands. Clients should be prepared to receive an error
+response after any command.
+
+As a last resort, if the server has a serious error that prevents it from
+continuing service to the current client, the server will close the
+connection.
+
+Job Lifecycle
+-------------
+
+A job in beanstalk gets created by a client with the "put" command. During its
+life it can be in one of four states: "ready", "reserved", "delayed", or
+"buried". After the put command, a job typically starts out ready. It waits in
+the ready queue until a worker comes along and runs the "reserve" command. If
+this job is next in the queue, it will be reserved for the worker. The worker
+will execute the job; when it is finished the worker will send a "delete"
+command to delete the job.
+
+Here is a picture of the typical job lifecycle:
+
+
+   put            reserve               delete
+  -----> [READY] ---------> [RESERVED] --------> *poof*
+
+
+
+Here is a picture with more possibilities:
+
+
+
+   put with delay               release with delay
+  ----------------> [DELAYED] <------------.
+                        |                   |
+                        | (time passes)     |
+                        |                   |
+   put                  v     reserve       |       delete
+  -----------------> [READY] ---------> [RESERVED] --------> *poof*
+                       ^  ^                |  |
+                       |   \  release      |  |
+                       |    `-------------'   |
+                       |                      |
+                       | kick                 |
+                       |                      |
+                       |       bury           |
+                    [BURIED] <---------------'
+                       |
+                       |  delete
+                        `--------> *poof*
+
+
+The system has one or more tubes. Each tube consists of a ready queue and a
+delay queue. Each job spends its entire life in one tube. Consumers can show
+interest in tubes by sending the "watch" command; they can show disinterest by
+sending the "ignore" command. This set of interesting tubes is said to be a
+consumer's "watch list". When a client reserves a job, it may come from any of
+the tubes in its watch list.
+
+When a client connects, its watch list is initially just the tube named
+"default". If it submits jobs without having sent a "use" command, they will
+live in the tube named "default".
+
+Tubes are created on demand whenever they are referenced. If a tube is empty
+(that is, it contains no ready, delayed, or buried jobs) and no client refers
+to it, it will be deleted.
+
+Producer Commands
+-----------------
+
+The "put" command is for any process that wants to insert a job into the queue.
+It comprises a command line followed by the job body:
+
+put <pri> <delay> <ttr> <bytes>\r\n
+<data>\r\n
+
+It inserts a job into the client's currently used tube (see the "use" command
+below).
+
+ - <pri> is an integer < 2**32. Jobs with smaller priority values will be
+   scheduled before jobs with larger priorities. The most urgent priority is 0;
+   the least urgent priority is 4,294,967,295.
+
+ - <delay> is an integer number of seconds to wait before putting the job in
+   the ready queue. The job will be in the "delayed" state during this time.
+
+ - <ttr> -- time to run -- is an integer number of seconds to allow a worker
+   to run this job. This time is counted from the moment a worker reserves
+   this job. If the worker does not delete, release, or bury the job within
+   <ttr> seconds, the job will time out and the server will release the job.
+   The minimum ttr is 1. If the client sends 0, the server will silently
+   increase the ttr to 1.
+
+ - <bytes> is an integer indicating the size of the job body, not including the
+   trailing "\r\n". This value must be less than max-job-size (default: 2**16).
+
+ - <data> is the job body -- a sequence of bytes of length <bytes> from the
+   previous line.
+
+After sending the command line and body, the client waits for a reply, which
+may be:
+
+ - "INSERTED <id>\r\n" to indicate success.
+
+   - <id> is the integer id of the new job
+
+ - "BURIED <id>\r\n" if the server ran out of memory trying to grow the
+   priority queue data structure.
+
+   - <id> is the integer id of the new job
+
+ - "EXPECTED_CRLF\r\n" The job body must be followed by a CR-LF pair, that is,
+   "\r\n". These two bytes are not counted in the job size given by the client
+   in the put command line.
+
+ - "JOB_TOO_BIG\r\n" The client has requested to put a job with a body larger
+   than max-job-size bytes.
+
+ - "DRAINING\r\n" This means that the server has been put into "drain mode"
+   and is no longer accepting new jobs. The client should try another server
+   or disconnect and try again later.
+
+The "use" command is for producers. Subsequent put commands will put jobs into
+the tube specified by this command. If no use command has been issued, jobs
+will be put into the tube named "default".
+
+use <tube>\r\n
+
+ - <tube> is a name at most 200 bytes. It specifies the tube to use. If the
+   tube does not exist, it will be created.
+
+The only reply is:
+
+USING <tube>\r\n
+
+ - <tube> is the name of the tube now being used.
+
+Worker Commands
+---------------
+
+A process that wants to consume jobs from the queue uses "reserve", "delete",
+"release", and "bury". The first worker command, "reserve", looks like this:
+
+reserve\r\n
+
+Alternatively, you can specify a timeout as follows:
+
+reserve-with-timeout <seconds>\r\n
+
+This will return a newly-reserved job. If no job is available to be reserved,
+beanstalkd will wait to send a response until one becomes available. Once a
+job is reserved for the client, the client has limited time to run (TTR) the
+job before the job times out. When the job times out, the server will put the
+job back into the ready queue. Both the TTR and the actual time left can be
+found in response to the stats-job command.
+
+A timeout value of 0 will cause the server to immediately return either a
+response or TIMED_OUT.  A positive value of timeout will limit the amount of
+time the client will block on the reserve request until a job becomes
+available.
+
+During the TTR of a reserved job, the last second is kept by the server as a
+safety margin, during which the client will not be made to wait for another
+job. If the client issues a reserve command during the safety margin, or if
+the safety margin arrives while the client is waiting on a reserve command,
+the server will respond with:
+
+DEADLINE_SOON\r\n
+
+This gives the client a chance to delete or release its reserved job before
+the server automatically releases it.
+
+TIMED_OUT\r\n
+
+If a non-negative timeout was specified and the timeout exceeded before a job
+became available, or if the client's connection is half-closed, the server
+will respond with TIMED_OUT.
+
+Otherwise, the only other response to this command is a successful reservation
+in the form of a text line followed by the job body:
+
+RESERVED <id> <bytes>\r\n
+<data>\r\n
+
+ - <id> is the job id -- an integer unique to this job in this instance of
+   beanstalkd.
+
+ - <bytes> is an integer indicating the size of the job body, not including
+   the trailing "\r\n".
+
+ - <data> is the job body -- a sequence of bytes of length <bytes> from the
+   previous line. This is a verbatim copy of the bytes that were originally
+   sent to the server in the put command for this job.
+
+The delete command removes a job from the server entirely. It is normally used
+by the client when the job has successfully run to completion. A client can
+delete jobs that it has reserved, ready jobs, delayed jobs, and jobs that are
+buried. The delete command looks like this:
+
+delete <id>\r\n
+
+ - <id> is the job id to delete.
+
+The client then waits for one line of response, which may be:
+
+ - "DELETED\r\n" to indicate success.
+
+ - "NOT_FOUND\r\n" if the job does not exist or is not either reserved by the
+   client, ready, or buried. This could happen if the job timed out before the
+   client sent the delete command.
+
+The release command puts a reserved job back into the ready queue (and marks
+its state as "ready") to be run by any client. It is normally used when the job
+fails because of a transitory error. It looks like this:
+
+release <id> <pri> <delay>\r\n
+
+ - <id> is the job id to release.
+
+ - <pri> is a new priority to assign to the job.
+
+ - <delay> is an integer number of seconds to wait before putting the job in
+   the ready queue. The job will be in the "delayed" state during this time.
+
+The client expects one line of response, which may be:
+
+ - "RELEASED\r\n" to indicate success.
+
+ - "BURIED\r\n" if the server ran out of memory trying to grow the priority
+   queue data structure.
+
+ - "NOT_FOUND\r\n" if the job does not exist or is not reserved by the client.
+
+The bury command puts a job into the "buried" state. Buried jobs are put into a
+FIFO linked list and will not be touched by the server again until a client
+kicks them with the "kick" command.
+
+The bury command looks like this:
+
+bury <id> <pri>\r\n
+
+ - <id> is the job id to release.
+
+ - <pri> is a new priority to assign to the job.
+
+There are two possible responses:
+
+ - "BURIED\r\n" to indicate success.
+
+ - "NOT_FOUND\r\n" if the job does not exist or is not reserved by the client.
+
+The "touch" command allows a worker to request more time to work on a job.
+This is useful for jobs that potentially take a long time, but you still want
+the benefits of a TTR pulling a job away from an unresponsive worker.  A worker
+may periodically tell the server that it's still alive and processing a job
+(e.g. it may do this on DEADLINE_SOON).
+
+The touch command looks like this:
+
+touch <id>\r\n
+
+ - <id> is the ID of a job reserved by the current connection.
+
+There are two possible responses:
+
+ - "TOUCHED\r\n" to indicate success.
+
+ - "NOT_FOUND\r\n" if the job does not exist or is not reserved by the client.
+
+The "watch" command adds the named tube to the watch list for the current
+connection. A reserve command will take a job from any of the tubes in the
+watch list. For each new connection, the watch list initially consists of one
+tube, named "default".
+
+watch <tube>\r\n
+
+ - <tube> is a name at most 200 bytes. It specifies a tube to add to the watch
+   list. If the tube doesn't exist, it will be created.
+
+The reply is:
+
+WATCHING <count>\r\n
+
+ - <count> is the integer number of tubes currently in the watch list.
+
+The "ignore" command is for consumers. It removes the named tube from the
+watch list for the current connection.
+
+ignore <tube>\r\n
+
+The reply is one of:
+
+ - "WATCHING <count>\r\n" to indicate success.
+
+   - <count> is the integer number of tubes currently in the watch list.
+
+ - "NOT_IGNORED\r\n" if the client attempts to ignore the only tube in its
+   watch list.
+
+Other Commands
+--------------
+
+The peek commands let the client inspect a job in the system. There are four
+variations. All but the first operate only on the currently used tube.
+
+ - "peek <id>\r\n" - return job <id>.
+
+ - "peek-ready\r\n" - return the next ready job.
+
+ - "peek-delayed\r\n" - return the delayed job with the shortest delay left.
+
+ - "peek-buried\r\n" - return the next job in the list of buried jobs.
+
+There are two possible responses, either a single line:
+
+ - "NOT_FOUND\r\n" if the requested job doesn't exist or there are no jobs in
+   the requested state.
+
+Or a line followed by a chunk of data, if the command was successful:
+
+FOUND <id> <bytes>\r\n
+<data>\r\n
+
+ - <id> is the job id.
+
+ - <bytes> is an integer indicating the size of the job body, not including
+   the trailing "\r\n".
+
+ - <data> is the job body -- a sequence of bytes of length <bytes> from the
+   previous line.
+
+The kick command applies only to the currently used tube. It moves jobs into
+the ready queue. If there are any buried jobs, it will only kick buried jobs.
+Otherwise it will kick delayed jobs. It looks like:
+
+kick <bound>\r\n
+
+ - <bound> is an integer upper bound on the number of jobs to kick. The server
+   will kick no more than <bound> jobs.
+
+The response is of the form:
+
+KICKED <count>\r\n
+
+ - <count> is an integer indicating the number of jobs actually kicked.
+
+The kick-job command is a variant of kick that operates with a single job
+identified by its job id. If the given job id exists and is in a buried or
+delayed state, it will be moved to the ready queue of the the same tube where it
+currently belongs. The syntax is:
+
+kick-job <id>\r\n
+
+ - <id> is the job id to kick.
+
+The response is one of:
+
+ - "NOT_FOUND\r\n" if the job does not exist or is not in a kickable state. This
+   can also happen upon internal errors.
+
+ - "KICKED\r\n" when the operation succeeded.
+
+The stats-job command gives statistical information about the specified job if
+it exists. Its form is:
+
+stats-job <id>\r\n
+
+ - <id> is a job id.
+
+The response is one of:
+
+ - "NOT_FOUND\r\n" if the job does not exist.
+
+ - "OK <bytes>\r\n<data>\r\n"
+
+   - <bytes> is the size of the following data section in bytes.
+
+   - <data> is a sequence of bytes of length <bytes> from the previous line. It
+     is a YAML file with statistical information represented a dictionary.
+
+The stats-job data is a YAML file representing a single dictionary of strings
+to scalars. It contains these keys:
+
+ - "id" is the job id
+
+ - "tube" is the name of the tube that contains this job
+
+ - "state" is "ready" or "delayed" or "reserved" or "buried"
+
+ - "pri" is the priority value set by the put, release, or bury commands.
+
+ - "age" is the time in seconds since the put command that created this job.
+
+ - "time-left" is the number of seconds left until the server puts this job
+   into the ready queue. This number is only meaningful if the job is
+   reserved or delayed. If the job is reserved and this amount of time
+   elapses before its state changes, it is considered to have timed out.
+
+ - "file" is the number of the earliest binlog file containing this job.
+   If -b wasn't used, this will be 0.
+
+ - "reserves" is the number of times this job has been reserved.
+
+ - "timeouts" is the number of times this job has timed out during a
+   reservation.
+
+ - "releases" is the number of times a client has released this job from a
+   reservation.
+
+ - "buries" is the number of times this job has been buried.
+
+ - "kicks" is the number of times this job has been kicked.
+
+The stats-tube command gives statistical information about the specified tube
+if it exists. Its form is:
+
+stats-tube <tube>\r\n
+
+ - <tube> is a name at most 200 bytes. Stats will be returned for this tube.
+
+The response is one of:
+
+ - "NOT_FOUND\r\n" if the tube does not exist.
+
+ - "OK <bytes>\r\n<data>\r\n"
+
+   - <bytes> is the size of the following data section in bytes.
+
+   - <data> is a sequence of bytes of length <bytes> from the previous line. It
+     is a YAML file with statistical information represented a dictionary.
+
+The stats-tube data is a YAML file representing a single dictionary of strings
+to scalars. It contains these keys:
+
+ - "name" is the tube's name.
+
+ - "current-jobs-urgent" is the number of ready jobs with priority < 1024 in
+   this tube.
+
+ - "current-jobs-ready" is the number of jobs in the ready queue in this tube.
+
+ - "current-jobs-reserved" is the number of jobs reserved by all clients in
+   this tube.
+
+ - "current-jobs-delayed" is the number of delayed jobs in this tube.
+
+ - "current-jobs-buried" is the number of buried jobs in this tube.
+
+ - "total-jobs" is the cumulative count of jobs created in this tube in
+   the current beanstalkd process.
+
+ - "current-using" is the number of open connections that are currently
+   using this tube.
+
+ - "current-waiting" is the number of open connections that have issued a
+   reserve command while watching this tube but not yet received a response.
+
+ - "current-watching" is the number of open connections that are currently
+   watching this tube.
+
+ - "pause" is the number of seconds the tube has been paused for.
+
+ - "cmd-delete" is the cumulative number of delete commands for this tube
+
+ - "cmd-pause-tube" is the cumulative number of pause-tube commands for this
+   tube.
+
+ - "pause-time-left" is the number of seconds until the tube is un-paused.
+
+The stats command gives statistical information about the system as a whole.
+Its form is:
+
+stats\r\n
+
+The server will respond:
+
+OK <bytes>\r\n
+<data>\r\n
+
+ - <bytes> is the size of the following data section in bytes.
+
+ - <data> is a sequence of bytes of length <bytes> from the previous line. It
+   is a YAML file with statistical information represented a dictionary.
+
+The stats data for the system is a YAML file representing a single dictionary
+of strings to scalars. Entries described as "cumulative" are reset when the
+beanstalkd process starts; they are not stored on disk with the -b flag.
+
+ - "current-jobs-urgent" is the number of ready jobs with priority < 1024.
+
+ - "current-jobs-ready" is the number of jobs in the ready queue.
+
+ - "current-jobs-reserved" is the number of jobs reserved by all clients.
+
+ - "current-jobs-delayed" is the number of delayed jobs.
+
+ - "current-jobs-buried" is the number of buried jobs.
+
+ - "cmd-put" is the cumulative number of put commands.
+
+ - "cmd-peek" is the cumulative number of peek commands.
+
+ - "cmd-peek-ready" is the cumulative number of peek-ready commands.
+
+ - "cmd-peek-delayed" is the cumulative number of peek-delayed commands.
+
+ - "cmd-peek-buried" is the cumulative number of peek-buried commands.
+
+ - "cmd-reserve" is the cumulative number of reserve commands.
+
+ - "cmd-use" is the cumulative number of use commands.
+
+ - "cmd-watch" is the cumulative number of watch commands.
+
+ - "cmd-ignore" is the cumulative number of ignore commands.
+
+ - "cmd-delete" is the cumulative number of delete commands.
+
+ - "cmd-release" is the cumulative number of release commands.
+
+ - "cmd-bury" is the cumulative number of bury commands.
+
+ - "cmd-kick" is the cumulative number of kick commands.
+
+ - "cmd-stats" is the cumulative number of stats commands.
+
+ - "cmd-stats-job" is the cumulative number of stats-job commands.
+
+ - "cmd-stats-tube" is the cumulative number of stats-tube commands.
+
+ - "cmd-list-tubes" is the cumulative number of list-tubes commands.
+
+ - "cmd-list-tube-used" is the cumulative number of list-tube-used commands.
+
+ - "cmd-list-tubes-watched" is the cumulative number of list-tubes-watched
+   commands.
+
+ - "cmd-pause-tube" is the cumulative number of pause-tube commands
+
+ - "job-timeouts" is the cumulative count of times a job has timed out.
+
+ - "total-jobs" is the cumulative count of jobs created.
+
+ - "max-job-size" is the maximum number of bytes in a job.
+
+ - "current-tubes" is the number of currently-existing tubes.
+
+ - "current-connections" is the number of currently open connections.
+
+ - "current-producers" is the number of open connections that have each
+   issued at least one put command.
+
+ - "current-workers" is the number of open connections that have each issued
+   at least one reserve command.
+
+ - "current-waiting" is the number of open connections that have issued a
+   reserve command but not yet received a response.
+
+ - "total-connections" is the cumulative count of connections.
+
+ - "pid" is the process id of the server.
+
+ - "version" is the version string of the server.
+
+ - "rusage-utime" is the cumulative user CPU time of this process in seconds
+   and microseconds.
+
+ - "rusage-stime" is the cumulative system CPU time of this process in
+   seconds and microseconds.
+
+ - "uptime" is the number of seconds since this server process started running.
+
+ - "binlog-oldest-index" is the index of the oldest binlog file needed to
+   store the current jobs
+
+ - "binlog-current-index" is the index of the current binlog file being
+   written to. If binlog is not active this value will be 0
+
+ - "binlog-max-size" is the maximum size in bytes a binlog file is allowed
+   to get before a new binlog file is opened
+
+ - "binlog-records-written" is the cumulative number of records written
+   to the binlog
+
+ - "binlog-records-migrated" is the cumulative number of records written
+   as part of compaction
+
+ - "id" a unique id for this server process. The id is generated on each startup and
+    is always a random series of 8 bytes base16 encoded
+
+ - "hostname" the hostname of the machine as determined by uname
+
+The list-tubes command returns a list of all existing tubes. Its form is:
+
+list-tubes\r\n
+
+The response is:
+
+OK <bytes>\r\n
+<data>\r\n
+
+ - <bytes> is the size of the following data section in bytes.
+
+ - <data> is a sequence of bytes of length <bytes> from the previous line. It
+   is a YAML file containing all tube names as a list of strings.
+
+The list-tube-used command returns the tube currently being used by the
+client. Its form is:
+
+list-tube-used\r\n
+
+The response is:
+
+USING <tube>\r\n
+
+ - <tube> is the name of the tube being used.
+
+The list-tubes-watched command returns a list tubes currently being watched by
+the client. Its form is:
+
+list-tubes-watched\r\n
+
+The response is:
+
+OK <bytes>\r\n
+<data>\r\n
+
+ - <bytes> is the size of the following data section in bytes.
+
+ - <data> is a sequence of bytes of length <bytes> from the previous line. It
+   is a YAML file containing watched tube names as a list of strings.
+
+The quit command simply closes the connection. Its form is:
+
+quit\r\n
+
+The pause-tube command can delay any new job being reserved for a given time. Its form is:
+
+pause-tube <tube-name> <delay>\r\n
+
+ - <tube> is the tube to pause
+
+ - <delay> is an integer number of seconds to wait before reserving any more
+   jobs from the queue
+
+There are two possible responses:
+
+ - "PAUSED\r\n" to indicate success.
+
+ - "NOT_FOUND\r\n" if the tube does not exist.

File setup.py

View file
  • Ignore whitespace
+from setuptools import setup
+
+setup(name='beanstalktc',
+      version='0.1.0',
+      description='An async beanstalkd client for Tornado',
+      author='Jacob Sondergaard',
+      author_email='jacob@nephics.com',
+      license="http://www.apache.org/licenses/LICENSE-2.0",
+      url='https://bitbucket.org/nephics/beanstalktc',
+      packages=['beanstalktc'],
+      requires=['tornado(>=2.4)']
+)