1. Joe Heck
  2. superlance

Commits

Mike Naberezny  committed 567747e

[svn r866] Tagged release 0.5.

  • Participants
  • Parent commits 7b45e6b
  • Branches superlance

Comments (0)

Files changed (11)

File tags/0.5/CHANGES.txt

View file
  • Ignore whitespace
+superlance Changelog
+====================
+
+0.5 (2009-05-24)
+
+- Added ``memmon`` script, originally bundled with supervisor and
+  now moved to superlance.
+
+0.4 (2009-02-11)
+----------------
+
+- Added ``eager`` and ``not-eager`` options to httpok.  If ``not-eager``
+  is set, and no process being monitored is in the RUNNING state, skip
+  the URL check / mail message.
+
+0.3 (2008-12-10)
+----------------
+
+- Added ``gcore`` and ``coredir`` options to httpok.
+
+0.2 (2008-11-21)
+----------------
+
+- Added crashmail script.
+
+0.1 (2008-09-18)
+----------------
+
+- Initial release

File tags/0.5/README.txt

View file
  • Ignore whitespace
+superlance plugins for supervisor
+=================================
+
+Superlance is a package of plugin utilities for monitoring and
+controlling processes that run under `supervisor
+<http://supervisord.org>`_.
+
+Currently, it provides two scripts:
+
+``httpok`` -- This script can be used as a supervisor event listener
+(subscribed to TICK events) which will restart a "hung" HTTP server
+process, which is defined as a process in the RUNNING state which does
+not respond in an appropriate or timely manner to an HTTP GET request.
+
+``crashmail`` -- This script will email a user when a process enters
+the EXITED state unexpectedly.
+
+``memmon`` -- See the description below.
+
+
+Memmon Overview
+---------------
+
+memmon is a supervisor "event listener" which may be subscribed to a
+concrete TICK_x event. When memmon receives a TICK_x event (TICK_60 is
+recommended, indicating activity every 60 seconds), memmon checks that a
+configurable list of programs (or all programs running under supervisor) are
+not exceeding a configurable about of memory (resident segment size, or RSS).
+If one or more of these processes is consuming more than the amount of memory
+that memmon believes it should, memmon will restart the process. memmon can be
+configured to send an email notification when it restarts a process.
+
+memmon is known to work on Linux and Mac OS X, but has not been tested on
+other operating systems (it relies on ps output and command-line switches).
+
+memmon is incapable of monitoring the process status of processes which are
+not supervisord child processes.
+
+Memmon Command
+--------------
+
+memmon is a "console script" installed when you install supervisor. Although
+memmon is an executable program, it isn't useful as a general-purpose script:
+it must be run as a supervisor event listener to do anything useful. memmon
+accepts the following options:
+
+  Option       Argument(s)      Description
+  -----------  ---------------  ----------------------
+  -h           N/A              Show program help.
+  --help
+
+  -p           name/size pair   A name/size pair, e.g. "foo=1MB". The name 
+  --program                     represents the supervisor program name that 
+                                you'd like memmon to monitor, the size represents
+                                the number of bytes (suffix-multiplied using "KB", 
+                                "MB" or "GB") that should be considered "too much".
+                                Multiple -p options can be provided to memmon to
+                                signify that you'd like to monitor more than one 
+                                program. Programs can be specified as a "namespec",
+                                to disambiguate same-named programs in different
+                                groups, e.g. "foo:bar" represents the program "bar"
+                                in the "foo" group.
+
+  -g           name/size pair   A groupname/size pair, e.g. "group=1MB". The name
+  --groupname                   represents the supervisor group name that you'd
+                                like memmon to monitor, the size represents the
+                                number of bytes (suffix-multiplied using "KB", "MB"
+                                or "GB") that should be considered "too much". 
+                                Multiple -g options can be provided to memmon to
+                                signify that you'd like to monitor more than one
+                                group.  If any process in this group exceeds the 
+                                maximum, it will be restarted.   
+
+  -a           size             A size (suffix-multiplied using "KB", "MB" or "GB") 
+  --any                         that should be considered "too much". If any program
+                                running as a child of supervisor exceeds this maximum,
+	 	                            it will be restarted. E.g. 100MB. 
+
+  -s           command          A command that will send mail if passed the email 
+  --sendmail                    body (including the headers).  Defaults to
+    _program                    "/usr/sbin/sendmail -t -i". Specifying this doesn't 
+                                cause memmon to send mail by itself (see the 
+                                -m/--email option). 
+
+  -m           email address    An email address to which to send email when a process
+  --email                       is restarted. By default, memmon will not send any mail
+                                unless an email address is specified. 
+
+Configuring Memmon Into the Supervisor Config
+---------------------------------------------
+
+An [eventlistener:x] section must be placed in supervisord.conf in order for
+memmon to begin working. See the "Events" chapter in the Supervisor manual
+for more information about event listeners. The following examples assume that
+the memmon is on your system PATH.
+
+    memmon Example Configuration 1
+
+        This configuration causes memmon to restart any process which is a
+        child of supervisord consuming more than 200MB of RSS, and will send
+        mail to bob@example.com when it restarts a process using the default
+        sendmail command.
+
+        [eventlistener:memmon]
+        command=memmon -a 200MB -m bob@example.com
+        events=TICK_60
+
+    memmon Example Configuration 2
+
+        This configuration causes memmon to restart any process with the
+        supervisor program name "foo" consuming more than 200MB of RSS, and
+        will send mail to bob@example.com when it restarts a process using the
+        default sendmail command.
+
+        [eventlistener:memmon]
+        command=memmon -p foo=200MB -m bob@example.com
+        events=TICK_60
+
+    memmon Example Configuration 3            
+
+        This configuration causes memmon to restart any process in the process
+        group "bar" consuming more than 200MB of RSS, and will send mail to
+        bob@example.com when it restarts a process using the default sendmail
+        command.
+
+        [eventlistener:memmon]
+        command=memmon -g bar=200MB -m bob@example.com
+        events=TICK_60

File tags/0.5/ez_setup.py

View file
  • Ignore whitespace
+#!python
+"""Bootstrap setuptools installation
+
+If you want to use setuptools in your package's setup.py, just include this
+file in the same directory with it, and add this to the top of your setup.py::
+
+    from ez_setup import use_setuptools
+    use_setuptools()
+
+If you want to require a specific version of setuptools, set a download
+mirror, or use an alternate download directory, you can do so by supplying
+the appropriate options to ``use_setuptools()``.
+
+This file can also be run as a script to install or upgrade setuptools.
+"""
+import sys
+DEFAULT_VERSION = "0.6c9"
+DEFAULT_URL     = "http://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3]
+
+md5_data = {
+    'setuptools-0.6b1-py2.3.egg': '8822caf901250d848b996b7f25c6e6ca',
+    'setuptools-0.6b1-py2.4.egg': 'b79a8a403e4502fbb85ee3f1941735cb',
+    'setuptools-0.6b2-py2.3.egg': '5657759d8a6d8fc44070a9d07272d99b',
+    'setuptools-0.6b2-py2.4.egg': '4996a8d169d2be661fa32a6e52e4f82a',
+    'setuptools-0.6b3-py2.3.egg': 'bb31c0fc7399a63579975cad9f5a0618',
+    'setuptools-0.6b3-py2.4.egg': '38a8c6b3d6ecd22247f179f7da669fac',
+    'setuptools-0.6b4-py2.3.egg': '62045a24ed4e1ebc77fe039aa4e6f7e5',
+    'setuptools-0.6b4-py2.4.egg': '4cb2a185d228dacffb2d17f103b3b1c4',
+    'setuptools-0.6c1-py2.3.egg': 'b3f2b5539d65cb7f74ad79127f1a908c',
+    'setuptools-0.6c1-py2.4.egg': 'b45adeda0667d2d2ffe14009364f2a4b',
+    'setuptools-0.6c2-py2.3.egg': 'f0064bf6aa2b7d0f3ba0b43f20817c27',
+    'setuptools-0.6c2-py2.4.egg': '616192eec35f47e8ea16cd6a122b7277',
+    'setuptools-0.6c3-py2.3.egg': 'f181fa125dfe85a259c9cd6f1d7b78fa',
+    'setuptools-0.6c3-py2.4.egg': 'e0ed74682c998bfb73bf803a50e7b71e',
+    'setuptools-0.6c3-py2.5.egg': 'abef16fdd61955514841c7c6bd98965e',
+    'setuptools-0.6c4-py2.3.egg': 'b0b9131acab32022bfac7f44c5d7971f',
+    'setuptools-0.6c4-py2.4.egg': '2a1f9656d4fbf3c97bf946c0a124e6e2',
+    'setuptools-0.6c4-py2.5.egg': '8f5a052e32cdb9c72bcf4b5526f28afc',
+    'setuptools-0.6c5-py2.3.egg': 'ee9fd80965da04f2f3e6b3576e9d8167',
+    'setuptools-0.6c5-py2.4.egg': 'afe2adf1c01701ee841761f5bcd8aa64',
+    'setuptools-0.6c5-py2.5.egg': 'a8d3f61494ccaa8714dfed37bccd3d5d',
+    'setuptools-0.6c6-py2.3.egg': '35686b78116a668847237b69d549ec20',
+    'setuptools-0.6c6-py2.4.egg': '3c56af57be3225019260a644430065ab',
+    'setuptools-0.6c6-py2.5.egg': 'b2f8a7520709a5b34f80946de5f02f53',
+    'setuptools-0.6c7-py2.3.egg': '209fdf9adc3a615e5115b725658e13e2',
+    'setuptools-0.6c7-py2.4.egg': '5a8f954807d46a0fb67cf1f26c55a82e',
+    'setuptools-0.6c7-py2.5.egg': '45d2ad28f9750e7434111fde831e8372',
+    'setuptools-0.6c8-py2.3.egg': '50759d29b349db8cfd807ba8303f1902',
+    'setuptools-0.6c8-py2.4.egg': 'cba38d74f7d483c06e9daa6070cce6de',
+    'setuptools-0.6c8-py2.5.egg': '1721747ee329dc150590a58b3e1ac95b',
+    'setuptools-0.6c9-py2.3.egg': 'a83c4020414807b496e4cfbe08507c03',
+    'setuptools-0.6c9-py2.4.egg': '260a2be2e5388d66bdaee06abec6342a',
+    'setuptools-0.6c9-py2.5.egg': 'fe67c3e5a17b12c0e7c541b7ea43a8e6',
+    'setuptools-0.6c9-py2.6.egg': 'ca37b1ff16fa2ede6e19383e7b59245a',
+}
+
+import sys, os
+try: from hashlib import md5
+except ImportError: from md5 import md5
+
+def _validate_md5(egg_name, data):
+    if egg_name in md5_data:
+        digest = md5(data).hexdigest()
+        if digest != md5_data[egg_name]:
+            print >>sys.stderr, (
+                "md5 validation of %s failed!  (Possible download problem?)"
+                % egg_name
+            )
+            sys.exit(2)
+    return data
+
+def use_setuptools(
+    version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
+    download_delay=15
+):
+    """Automatically find/download setuptools and make it available on sys.path
+
+    `version` should be a valid setuptools version number that is available
+    as an egg for download under the `download_base` URL (which should end with
+    a '/').  `to_dir` is the directory where setuptools will be downloaded, if
+    it is not already available.  If `download_delay` is specified, it should
+    be the number of seconds that will be paused before initiating a download,
+    should one be required.  If an older version of setuptools is installed,
+    this routine will print a message to ``sys.stderr`` and raise SystemExit in
+    an attempt to abort the calling script.
+    """
+    was_imported = 'pkg_resources' in sys.modules or 'setuptools' in sys.modules
+    def do_download():
+        egg = download_setuptools(version, download_base, to_dir, download_delay)
+        sys.path.insert(0, egg)
+        import setuptools; setuptools.bootstrap_install_from = egg
+    try:
+        import pkg_resources
+    except ImportError:
+        return do_download()       
+    try:
+        pkg_resources.require("setuptools>="+version); return
+    except pkg_resources.VersionConflict, e:
+        if was_imported:
+            print >>sys.stderr, (
+            "The required version of setuptools (>=%s) is not available, and\n"
+            "can't be installed while this script is running. Please install\n"
+            " a more recent version first, using 'easy_install -U setuptools'."
+            "\n\n(Currently using %r)"
+            ) % (version, e.args[0])
+            sys.exit(2)
+        else:
+            del pkg_resources, sys.modules['pkg_resources']    # reload ok
+            return do_download()
+    except pkg_resources.DistributionNotFound:
+        return do_download()
+
+def download_setuptools(
+    version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
+    delay = 15
+):
+    """Download setuptools from a specified location and return its filename
+
+    `version` should be a valid setuptools version number that is available
+    as an egg for download under the `download_base` URL (which should end
+    with a '/'). `to_dir` is the directory where the egg will be downloaded.
+    `delay` is the number of seconds to pause before an actual download attempt.
+    """
+    import urllib2, shutil
+    egg_name = "setuptools-%s-py%s.egg" % (version,sys.version[:3])
+    url = download_base + egg_name
+    saveto = os.path.join(to_dir, egg_name)
+    src = dst = None
+    if not os.path.exists(saveto):  # Avoid repeated downloads
+        try:
+            from distutils import log
+            if delay:
+                log.warn("""
+---------------------------------------------------------------------------
+This script requires setuptools version %s to run (even to display
+help).  I will attempt to download it for you (from
+%s), but
+you may need to enable firewall access for this script first.
+I will start the download in %d seconds.
+
+(Note: if this machine does not have network access, please obtain the file
+
+   %s
+
+and place it in this directory before rerunning this script.)
+---------------------------------------------------------------------------""",
+                    version, download_base, delay, url
+                ); from time import sleep; sleep(delay)
+            log.warn("Downloading %s", url)
+            src = urllib2.urlopen(url)
+            # Read/write all in one block, so we don't create a corrupt file
+            # if the download is interrupted.
+            data = _validate_md5(egg_name, src.read())
+            dst = open(saveto,"wb"); dst.write(data)
+        finally:
+            if src: src.close()
+            if dst: dst.close()
+    return os.path.realpath(saveto)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+def main(argv, version=DEFAULT_VERSION):
+    """Install or upgrade setuptools and EasyInstall"""
+    try:
+        import setuptools
+    except ImportError:
+        egg = None
+        try:
+            egg = download_setuptools(version, delay=0)
+            sys.path.insert(0,egg)
+            from setuptools.command.easy_install import main
+            return main(list(argv)+[egg])   # we're done here
+        finally:
+            if egg and os.path.exists(egg):
+                os.unlink(egg)
+    else:
+        if setuptools.__version__ == '0.0.1':
+            print >>sys.stderr, (
+            "You have an obsolete version of setuptools installed.  Please\n"
+            "remove it from your system entirely before rerunning this script."
+            )
+            sys.exit(2)
+
+    req = "setuptools>="+version
+    import pkg_resources
+    try:
+        pkg_resources.require(req)
+    except pkg_resources.VersionConflict:
+        try:
+            from setuptools.command.easy_install import main
+        except ImportError:
+            from easy_install import main
+        main(list(argv)+[download_setuptools(delay=0)])
+        sys.exit(0) # try to force an exit
+    else:
+        if argv:
+            from setuptools.command.easy_install import main
+            main(argv)
+        else:
+            print "Setuptools version",version,"or greater has been installed."
+            print '(Run "ez_setup.py -U setuptools" to reinstall or upgrade.)'
+
+def update_md5(filenames):
+    """Update our built-in md5 registry"""
+
+    import re
+
+    for name in filenames:
+        base = os.path.basename(name)
+        f = open(name,'rb')
+        md5_data[base] = md5(f.read()).hexdigest()
+        f.close()
+
+    data = ["    %r: %r,\n" % it for it in md5_data.items()]
+    data.sort()
+    repl = "".join(data)
+
+    import inspect
+    srcfile = inspect.getsourcefile(sys.modules[__name__])
+    f = open(srcfile, 'rb'); src = f.read(); f.close()
+
+    match = re.search("\nmd5_data = {\n([^}]+)}", src)
+    if not match:
+        print >>sys.stderr, "Internal error!"
+        sys.exit(2)
+
+    src = src[:match.start(1)] + repl + src[match.end(1):]
+    f = open(srcfile,'w')
+    f.write(src)
+    f.close()
+
+
+if __name__=='__main__':
+    if len(sys.argv)>2 and sys.argv[1]=='--md5update':
+        update_md5(sys.argv[2:])
+    else:
+        main(sys.argv[1:])
+
+
+
+
+
+

File tags/0.5/setup.py

View file
  • Ignore whitespace
+import os
+
+from ez_setup import use_setuptools
+use_setuptools()
+
+from setuptools import setup, find_packages
+
+here = os.path.abspath(os.path.dirname(__file__))
+try:
+    README = open(os.path.join(here, 'README.txt')).read()
+except (IOError, OSError):
+    README = ''
+try:
+    CHANGES = open(os.path.join(here, 'CHANGES.txt')).read()
+except (IOError, OSError):
+    CHANGES = ''
+
+setup(name='superlance',
+      version='0.5',
+      description='superlance plugins for supervisord',
+      long_description=README + '\n\n' +  CHANGES,
+      classifiers=[
+        "Development Status :: 3 - Alpha",
+        'Environment :: No Input/Output (Daemon)',
+        'Intended Audience :: System Administrators',
+        'Natural Language :: English',
+        'Operating System :: POSIX',
+        'Topic :: System :: Boot',
+        'Topic :: System :: Monitoring',
+        'Topic :: System :: Systems Administration',
+        ],
+      author='Chris McDonough',
+      author_email='chrism@plope.com',
+      url='http://supervisord.org',
+      keywords = 'supervisor monitoring',
+      packages = find_packages(),
+      include_package_data=True,
+      zip_safe=False,
+      install_requires=[
+            'setuptools',
+            'supervisor',
+            ],
+      tests_require=[
+            'supervisor',
+            ],
+      test_suite="superlance",
+      entry_points = """\
+      [console_scripts]
+      httpok = superlance.httpok:main
+      crashmail = superlance.crashmail:main
+      memmon = superlance.memmon:main
+      """
+      )
+

File tags/0.5/superlance/__init__.py

View file
  • Ignore whitespace
+# superlance package

File tags/0.5/superlance/crashmail.py

View file
  • Ignore whitespace
+#!/usr/bin/env python -u
+##############################################################################
+#
+# Copyright (c) 2007 Agendaless Consulting and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the BSD-like license at
+# http://www.repoze.org/LICENSE.txt.  A copy of the license should accompany
+# this distribution.  THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL
+# EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND
+# FITNESS FOR A PARTICULAR PURPOSE
+#
+##############################################################################
+
+# A event listener meant to be subscribed to PROCESS_STATE_CHANGE
+# events.  It will send mail when processes that are children of
+# supervisord transition unexpectedly to the EXITED state.
+
+# A supervisor config snippet that tells supervisor to use this script
+# as a listener is below.
+#
+# [eventlistener:crashmail]
+# command=python -u /bin/paster serve myserver.ini
+# events=PROCESS_STATE
+
+doc = """\
+crashmail.py [-p processname] [-a] [-o string] [-m mail_address]
+             [-s sendmail] URL
+
+Options:
+
+-p -- specify a supervisor process_name.  Send mail when this process
+      transitions to the EXITED state unexpectedly. If this process is
+      part of a group, it can be specified using the
+      'process_name:group_name' syntax.
+      
+-a -- Send mail when any child of the supervisord transitions
+      unexpectedly to the EXITED state unexpectedly.  Overrides any -p
+      parameters passed in the same crashmail process invocation.
+
+-o -- Specify a parameter used as a prefix in the mail subject header.
+
+-s -- the sendmail command to use to send email
+      (e.g. "/usr/sbin/sendmail -t -i").  Must be a command which accepts
+      header and message data on stdin and sends mail.  Default is
+      "/usr/sbin/sendmail -t -i".
+
+-m -- specify an email address.  The script will send mail to this
+      address when crashmail detects a process crash.  If no email
+      address is specified, email will not be sent.
+
+The -p option may be specified more than once, allowing for
+specification of multiple processes.  Specifying -a overrides any
+selection of -p.
+
+A sample invocation:
+
+crashmail.py -p program1 -p group1:program2 -m dev@example.com
+
+"""
+
+import os
+import sys
+
+from supervisor import childutils
+
+def usage():
+    print doc
+    sys.exit(255)
+
+class CrashMail:
+
+    def __init__(self, programs, any, email, sendmail, optionalheader):
+
+        self.programs = programs
+        self.any = any
+        self.email = email
+        self.sendmail = sendmail
+        self.optionalheader = optionalheader
+        self.stdin = sys.stdin
+        self.stdout = sys.stdout
+        self.stderr = sys.stderr
+
+    def runforever(self, test=False):
+        while 1:
+            # we explicitly use self.stdin, self.stdout, and self.stderr
+            # instead of sys.* so we can unit test this code
+            headers, payload = childutils.listener.wait(self.stdin, self.stdout)
+
+            if not headers['eventname'] == 'PROCESS_STATE_EXITED':
+                # do nothing with non-TICK events
+                childutils.listener.ok(self.stdout)
+                if test:
+                    self.stderr.write('non-exited event\n')
+                    self.stderr.flush()
+                    break
+                continue
+
+            pheaders, pdata = childutils.eventdata(payload+'\n')
+
+            if int(pheaders['expected']):
+                childutils.listener.ok(self.stdout)
+                if test:
+                    self.stderr.write('expected exit\n')
+                    self.stderr.flush()
+                    break
+                continue
+
+            msg = ('Process %(processname)s in group %(groupname)s exited '
+                   'unexpectedly (pid %(pid)s) from state %(from_state)s' %
+                   pheaders)
+            
+            subject = ' %s crashed at %s' % (pheaders['processname'],
+                                             childutils.get_asctime())
+            if self.optionalheader:
+                subject = self.optionalheader + ':' + subject
+
+            self.stderr.write('unexpected exit, mailing\n')
+            self.stderr.flush()
+
+            self.mail(self.email, subject, msg)
+
+            childutils.listener.ok(self.stdout)
+            if test:
+                break
+
+    def mail(self, email, subject, msg):
+        body =  'To: %s\n' % self.email
+        body += 'Subject: %s\n' % subject
+        body += '\n'
+        body += msg
+        m = os.popen(self.sendmail, 'w')
+        m.write(body)
+        m.close()
+        self.stderr.write('Mailed:\n\n%s' % body)
+        self.mailed = body
+
+def main(argv=sys.argv):
+    import getopt
+    short_args="hp:ao:s:m:"
+    long_args=[
+        "help",
+        "program=",
+        "any",
+        "optionalheader="
+        "sendmail_program=",
+        "email=",
+        ]
+    arguments = argv[1:]
+    try:
+        opts, args = getopt.getopt(arguments, short_args, long_args)
+    except:
+        usage()
+
+    programs = []
+    any = False
+    sendmail = '/usr/sbin/sendmail -t -i'
+    email = None
+    timeout = 10
+    status = '200'
+    inbody = None
+    optionalheader = None
+
+    for option, value in opts:
+
+        if option in ('-h', '--help'):
+            usage()
+
+        if option in ('-p', '--program'):
+            programs.append(value)
+
+        if option in ('-a', '--any'):
+            any = True
+
+        if option in ('-s', '--sendmail_program'):
+            sendmail = value
+
+        if option in ('-m', '--email'):
+            email = value
+
+        if option in ('-o', '--optionalheader'):
+            optionalheader = value
+
+    url = arguments[-1]
+
+    if not 'SUPERVISOR_SERVER_URL' in os.environ:
+        sys.stderr.write('httpok must be run as a supervisor event '
+                         'listener\n')
+        sys.stderr.flush()
+        return
+        
+    prog = CrashMail(programs, any, email, sendmail, optionalheader)
+    prog.runforever()
+
+if __name__ == '__main__':
+    main()
+    
+    
+    

File tags/0.5/superlance/grower.py

View file
  • Ignore whitespace
+#!/usr/bin/env python
+##############################################################################
+#
+# Copyright (c) 2007 Agendaless Consulting and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the BSD-like license at
+# http://www.repoze.org/LICENSE.txt.  A copy of the license should accompany
+# this distribution.  THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL
+# EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND
+# FITNESS FOR A PARTICULAR PURPOSE
+#
+##############################################################################
+
+# A process which leaks 1MB per second on purpose so we can test the
+# memmon killer
+
+import time
+L = []
+M = pow(2, 20)
+
+while 1:
+    L.append('x'*M)
+    time.sleep(1)
+    

File tags/0.5/superlance/httpok.py

View file
  • Ignore whitespace
+#!/usr/bin/env python -u
+##############################################################################
+#
+# Copyright (c) 2007 Agendaless Consulting and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the BSD-like license at
+# http://www.repoze.org/LICENSE.txt.  A copy of the license should accompany
+# this distribution.  THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL
+# EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND
+# FITNESS FOR A PARTICULAR PURPOSE
+#
+##############################################################################
+
+# A event listener meant to be subscribed to TICK_60 (or TICK_5)
+# events, which restarts processes that are children of
+# supervisord based on the response from an HTTP port.
+
+# A supervisor config snippet that tells supervisor to use this script
+# as a listener is below.
+#
+# [eventlistener:httpok]
+# command=python -u /bin/httpok http://localhost:8080/tasty/service
+# events=TICK_60
+
+doc = """\
+httpok.py [-p processname] [-a] [-g] [-t timeout] [-c status_code] [-b inbody]
+          [-m mail_address] [-s sendmail] URL
+
+Options:
+
+-p -- specify a supervisor process_name.  Restart the supervisor
+      process named 'process_name' if it's in the RUNNING state when
+      the URL returns an unexpected result or times out.  If this
+      process is part of a group, it can be specified using the
+      'process_name:group_name' syntax.
+      
+-a -- Restart any child of the supervisord under in the RUNNING state
+      if the URL returns an unexpected result or times out.  Overrides
+      any -p parameters passed in the same httpok process
+      invocation.
+
+-g -- The ``gcore`` program.  By default, this is ``/usr/bin/gcore
+      -o``.  The program should accept two arguments on the command
+      line: a filename and a pid.
+
+-d -- Core directory.  If a core directory is specified, httpok will
+      try to use the ``gcore`` program (see ``-g``) to write a core
+      file into this directory against each hung process before we
+      restart it.  Append gcore stdout output to email.
+
+-t -- The number of seconds that httpok should wait for a response
+      before timing out.  If this timeout is exceeded, httpok will
+      attempt to restart processes in the RUNNING state specified by
+      -p or -a.  This defaults to 10 seconds.
+
+-c -- specify an expected HTTP status code from a GET request to the
+      URL.  If this status code is not the status code provided by the
+      response, httpok will attempt to restart processes in the
+      RUNNING state specified by -p or -a.  This defaults to the
+      string, "200".
+
+-b -- specify a string which should be present in the body resulting
+      from the GET request.  If this string is not present in the
+      response, the processes in the RUNNING state specified by -p
+      or -a will be restarted.  The default is to ignore the
+      body.
+
+-s -- the sendmail command to use to send email
+      (e.g. "/usr/sbin/sendmail -t -i").  Must be a command which accepts
+      header and message data on stdin and sends mail.
+      Default is "/usr/sbin/sendmail -t -i".
+
+-m -- specify an email address.  The script will send mail to this
+      address when httpok attempts to restart processes.  If no email
+      address is specified, email will not be sent.
+
+-e -- "eager":  check URL / emit mail even if no process we are monitoring
+      is in the RUNNING state.  Enabled by default.
+
+-E -- not "eager":  do not check URL / emit mail if no process we are
+      monitoring is in the RUNNING state.
+
+URL -- The URL to which to issue a GET request.
+
+The -p option may be specified more than once, allowing for
+specification of multiple processes.  Specifying -a overrides any
+selection of -p.
+
+A sample invocation:
+
+httpok.py -p program1 -p group1:program2 http://localhost:8080/tasty
+
+"""
+
+import os
+import sys
+import time
+import urlparse
+import xmlrpclib
+
+from supervisor import childutils
+from supervisor.states import ProcessStates
+from supervisor.options import make_namespec
+
+import timeoutconn
+
+def usage():
+    print doc
+    sys.exit(255)
+
+class HTTPOk:
+    connclass = None
+    def __init__(self, rpc, programs, any, url, timeout, status, inbody,
+                 email, sendmail, coredir, gcore, eager):
+        self.rpc = rpc
+        self.programs = programs
+        self.any = any
+        self.url = url
+        self.timeout = timeout
+        self.status = status
+        self.inbody = inbody
+        self.email = email
+        self.sendmail = sendmail
+        self.coredir = coredir
+        self.gcore = gcore
+        self.eager = eager
+        self.stdin = sys.stdin
+        self.stdout = sys.stdout
+        self.stderr = sys.stderr
+
+    def listProcesses(self, state=None):
+        return [x for x in self.rpc.supervisor.getAllProcessInfo()
+                   if x['name'] in self.programs and
+                      (state is None or x['state'] == state)]
+
+    def runforever(self, test=False):
+        parsed = urlparse.urlsplit(self.url)
+        scheme = parsed[0].lower()
+        hostport = parsed[1]
+        path = parsed[2]
+        params = parsed[3]
+        query = parsed[4]
+
+        if self.connclass:
+            ConnClass = self.connclass
+        elif scheme == 'http':
+            ConnClass = timeoutconn.TimeoutHTTPConnection
+        elif scheme == 'https':
+            ConnClass = timeoutconn.TimeoutHTTPSConnection
+        else:
+            raise ValueError('Bad scheme %s' % scheme)
+
+        while 1:
+            # we explicitly use self.stdin, self.stdout, and self.stderr
+            # instead of sys.* so we can unit test this code
+            headers, payload = childutils.listener.wait(self.stdin, self.stdout)
+
+            if not headers['eventname'].startswith('TICK'):
+                # do nothing with non-TICK events
+                childutils.listener.ok(self.stdout)
+                if test:
+                    break
+                continue
+
+            conn = ConnClass(hostport)
+            conn.timeout = self.timeout
+
+            if query:
+                path += '?' + query
+
+            act = False
+
+            specs = self.listProcesses(ProcessStates.RUNNING)
+            if self.eager or len(specs) > 0:
+
+                try:
+                    conn.request('GET', path)
+                    res = conn.getresponse()
+                    body = res.read()
+                    status = res.status
+                    msg = 'status contacting %s: %s %s' % (self.url,
+                                                           res.status,
+                                                           res.reason)
+                except Exception, why:
+                    body = ''
+                    status = None
+                    msg = 'error contacting %s:\n\n %s' % (self.url, why)
+
+                if str(status) != str(self.status):
+                    subject = 'httpok for %s: bad status returned' % self.url
+                    self.act(subject, msg)
+                elif self.inbody and self.inbody not in res.body:
+                    act = True
+                    subject = 'httpok for %s: bad body returned' % self.url
+                    self.act(subject, msg)
+
+            childutils.listener.ok(self.stdout)
+            if test:
+                break
+
+    def act(self, subject, msg):
+        messages = [msg]
+
+        def write(msg):
+            self.stderr.write('%s\n' % msg)
+            self.stderr.flush()
+            messages.append(msg)
+
+        try:
+            specs = self.rpc.supervisor.getAllProcessInfo()
+        except Exception, why:
+            write('Exception retrieving process info %s, not acting' % why)
+            return
+            
+        waiting = list(self.programs)
+            
+        if self.any:
+            write('Restarting all running processes')
+            for spec in specs:
+                name = spec['name']
+                group = spec['group']
+                self.restart(spec, write)
+                namespec = make_namespec(name, group)
+                if name in waiting:
+                    waiting.remove(name)
+                if namespec in waiting:
+                    waiting.remove(namespec)
+        else:
+            write('Restarting selected processes %s' % self.programs)
+            for spec in specs:
+                name = spec['name']
+                group = spec['group']
+                namespec = make_namespec(name, group)
+                if (name in self.programs) or (namespec in self.programs):
+                    self.restart(spec, write)
+                    if name in waiting:
+                        waiting.remove(name)
+                    if namespec in waiting:
+                        waiting.remove(namespec)
+
+        if waiting:
+            write(
+                'Programs not restarted because they did not exist: %s' %
+                waiting)
+
+        if self.email:
+            now = time.asctime()
+            message = '\n'.join(messages)
+            self.mail(self.email, subject, message)
+
+    def mail(self, email, subject, msg):
+        body =  'To: %s\n' % self.email
+        body += 'Subject: %s\n' % subject
+        body += '\n'
+        body += msg
+        m = os.popen(self.sendmail, 'w')
+        m.write(body)
+        m.close()
+        self.stderr.write('Mailed:\n\n%s' % body)
+        self.mailed = body
+
+    def restart(self, spec, write):
+        namespec = make_namespec(spec['group'], spec['name'])
+        if spec['state'] is ProcessStates.RUNNING:
+            if self.coredir and self.gcore:
+                corename = os.path.join(self.coredir, namespec)
+                m = os.popen(self.gcore + ' "%s" %s' % (corename, spec['pid']))
+                write('gcore output for %s:\n\n %s' % (namespec, m.read()))
+                m.close()
+            write('%s is in RUNNING state, restarting' % namespec)
+            try:
+                self.rpc.supervisor.stopProcess(namespec)
+            except xmlrpclib.Fault, what:
+                write('Failed to stop process %s: %s' % (
+                    namespec, what))
+
+            try:
+                self.rpc.supervisor.startProcess(namespec)
+            except xmlrpclib.Fault, what:
+                write('Failed to start process %s: %s' % (
+                    namespec, what))
+            else:
+                write('%s restarted' % namespec)
+
+        else:
+            write('%s not in RUNNING state, NOT restarting' % namespec)
+            
+
+def main(argv=sys.argv):
+    import getopt
+    short_args="hp:at:c:b:s:m:g:d:eE"
+    long_args=[
+        "help",
+        "program=",
+        "any",
+        "timeout=",
+        "code=",
+        "body=",
+        "sendmail_program=",
+        "email=",
+        "gcore=",
+        "coredir=",
+        "eager",
+        "not-eager",
+        ]
+    arguments = argv[1:]
+    try:
+        opts, args = getopt.getopt(arguments, short_args, long_args)
+    except:
+        usage()
+
+    if not args:
+        usage()
+    if len(args) > 1:
+        usage()
+
+    programs = []
+    any = False
+    sendmail = '/usr/sbin/sendmail -t -i'
+    gcore = '/usr/bin/gcore -o'
+    coredir = None
+    eager = True
+    email = None
+    timeout = 10
+    status = '200'
+    inbody = None
+
+    for option, value in opts:
+
+        if option in ('-h', '--help'):
+            usage()
+
+        if option in ('-p', '--program'):
+            programs.append(value)
+
+        if option in ('-a', '--any'):
+            any = True
+
+        if option in ('-s', '--sendmail_program'):
+            sendmail = value
+
+        if option in ('-m', '--email'):
+            email = value
+
+        if option in ('-t', '--timeout'):
+            timeout = int(value)
+
+        if option in ('-c', '--code'):
+            status = value
+
+        if option in ('-b', '--body'):
+            inbody = value
+
+        if option in ('-g', '--gcore'):
+            gcore = value
+
+        if option in ('-d', '--coredir'):
+            coredir = value
+
+        if option in ('-e', '--eager'):
+            eager = True
+
+        if option in ('-E', '--not-eager'):
+            eager = False
+
+    url = arguments[-1]
+
+    try:
+        rpc = childutils.getRPCInterface(os.environ)
+    except KeyError, why:
+        if why[0] != 'SUPERVISOR_SERVER_URL':
+            raise
+        sys.stderr.write('httpok must be run as a supervisor event '
+                         'listener\n')
+        sys.stderr.flush()
+        return
+
+    prog = HTTPOk(rpc, programs, any, url, timeout, status, inbody, email,
+                  sendmail, coredir, gcore, eager)
+    prog.runforever()
+
+if __name__ == '__main__':
+    main()
+    
+    
+    

File tags/0.5/superlance/memmon.py

View file
  • Ignore whitespace
+#!/usr/bin/env python -u
+##############################################################################
+#
+# Copyright (c) 2007 Agendaless Consulting and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the BSD-like license at
+# http://www.repoze.org/LICENSE.txt.  A copy of the license should accompany
+# this distribution.  THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL
+# EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND
+# FITNESS FOR A PARTICULAR PURPOSE
+#
+##############################################################################
+
+# A event listener meant to be subscribed to TICK_60 (or TICK_5)
+# events, which restarts any processes that are children of
+# supervisord that consume "too much" memory.  Performs horrendous
+# screenscrapes of ps output.  Works on Linux and OS X (Tiger/Leopard)
+# as far as I know.
+
+# A supervisor config snippet that tells supervisor to use this script
+# as a listener is below.
+#
+# [eventlistener:memmon]
+# command=python memmon.py [options]
+# events=TICK_60
+
+doc = """\
+memmon.py [-p processname=byte_size]  [-g groupname=byte_size] 
+          [-a byte_size] [-s sendmail] [-m email_address]
+
+Options:
+
+-p -- specify a process_name=byte_size pair.  Restart the supervisor
+      process named 'process_name' when it uses more than byte_size
+      RSS.  If this process is in a group, it can be specified using
+      the 'process_name:group_name' syntax.
+      
+-g -- specify a group_name=byte_size pair.  Restart any process in this group
+      when it uses more than byte_size RSS.
+      
+-a -- specify a global byte_size.  Restart any child of the supervisord
+      under which this runs if it uses more than byte_size RSS.
+
+-s -- the sendmail command to use to send email
+      (e.g. "/usr/sbin/sendmail -t -i").  Must be a command which accepts
+      header and message data on stdin and sends mail.
+      Default is "/usr/sbin/sendmail -t -i".
+
+-m -- specify an email address.  The script will send mail to this
+      address when any process is restarted.  If no email address is
+      specified, email will not be sent.
+
+The -p and -g options may be specified more than once, allowing for
+specification of multiple groups and processes.
+
+Any byte_size can be specified as a plain integer (10000) or a
+suffix-multiplied integer (e.g. 1GB).  Valid suffixes are 'KB', 'MB'
+and 'GB'.
+
+A sample invocation:
+
+memmon.py -p program1=200MB -p theprog:thegroup=100MB -g thegroup=100MB -a 1GB -s "/usr/sbin/sendmail -t -i" -m chrism@plope.com
+"""
+
+import os
+import sys
+import time
+import xmlrpclib
+
+from supervisor import childutils
+from supervisor.datatypes import byte_size
+
+def usage():
+    print doc
+    sys.exit(255)
+
+def shell(cmd):
+    return os.popen(cmd).read()
+
+class Memmon:
+    def __init__(self, programs, groups, any, sendmail, email, rpc):
+        self.programs = programs
+        self.groups = groups
+        self.any = any
+        self.sendmail = sendmail
+        self.email = email
+        self.rpc = rpc
+        self.stdin = sys.stdin
+        self.stdout = sys.stdout
+        self.stderr = sys.stderr
+        self.pscommand = 'ps -orss= -p %s'
+        self.mailed = False # for unit tests
+
+    def runforever(self, test=False):
+        while 1:
+            # we explicitly use self.stdin, self.stdout, and self.stderr
+            # instead of sys.* so we can unit test this code
+            headers, payload = childutils.listener.wait(self.stdin, self.stdout)
+
+            if not headers['eventname'].startswith('TICK'):
+                # do nothing with non-TICK events
+                childutils.listener.ok(self.stdout)
+                if test:
+                    break
+                continue
+
+            status = []
+            if self.programs:
+                status.append(
+                    'Checking programs %s' % ', '.join(
+                    [ '%s=%s' % x for x in self.programs.items() ] )
+                    )
+
+            if self.groups:
+                status.append(
+                    'Checking groups %s' % ', '.join(
+                    [ '%s=%s' % x for x in self.groups.items() ] )
+                    )
+            if self.any is not None:
+                status.append('Checking any=%s' % self.any)
+
+            self.stderr.write('\n'.join(status) + '\n')
+
+            infos = self.rpc.supervisor.getAllProcessInfo()
+
+            for info in infos:
+                pid = info['pid']
+                name = info['name']
+                group = info['group']
+                pname = '%s:%s' % (group, name)
+
+                data = shell(self.pscommand % pid)
+                if not data:
+                    # no such pid (deal with race conditions)
+                    continue
+
+                try:
+                    rss = data.lstrip().rstrip()
+                    rss = int(rss) * 1024 # rss is in KB
+                except ValueError:
+                    # line doesn't contain any data, or rss cant be intified
+                    continue
+
+                for n in name, pname:
+                    if n in self.programs:
+                        self.stderr.write('RSS of %s is %s\n' % (pname, rss))
+                        if  rss > self.programs[name]:
+                            self.restart(pname, rss)
+                            continue
+
+                if group in self.groups:
+                    self.stderr.write('RSS of %s is %s\n' % (pname, rss))
+                    if rss > self.groups[group]:
+                        self.restart(pname, rss)
+                        continue
+
+                if self.any is not None:
+                    self.stderr.write('RSS of %s is %s\n' % (pname, rss))
+                    if rss > self.any:
+                        self.restart(pname, rss)
+                        continue
+
+            self.stderr.flush()
+            childutils.listener.ok(self.stdout)
+            if test:
+                break
+
+    def restart(self, name, rss):
+        self.stderr.write('Restarting %s\n' % name)
+
+        try:
+            self.rpc.supervisor.stopProcess(name)
+        except xmlrpclib.Fault, what:
+            msg = ('Failed to stop process %s (RSS %s), exiting: %s' %
+                   (name, rss, what))
+            self.stderr.write(str(msg))
+            if self.email:
+                subject = 'memmon: failed to stop process %s, exiting' % name
+                self.mail(self.email, subject, msg)
+            raise
+
+        try:
+            self.rpc.supervisor.startProcess(name)
+        except xmlrpclib.Fault, what:
+            msg = ('Failed to start process %s after stopping it, '
+                   'exiting: %s' % (name, what))
+            self.stderr.write(str(msg))
+            if self.email:
+                subject = 'memmon: failed to start process %s, exiting' % name
+                self.mail(self.email, subject, msg)
+            raise
+
+        if self.email:
+            now = time.asctime()
+            msg = (
+                'memmon.py restarted the process named %s at %s because '
+                'it was consuming too much memory (%s bytes RSS)' % (
+                name, now, rss)
+                )
+            subject = 'memmon: process %s restarted' % name
+            self.mail(self.email, subject, msg)
+
+    def mail(self, email, subject, msg):
+        body =  'To: %s\n' % self.email
+        body += 'Subject: %s\n' % subject
+        body += '\n'
+        body += msg
+        m = os.popen(self.sendmail, 'w')
+        m.write(body)
+        m.close()
+        self.mailed = body
+        
+def parse_namesize(option, value):
+    try:
+        name, size = value.split('=')
+    except ValueError:
+        print 'Unparseable value %r for %r' % (value, option)
+        usage()
+    size = parse_size(option, size)
+    return name, size
+
+def parse_size(option, value):
+    try:
+        size = byte_size(value)
+    except:
+        print 'Unparseable byte_size in %r for %r' % (value, option)
+        usage()
+        
+    return size
+
+def main():
+    import getopt
+    short_args="hp:g:a:s:m:"
+    long_args=[
+        "help",
+        "program=",
+        "group=",
+        "any=",
+        "sendmail_program=",
+        "email=",
+        ]
+    arguments = sys.argv[1:]
+    if not arguments:
+        usage()
+    try:
+        opts, args=getopt.getopt(arguments, short_args, long_args)
+    except:
+        print __doc__
+        sys.exit(2)
+
+    programs = {}
+    groups = {}
+    any = None
+    sendmail = '/usr/sbin/sendmail -t -i'
+    email = None
+
+    for option, value in opts:
+
+        if option in ('-h', '--help'):
+            usage()
+
+        if option in ('-p', '--program'):
+            name, size = parse_namesize(option, value)
+            programs[name] = size
+
+        if option in ('-g', '--group'):
+            name, size = parse_namesize(option, value)
+            groups[name] = size
+
+        if option in ('-a', '--any'):
+            size = parse_size(option, value)
+            any = size
+
+        if option in ('-s', '--sendmail_program'):
+            sendmail = value
+
+        if option in ('-m', '--email'):
+            email = value
+
+    rpc = childutils.getRPCInterface(os.environ)
+    memmon = Memmon(programs, groups, any, sendmail, email, rpc)
+    memmon.runforever()
+
+if __name__ == '__main__':
+    main()
+    
+    
+    

File tags/0.5/superlance/tests.py

View file
  • Ignore whitespace
+import sys
+import unittest
+from StringIO import StringIO
+
+class HTTPOkTests(unittest.TestCase):
+    def _getTargetClass(self):
+        from superlance.httpok import HTTPOk
+        return HTTPOk
+    
+    def _makeOne(self, *opts):
+        return self._getTargetClass()(*opts)
+
+    def _makeOnePopulated(self, programs, any, response=None, exc=None,
+                          gcore=None, coredir=None, eager=True):
+        if response is None:
+            response = DummyResponse()
+        rpc = DummyRPCServer()
+        sendmail = 'cat - > /dev/null'
+        email = 'chrism@plope.com'
+        url = 'http://foo/bar'
+        timeout = 10
+        status = '200'
+        inbody = None
+        gcore = gcore
+        coredir = coredir
+        prog = self._makeOne(rpc, programs, any, url, timeout, status,
+                             inbody, email, sendmail, coredir, gcore, eager)
+        prog.stdin = StringIO()
+        prog.stdout = StringIO()
+        prog.stderr = StringIO()
+        prog.connclass = make_connection(response, exc=exc)
+        return prog
+
+    def test_listProcesses_no_programs(self):
+        programs = []
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        specs = list(prog.listProcesses())
+        self.assertEqual(len(specs), 0)
+
+    def test_listProcesses_w_RUNNING_programs_default_state(self):
+        programs = ['foo']
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        specs = list(prog.listProcesses())
+        self.assertEqual(len(specs), 1)
+        self.assertEqual(specs[0],
+                         DummySupervisorRPCNamespace.all_process_info[0])
+
+    def test_listProcesses_w_nonRUNNING_programs_default_state(self):
+        programs = ['bar']
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        specs = list(prog.listProcesses())
+        self.assertEqual(len(specs), 1)
+        self.assertEqual(specs[0],
+                         DummySupervisorRPCNamespace.all_process_info[1])
+
+    def test_listProcesses_w_nonRUNNING_programs_RUNNING_state(self):
+        programs = ['bar']
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        specs = list(prog.listProcesses(ProcessStates.RUNNING))
+        self.assertEqual(len(specs), 0, (prog.programs, specs))
+
+    def test_runforever_eager_notatick(self):
+        programs = {'foo':0, 'bar':0, 'baz_01':0 }
+        groups = {}
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        prog.stdin.write('eventname:NOTATICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        self.assertEqual(prog.stderr.getvalue(), '')
+
+    def test_runforever_eager_error_on_request_some(self):
+        programs = ['foo', 'bar', 'baz_01', 'notexisting']
+        any = None
+        prog = self._makeOnePopulated(programs, any, exc=True)
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = prog.stderr.getvalue().split('\n')
+        #self.assertEqual(len(lines), 7)
+        self.assertEqual(lines[0],
+                         ("Restarting selected processes ['foo', 'bar', "
+                          "'baz_01', 'notexisting']")
+                         )
+        self.assertEqual(lines[1], 'foo is in RUNNING state, restarting')
+        self.assertEqual(lines[2], 'foo restarted')
+        self.assertEqual(lines[3], 'bar not in RUNNING state, NOT restarting')
+        self.assertEqual(lines[4],
+                         'baz:baz_01 not in RUNNING state, NOT restarting')
+        self.assertEqual(lines[5],
+          "Programs not restarted because they did not exist: ['notexisting']")
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 12)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+    def test_runforever_eager_error_on_request_any(self):
+        programs = []
+        any = True
+        prog = self._makeOnePopulated(programs, any, exc=True)
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = prog.stderr.getvalue().split('\n')
+        #self.assertEqual(len(lines), 6)
+        self.assertEqual(lines[0], 'Restarting all running processes')
+        self.assertEqual(lines[1], 'foo is in RUNNING state, restarting')
+        self.assertEqual(lines[2], 'foo restarted')
+        self.assertEqual(lines[3], 'bar not in RUNNING state, NOT restarting')
+        self.assertEqual(lines[4],
+                         'baz:baz_01 not in RUNNING state, NOT restarting')
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 11)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+    def test_runforever_eager_error_on_process_stop(self):
+        programs = ['FAILED']
+        any = False
+        prog = self._makeOnePopulated(programs, any, exc=True)
+        prog.rpc.supervisor.all_process_info = _FAIL
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = prog.stderr.getvalue().split('\n')
+        #self.assertEqual(len(lines), 5)
+        self.assertEqual(lines[0], "Restarting selected processes ['FAILED']")
+        self.assertEqual(lines[1], 'foo:FAILED is in RUNNING state, restarting')
+        self.assertEqual(lines[2],
+                    "Failed to stop process foo:FAILED: <Fault 30: 'FAILED'>")
+        self.assertEqual(lines[3], 'foo:FAILED restarted')
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 10)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+    def test_runforever_eager_error_on_process_start(self):
+        programs = ['SPAWN_ERROR']
+        any = False
+        prog = self._makeOnePopulated(programs, any, exc=True)
+        prog.rpc.supervisor.all_process_info = _FAIL
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = prog.stderr.getvalue().split('\n')
+        #self.assertEqual(len(lines), 4)
+        self.assertEqual(lines[0],
+                         "Restarting selected processes ['SPAWN_ERROR']")
+        self.assertEqual(lines[1],
+                         'foo:SPAWN_ERROR is in RUNNING state, restarting')
+        self.assertEqual(lines[2],
+           "Failed to start process foo:SPAWN_ERROR: <Fault 50: 'SPAWN_ERROR'>")
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 9)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+    def test_runforever_eager_gcore(self):
+        programs = ['foo', 'bar', 'baz_01', 'notexisting']
+        any = None
+        prog = self._makeOnePopulated(programs, any, exc=True, gcore="true",
+                                      coredir="/tmp")
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = prog.stderr.getvalue().split('\n')
+        self.assertEqual(lines[0],
+                         ("Restarting selected processes ['foo', 'bar', "
+                          "'baz_01', 'notexisting']")
+                         )
+        self.assertEqual(lines[1], 'gcore output for foo:')
+        self.assertEqual(lines[2], '')
+        self.assertEqual(lines[3], ' ')
+        self.assertEqual(lines[4], 'foo is in RUNNING state, restarting')
+        self.assertEqual(lines[5], 'foo restarted')
+        self.assertEqual(lines[6], 'bar not in RUNNING state, NOT restarting')
+        self.assertEqual(lines[7],
+                         'baz:baz_01 not in RUNNING state, NOT restarting')
+        self.assertEqual(lines[8],
+          "Programs not restarted because they did not exist: ['notexisting']")
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 15)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+    def test_runforever_not_eager_none_running(self):
+        programs = ['bar', 'baz_01']
+        any = None
+        prog = self._makeOnePopulated(programs, any, exc=True, gcore="true",
+                                      coredir="/tmp", eager=False)
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = filter(None, prog.stderr.getvalue().split('\n'))
+        self.assertEqual(len(lines), 0, lines)
+        self.failIf('mailed' in prog.__dict__)
+
+    def test_runforever_not_eager_running(self):
+        programs = ['foo', 'bar']
+        any = None
+        prog = self._makeOnePopulated(programs, any, exc=True, eager=False)
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = filter(None, prog.stderr.getvalue().split('\n'))
+        self.assertEqual(lines[0],
+                         ("Restarting selected processes ['foo', 'bar']")
+                         )
+        self.assertEqual(lines[1], 'foo is in RUNNING state, restarting')
+        self.assertEqual(lines[2], 'foo restarted')
+        self.assertEqual(lines[3], 'bar not in RUNNING state, NOT restarting')
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 10)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+class CrashMailTests(unittest.TestCase):
+    def _getTargetClass(self):
+        from superlance.crashmail import CrashMail
+        return CrashMail
+    
+    def _makeOne(self, *opts):
+        return self._getTargetClass()(*opts)
+
+    def setUp(self):
+        import tempfile
+        self.tempdir = tempfile.mkdtemp()
+
+    def tearDown(self):
+        import shutil
+        shutil.rmtree(self.tempdir)
+
+    def _makeOnePopulated(self, programs, any, response=None):
+
+        import os
+        sendmail = 'cat - > %s' % os.path.join(self.tempdir, 'email.log')
+        email = 'chrism@plope.com'
+        header = '[foo]'
+        prog = self._makeOne(programs, any, email, sendmail, header)
+        prog.stdin = StringIO()
+        prog.stdout = StringIO()
+        prog.stderr = StringIO()
+        return prog
+
+    def test_runforever_not_process_state_exited(self):
+        programs = {'foo':0, 'bar':0, 'baz_01':0 }
+        groups = {}
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        prog.stdin.write('eventname:PROCESS_STATE len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        self.assertEqual(prog.stderr.getvalue(), 'non-exited event\n')
+
+    def test_runforever_expected_exit(self):
+        programs = ['foo']
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        payload=('expected:1 processname:foo groupname:bar '
+                 'from_state:RUNNING pid:1')
+        prog.stdin.write(
+            'eventname:PROCESS_STATE_EXITED len:%s\n' % len(payload))
+        prog.stdin.write(payload)
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        self.assertEqual(prog.stderr.getvalue(), 'expected exit\n')
+
+    def test_runforever_unexpected_exit(self):
+        programs = ['foo']
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        payload=('expected:0 processname:foo groupname:bar '
+                 'from_state:RUNNING pid:1')
+        prog.stdin.write(
+            'eventname:PROCESS_STATE_EXITED len:%s\n' % len(payload))
+        prog.stdin.write(payload)
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        output = prog.stderr.getvalue()
+        lines = output.split('\n')
+        self.assertEqual(lines[0], 'unexpected exit, mailing')
+        self.assertEqual(lines[1], 'Mailed:')
+        self.assertEqual(lines[2], '')
+        self.assertEqual(lines[3], 'To: chrism@plope.com')
+        self.failUnless('Subject: [foo]: foo crashed at' in lines[4])
+        self.assertEqual(lines[5], '')
+        self.failUnless(
+            'Process foo in group bar exited unexpectedly' in lines[6])
+        import os
+        mail = open(os.path.join(self.tempdir, 'email.log'), 'r').read()
+        self.failUnless(
+            'Process foo in group bar exited unexpectedly' in mail)
+
+
+class MemmonTests(unittest.TestCase):
+    def _getTargetClass(self):
+        from supervisor.memmon import Memmon
+        return Memmon
+    
+    def _makeOne(self, *opts):
+        return self._getTargetClass()(*opts)
+
+    def _makeOnePopulated(self, programs, groups, any):
+        from supervisor.tests.base import DummyRPCServer
+        rpc = DummyRPCServer()
+        sendmail = 'cat - > /dev/null'
+        email = 'chrism@plope.com'
+        memmon = self._makeOne(programs, groups, any, sendmail, email, rpc)
+        memmon.stdin = StringIO()
+        memmon.stdout = StringIO()
+        memmon.stderr = StringIO()
+        memmon.pscommand = 'echo 22%s'
+        return memmon
+        
+    def test_runforever_notatick(self):
+        programs = {'foo':0, 'bar':0, 'baz_01':0 }
+        groups = {}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:NOTATICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        self.assertEqual(memmon.stderr.getvalue(), '')
+
+    def test_runforever_tick_programs(self):
+        programs = {'foo':0, 'bar':0, 'baz_01':0 }
+        groups = {}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 8)
+        self.assertEqual(lines[0], 'Checking programs foo=0, bar=0, baz_01=0')
+        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[2], 'Restarting foo:foo')
+        self.assertEqual(lines[3], 'RSS of bar:bar is 2265088')
+        self.assertEqual(lines[4], 'Restarting bar:bar')
+        self.assertEqual(lines[5], 'RSS of baz:baz_01 is 2265088')
+        self.assertEqual(lines[6], 'Restarting baz:baz_01')
+        self.assertEqual(lines[7], '')
+        mailed = memmon.mailed.split('\n')
+        self.assertEqual(len(mailed), 4)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                         'Subject: memmon: process baz:baz_01 restarted')
+        self.assertEqual(mailed[2], '')
+        self.failUnless(mailed[3].startswith('memmon.py restarted'))
+
+    def test_runforever_tick_groups(self):
+        programs = {}
+        groups = {'foo':0}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 4)
+        self.assertEqual(lines[0], 'Checking groups foo=0')
+        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[2], 'Restarting foo:foo')
+        self.assertEqual(lines[3], '')
+        mailed = memmon.mailed.split('\n')
+        self.assertEqual(len(mailed), 4)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+          'Subject: memmon: process foo:foo restarted')
+        self.assertEqual(mailed[2], '')
+        self.failUnless(mailed[3].startswith('memmon.py restarted'))
+
+    def test_runforever_tick_any(self):
+        programs = {}
+        groups = {}
+        any = 0
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 8)
+        self.assertEqual(lines[0], 'Checking any=0')
+        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[2], 'Restarting foo:foo')
+        self.assertEqual(lines[3], 'RSS of bar:bar is 2265088')
+        self.assertEqual(lines[4], 'Restarting bar:bar')
+        self.assertEqual(lines[5], 'RSS of baz:baz_01 is 2265088')
+        self.assertEqual(lines[6], 'Restarting baz:baz_01')
+        self.assertEqual(lines[7], '')
+        mailed = memmon.mailed.split('\n')
+        self.assertEqual(len(mailed), 4)
+
+    def test_runforever_tick_programs_and_groups(self):
+        programs = {'baz_01':0}
+        groups = {'foo':0}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 7)
+        self.assertEqual(lines[0], 'Checking programs baz_01=0')
+        self.assertEqual(lines[1], 'Checking groups foo=0')
+        self.assertEqual(lines[2], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[3], 'Restarting foo:foo')
+        self.assertEqual(lines[4], 'RSS of baz:baz_01 is 2265088')
+        self.assertEqual(lines[5], 'Restarting baz:baz_01')
+        self.assertEqual(lines[6], '')
+        mailed = memmon.mailed.split('\n')
+        self.assertEqual(len(mailed), 4)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                         'Subject: memmon: process baz:baz_01 restarted')
+        self.assertEqual(mailed[2], '')
+        self.failUnless(mailed[3].startswith('memmon.py restarted'))
+
+    def test_runforever_tick_programs_norestart(self):
+        programs = {'foo': sys.maxint}
+        groups = {}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 3)
+        self.assertEqual(lines[0], 'Checking programs foo=%s' % sys.maxint)
+        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[2], '')
+        self.assertEqual(memmon.mailed, False)
+
+    def test_stopprocess_fault_tick_programs_norestart(self):
+        programs = {'foo': sys.maxint}
+        groups = {}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 3)
+        self.assertEqual(lines[0], 'Checking programs foo=%s' % sys.maxint)
+        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[2], '')
+        self.assertEqual(memmon.mailed, False)
+
+    def test_stopprocess_fails_to_stop(self):
+        programs = {'BAD_NAME': 0}
+        groups = {}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        from supervisor.process import ProcessStates
+        memmon.rpc.supervisor.all_process_info =  [ {
+            'name':'BAD_NAME',
+            'group':'BAD_NAME',
+            'pid':11,
+            'state':ProcessStates.RUNNING,
+            'statename':'RUNNING',
+            'start':0,
+            'stop':0,
+            'spawnerr':'',
+            'now':0,
+            'description':'BAD_NAME description',
+             } ]
+        import xmlrpclib
+        self.assertRaises(xmlrpclib.Fault, memmon.runforever, True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 4)
+        self.assertEqual(lines[0], 'Checking programs BAD_NAME=%s' % 0)
+        self.assertEqual(lines[1], 'RSS of BAD_NAME:BAD_NAME is 2264064')
+        self.assertEqual(lines[2], 'Restarting BAD_NAME:BAD_NAME')
+        self.failUnless(lines[3].startswith('Failed'))
+        mailed = memmon.mailed.split('\n')
+        self.assertEqual(len(mailed), 4)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+          'Subject: memmon: failed to stop process BAD_NAME:BAD_NAME, exiting')
+        self.assertEqual(mailed[2], '')
+        self.failUnless(mailed[3].startswith('Failed'))
+
+
+def make_connection(response, exc=None):
+    class TestConnection:
+        def __init__(self, hostport):
+            self.hostport = hostport
+
+        def request(self, method, path):
+            if exc:
+                raise ValueError('foo')
+            self.method = method
+            self.path = path
+
+        def getresponse(self):
+            return response
+
+    return TestConnection
+
+class DummyResponse:
+    status = 200
+    reason = 'OK'
+    body = 'OK'
+    def read(self):
+        return self.body
+
+class DummyRPCServer:
+    def __init__(self):
+        self.supervisor = DummySupervisorRPCNamespace()
+        self.system = DummySystemRPCNamespace()
+
+class DummySystemRPCNamespace:
+    pass
+
+import time
+from supervisor.process import ProcessStates
+
+_NOW = time.time()
+
+_FAIL = [ {
+        'name':'FAILED',
+        'group':'foo',
+        'pid':11,
+        'state':ProcessStates.RUNNING,
+        'statename':'RUNNING',
+        'start':_NOW - 100,
+        'stop':0,
+        'spawnerr':'',
+        'now':_NOW,
+        'description':'foo description',
+        },
+{
+        'name':'SPAWN_ERROR',
+        'group':'foo',
+        'pid':11,
+        'state':ProcessStates.RUNNING,
+        'statename':'RUNNING',
+        'start':_NOW - 100,
+        'stop':0,
+        'spawnerr':'',
+        'now':_NOW,
+        'description':'foo description',
+        },]
+
+class DummySupervisorRPCNamespace:
+    _restartable = True
+    _restarted = False
+    _shutdown = False
+    _readlog_error = False
+
+
+    all_process_info = [
+        {
+        'name':'foo',
+        'group':'foo',
+        'pid':11,
+        'state':ProcessStates.RUNNING,
+        'statename':'RUNNING',
+        'start':_NOW - 100,
+        'stop':0,
+        'spawnerr':'',
+        'now':_NOW,
+        'description':'foo description',
+        },
+        {
+        'name':'bar',
+        'group':'bar',
+        'pid':12,
+        'state':ProcessStates.FATAL,
+        'statename':'FATAL',
+        'start':_NOW - 100,
+        'stop':_NOW - 50,
+        'spawnerr':'screwed',
+        'now':_NOW,
+        'description':'bar description',
+        },
+        {
+        'name':'baz_01',
+        'group':'baz',
+        'pid':12,
+        'state':ProcessStates.STOPPED,
+        'statename':'STOPPED',
+        'start':_NOW - 100,
+        'stop':_NOW - 25,
+        'spawnerr':'',
+        'now':_NOW,
+        'description':'baz description',
+        },
+        ]
+
+    def getAllProcessInfo(self):
+        return self.all_process_info
+
+    def startProcess(self, name):
+        from supervisor import xmlrpc
+        from xmlrpclib import Fault
+        if name.endswith('SPAWN_ERROR'):
+            raise Fault(xmlrpc.Faults.SPAWN_ERROR, 'SPAWN_ERROR')
+        return True
+
+    def stopProcess(self, name):
+        from supervisor import xmlrpc
+        from xmlrpclib import Fault
+        if name.endswith('FAILED'):
+            raise Fault(xmlrpc.Faults.FAILED, 'FAILED')
+        return True
+    
+
+def test_suite():
+    return unittest.findTestCases(sys.modules[__name__])
+
+if __name__ == '__main__':
+    unittest.main(defaultTest='test_suite')

File tags/0.5/superlance/timeoutconn.py

View file
  • Ignore whitespace
+import httplib
+import socket
+
+class TimeoutHTTPConnection(httplib.HTTPConnection):
+    """A customised HTTPConnection allowing a per-connection
+    timeout, specified at construction."""
+    timeout = None
+
+    def connect(self):
+        """Override HTTPConnection.connect to connect to
+        host/port specified in __init__."""