Commits

Anonymous committed 85ea98f

[svn r978] Refactored unit tests into their own files and added crashmailbatch and fatalmailbatch

Comments (0)

Files changed (18)

trunk/CHANGES.txt

 Next Release
 ------------
 
+- Separated unit tests into their own files
+
+- Created ``fatalmailbatch`` plugin
+
+- Created ``crashmailbatch`` plugin
+
 - Sphinxified documentation.
 
 - Fixed ``test_suite`` to use the correct module name in setup.py.

trunk/docs/crashmailbatch.rst

+:command:`crashmailbatch` Documentation
+==================================
+
+:command:`crashmailbatch` is a supervisor "event listener", intended to be
+subscribed to ``PROCESS_STATE`` and ``TICK_60`` events.  It monitors
+all processes running under a given supervisord instance.
+
+Similar to :command:`crashmail`, :command:`crashmailbatch` sends email 
+alerts when processes die unexpectedly.  The difference is that all alerts 
+generated within the configured time interval are batched together to avoid 
+sending too many emails.   
+
+:command:`crashmailbatch` is a "console script" installed when you install
+:mod:`superlance`.  Although :command:`crashmailbatch` is an executable 
+program, it isn't useful as a general-purpose script:  it must be run as a
+:command:`supervisor` event listener to do anything useful.
+
+Command-Line Syntax
+-------------------
+
+.. code-block:: sh
+
+   $ crashmailbatch --toEmail=<email address> --fromEmail=<email address> \
+           [--interval=<batch interval in minutes>] [--subject=<email subject>]
+   
+.. program:: crashmailbatch
+
+.. cmdoption:: -t <destination email>, --toEmail=<destination email>
+   
+   Specify an email address to which crash notification messages are sent.
+ 
+.. cmdoption:: -f <source email>, --fromEmail=<source email>
+   
+   Specify an email address from which crash notification messages are sent.
+
+.. cmdoption:: -i <interval>, --interval=<interval>
+   
+   Specify the time interval in minutes to use for batching notifcations.
+   Defaults to 1 minute.
+
+.. cmdoption:: -s <email subject>, --subject=<email subject>
+   
+   Override the email subject line.  Defaults to "Crash alert from supervisord"
+
+Configuring :command:`crashmailbatch` Into the Supervisor Config
+-----------------------------------------------------------
+
+An ``[eventlistener:x]`` section must be placed in :file:`supervisord.conf`
+in order for :command:`crashmailbatch` to do its work. See the "Events" chapter in
+the Supervisor manual for more information about event listeners.
+
+The following example assumes that :command:`crashmailbatch` is on your system
+:envvar:`PATH`.
+
+.. code-block:: ini
+
+   [eventlistener:crashmailbatch]
+   crashmailbatch --toEmail="alertme@fubar.com" --fromEmail="supervisord@fubar.com" 
+   events=PROCESS_STATE,TICK_60

trunk/docs/fatalmailbatch.rst

+:command:`fatalmailbatch` Documentation
+==================================
+
+:command:`fatalmailbatch` is a supervisor "event listener", intended to be
+subscribed to ``PROCESS_STATE`` and ``TICK_60`` events.  It monitors
+all processes running under a given supervisord instance.
+
+:command:`fatalmailbatch` sends email alerts when processes fail to start 
+too many times such that supervisord gives up retrying.  All of the fatal
+start events generated within the configured time interval are batched 
+together to avoid sending too many emails.   
+
+:command:`fatalmailbatch` is a "console script" installed when you install
+:mod:`superlance`.  Although :command:`fatalmailbatch` is an executable 
+program, it isn't useful as a general-purpose script:  it must be run as a
+:command:`supervisor` event listener to do anything useful.
+
+Command-Line Syntax
+-------------------
+
+.. code-block:: sh
+
+   $ fatalmailbatch --toEmail=<email address> --fromEmail=<email address> \
+           [--interval=<batch interval in minutes>] [--subject=<email subject>]
+   
+.. program:: fatalmailbatch
+
+.. cmdoption:: -t <destination email>, --toEmail=<destination email>
+   
+   Specify an email address to which fatal start notification messages are sent.
+ 
+.. cmdoption:: -f <source email>, --fromEmail=<source email>
+   
+   Specify an email address from which fatal start notification messages 
+   are sent.
+
+.. cmdoption:: -i <interval>, --interval=<interval>
+   
+   Specify the time interval in minutes to use for batching notifcations.
+   Defaults to 1 minute.
+
+.. cmdoption:: -s <email subject>, --subject=<email subject>
+   
+   Override the email subject line.  Defaults to "Fatal start alert from 
+   supervisord"
+
+Configuring :command:`fatalmailbatch` Into the Supervisor Config
+-----------------------------------------------------------
+
+An ``[eventlistener:x]`` section must be placed in :file:`supervisord.conf`
+in order for :command:`fatalmailbatch` to do its work. See the "Events" chapter in
+the Supervisor manual for more information about event listeners.
+
+The following example assumes that :command:`fatalmailbatch` is on your system
+:envvar:`PATH`.
+
+.. code-block:: ini
+
+   [eventlistener:fatalmailbatch]
+   fatalmailbatch --toEmail="alertme@fubar.com" --fromEmail="supervisord@fubar.com" 
+   events=PROCESS_STATE,TICK_60
             ],
       tests_require=[
             'supervisor',
+            'mock',
             ],
       test_suite='superlance.tests',
       entry_points = """\
       [console_scripts]
       httpok = superlance.httpok:main
       crashmail = superlance.crashmail:main
+      crashmailbatch = superlance.crashmailbatch:main
+      fatalmailbatch = superlance.fatalmailbatch:main
       memmon = superlance.memmon:main
       """
       )

trunk/superlance/crashmailbatch.py

+#!/usr/bin/env python -u
+##############################################################################
+#
+# Copyright (c) 2007 Agendaless Consulting and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the BSD-like license at
+# http://www.repoze.org/LICENSE.txt.  A copy of the license should accompany
+# this distribution.  THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL
+# EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND
+# FITNESS FOR A PARTICULAR PURPOSE
+#
+##############################################################################
+
+# A event listener meant to be subscribed to PROCESS_STATE_CHANGE
+# events.  It will send mail when processes that are children of
+# supervisord transition unexpectedly to the EXITED state.
+
+# A supervisor config snippet that tells supervisor to use this script
+# as a listener is below.
+#
+# [eventlistener:crashmailbatch]
+# command=python crashmailbatch
+# events=PROCESS_STATE,TICK_60
+
+doc = """\
+crashmailbatch.py [--interval=<batch interval in minutes>]
+        [--toEmail=<email address>]
+        [--fromEmail=<email address>]
+        [--subject=<email subject>]
+
+Options:
+
+--interval  - batch cycle length (in minutes).  The default is 1 minute.
+                  This means that all events in each cycle are batched together
+                  and sent as a single email
+                  
+--toEmail   - the email address to send alerts to
+
+--fromEmail - the email address to send alerts from
+
+--subject - the email subject line
+
+A sample invocation:
+
+crashmailbatch.py --toEmail="you@bar.com" --fromEmail="me@bar.com"
+
+"""
+
+from supervisor import childutils
+from superlance.process_state_email_monitor import ProcessStateEmailMonitor
+
+class CrashMailBatch(ProcessStateEmailMonitor):
+    
+    processStateEvents = ['PROCESS_STATE_EXITED']
+
+    def __init__(self, **kwargs):
+        kwargs['subject'] = kwargs.get('subject', 'Crash alert from supervisord')
+        ProcessStateEmailMonitor.__init__(self, **kwargs)
+        self.now = kwargs.get('now', None)
+ 
+    def getProcessStateChangeMsg(self, headers, payload):
+        pheaders, pdata = childutils.eventdata(payload+'\n')
+        
+        if int(pheaders['expected']):
+            return None
+        
+        txt = 'Process %(groupname)s:%(processname)s (pid %(pid)s) died \
+unexpectedly' % pheaders
+        return '%s -- %s' % (childutils.get_asctime(self.now), txt)
+
+def main():
+    crash = CrashMailBatch.createFromCmdLine()
+    crash.run()
+
+if __name__ == '__main__':
+    main()

trunk/superlance/fatalmailbatch.py

+#!/usr/bin/env python -u
+##############################################################################
+#
+# Copyright (c) 2007 Agendaless Consulting and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the BSD-like license at
+# http://www.repoze.org/LICENSE.txt.  A copy of the license should accompany
+# this distribution.  THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL
+# EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND
+# FITNESS FOR A PARTICULAR PURPOSE
+#
+##############################################################################
+
+# A event listener meant to be subscribed to PROCESS_STATE_CHANGE
+# events.  It will send mail when processes that are children of
+# supervisord transition unexpectedly to the EXITED state.
+
+# A supervisor config snippet that tells supervisor to use this script
+# as a listener is below.
+#
+# [eventlistener:fatalmailbatch]
+# command=python fatalmailbatch
+# events=PROCESS_STATE,TICK_60
+
+doc = """\
+fatalmailbatch.py [--interval=<batch interval in minutes>]
+        [--toEmail=<email address>]
+        [--fromEmail=<email address>]
+        [--subject=<email subject>]
+
+Options:
+
+--interval  - batch cycle length (in minutes).  The default is 1 minute.
+                  This means that all events in each cycle are batched together
+                  and sent as a single email
+                  
+--toEmail   - the email address to send alerts to
+
+--fromEmail - the email address to send alerts from
+
+--subject - the email subject line
+
+A sample invocation:
+
+fatalmailbatch.py --toEmail="you@bar.com" --fromEmail="me@bar.com"
+
+"""
+
+from supervisor import childutils
+from superlance.process_state_email_monitor import ProcessStateEmailMonitor
+
+class FatalMailBatch(ProcessStateEmailMonitor):
+    
+    processStateEvents = ['PROCESS_STATE_FATAL']
+
+    def __init__(self, **kwargs):
+        kwargs['subject'] = kwargs.get('subject', 'Fatal start alert from supervisord')
+        ProcessStateEmailMonitor.__init__(self, **kwargs)
+        self.now = kwargs.get('now', None)
+ 
+    def getProcessStateChangeMsg(self, headers, payload):
+        pheaders, pdata = childutils.eventdata(payload+'\n')
+
+        txt = 'Process %(groupname)s:%(processname)s failed to start too many \
+times' % pheaders
+        return '%s -- %s' % (childutils.get_asctime(self.now), txt)
+
+def main():
+    fatal = FatalMailBatch.createFromCmdLine()
+    fatal.run()
+
+if __name__ == '__main__':
+    main()

trunk/superlance/process_state_email_monitor.py

+#!/usr/bin/env python -u
+##############################################################################
+#
+# Copyright (c) 2007 Agendaless Consulting and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the BSD-like license at
+# http://www.repoze.org/LICENSE.txt.  A copy of the license should accompany
+# this distribution.  THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL
+# EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND
+# FITNESS FOR A PARTICULAR PURPOSE
+#
+##############################################################################
+import os
+import sys
+import smtplib
+import copy
+from email.mime.text import MIMEText
+from superlance.process_state_monitor import ProcessStateMonitor
+
+doc = """\
+Base class for common functionality when monitoring process state changes
+and sending email notification
+"""
+
+class ProcessStateEmailMonitor(ProcessStateMonitor):
+
+    @classmethod
+    def createFromCmdLine(cls):
+        from optparse import OptionParser
+
+        parser = OptionParser()
+        parser.add_option("-i", "--interval", dest="interval", type="int",
+                          help="batch interval in minutes (defaults to 1 minute)")
+        parser.add_option("-t", "--toEmail", dest="toEmail",
+                          help="destination email address")
+        parser.add_option("-f", "--fromEmail", dest="fromEmail",
+                          help="source email address")
+        parser.add_option("-s", "--subject", dest="subject",
+                          help="email subject")
+        (options, args) = parser.parse_args()
+
+        if not options.toEmail:
+            parser.print_help()
+            sys.exit(1)
+        if not options.fromEmail:
+            parser.print_help()
+            sys.exit(1)
+
+        if not 'SUPERVISOR_SERVER_URL' in os.environ:
+            sys.stderr.write('Must run as a supervisor event listener\n')
+            sys.exit(1)
+
+        return cls(**options.__dict__)
+
+    def __init__(self, **kwargs):
+        ProcessStateMonitor.__init__(self, **kwargs)
+        
+        self.fromEmail = kwargs['fromEmail']
+        self.toEmail = kwargs['toEmail']
+        self.subject = kwargs.get('subject', 'Alert from supervisord')
+        self.digestLen = 20
+            
+    def sendBatchNotification(self):
+        email = self.getBatchEmail()
+        if email:
+            self.sendEmail(email)
+            self.logEmail(email)
+
+    def logEmail(self, email):
+        email4Log = copy.copy(email)
+        if len(email4Log['body']) > self.digestLen:
+            email4Log['body'] = '%s...' % email4Log['body'][:self.digestLen]
+        self.writeToStderr("Sending notification email:\nTo: %(to)s\n\
+From: %(from)s\nSubject: %(subject)s\nBody:\n%(body)s\n" % email4Log)
+            
+    def getBatchEmail(self):
+        if len(self.batchMsgs):
+            return {
+                'to': self.toEmail,
+                'from': self.fromEmail,
+                'subject': self.subject,
+                'body': '\n'.join(self.getBatchMsgs()),
+            }
+        return None
+        
+    def sendEmail(self, email):
+        msg = MIMEText(email['body'])
+        msg['Subject'] = email['subject']
+        msg['From'] = email['from']
+        msg['To'] = email['to']
+
+        s = smtplib.SMTP('localhost')
+        s.sendmail(email['from'], [email['to']], msg.as_string())
+        s.quit()

trunk/superlance/process_state_monitor.py

+#!/usr/bin/env python -u
+##############################################################################
+#
+# Copyright (c) 2007 Agendaless Consulting and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the BSD-like license at
+# http://www.repoze.org/LICENSE.txt.  A copy of the license should accompany
+# this distribution.  THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL
+# EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND
+# FITNESS FOR A PARTICULAR PURPOSE
+#
+##############################################################################
+doc = """\
+Base class for common functionality when monitoring process state changes
+"""
+
+import os
+import sys
+
+from supervisor import childutils
+
+class ProcessStateMonitor:
+
+    # In child class, define a list of events to monitor
+    processStateEvents = []
+
+    def __init__(self, **kwargs):
+        self.interval = kwargs.get('interval', 1)
+        
+        self.debug = kwargs.get('debug', False)
+        self.stdin = kwargs.get('stdin', sys.stdin)
+        self.stdout = kwargs.get('stdout', sys.stdout)
+        self.stderr = kwargs.get('stderr', sys.stderr)
+        
+        self.batchMsgs = []
+        self.batchMins = 0
+ 
+    def run(self):
+        while 1:
+            hdrs, payload = childutils.listener.wait(self.stdin, self.stdout)
+            self.handleEvent(hdrs, payload)
+            childutils.listener.ok(self.stdout)
+    
+    def handleEvent(self, headers, payload):
+        if headers['eventname'] in self.processStateEvents:
+            self.handleProcessStateChangeEvent(headers, payload)
+        elif headers['eventname'] == 'TICK_60':
+            self.handleTick60Event(headers, payload)
+    
+    def handleProcessStateChangeEvent(self, headers, payload):
+        msg = self.getProcessStateChangeMsg(headers, payload)
+        if msg:
+            self.writeToStderr('%s\n' % msg)
+            self.batchMsgs.append(msg)
+
+    """
+    Override this method in child classes to customize messaging
+    """
+    def getProcessStateChangeMsg(self, headers, payload):
+        return None
+
+    def handleTick60Event(self, headers, payload):
+        self.batchMins += 1
+        if self.batchMins >= self.interval:
+            self.sendBatchNotification()
+            self.clearBatch()
+            
+    """
+    Override this method in child classes to send notification
+    """
+    def sendBatchNotification(self):
+        pass
+    
+    def getBatchMinutes(self):
+        return self.batchMins
+    
+    def getBatchMsgs(self):
+        return self.batchMsgs
+        
+    def clearBatch(self):
+        self.batchMins = 0;
+        self.batchMsgs = [];
+
+    def writeToStderr(self, msg):
+        self.stderr.write(msg)
+        self.stderr.flush()

trunk/superlance/tests.py

-import sys
-import unittest
-from StringIO import StringIO
-
-class HTTPOkTests(unittest.TestCase):
-    def _getTargetClass(self):
-        from superlance.httpok import HTTPOk
-        return HTTPOk
-    
-    def _makeOne(self, *opts):
-        return self._getTargetClass()(*opts)
-
-    def _makeOnePopulated(self, programs, any, response=None, exc=None,
-                          gcore=None, coredir=None, eager=True):
-        if response is None:
-            response = DummyResponse()
-        rpc = DummyRPCServer()
-        sendmail = 'cat - > /dev/null'
-        email = 'chrism@plope.com'
-        url = 'http://foo/bar'
-        timeout = 10
-        status = '200'
-        inbody = None
-        gcore = gcore
-        coredir = coredir
-        prog = self._makeOne(rpc, programs, any, url, timeout, status,
-                             inbody, email, sendmail, coredir, gcore, eager)
-        prog.stdin = StringIO()
-        prog.stdout = StringIO()
-        prog.stderr = StringIO()
-        prog.connclass = make_connection(response, exc=exc)
-        return prog
-
-    def test_listProcesses_no_programs(self):
-        programs = []
-        any = None
-        prog = self._makeOnePopulated(programs, any)
-        specs = list(prog.listProcesses())
-        self.assertEqual(len(specs), 0)
-
-    def test_listProcesses_w_RUNNING_programs_default_state(self):
-        programs = ['foo']
-        any = None
-        prog = self._makeOnePopulated(programs, any)
-        specs = list(prog.listProcesses())
-        self.assertEqual(len(specs), 1)
-        self.assertEqual(specs[0],
-                         DummySupervisorRPCNamespace.all_process_info[0])
-
-    def test_listProcesses_w_nonRUNNING_programs_default_state(self):
-        programs = ['bar']
-        any = None
-        prog = self._makeOnePopulated(programs, any)
-        specs = list(prog.listProcesses())
-        self.assertEqual(len(specs), 1)
-        self.assertEqual(specs[0],
-                         DummySupervisorRPCNamespace.all_process_info[1])
-
-    def test_listProcesses_w_nonRUNNING_programs_RUNNING_state(self):
-        programs = ['bar']
-        any = None
-        prog = self._makeOnePopulated(programs, any)
-        specs = list(prog.listProcesses(ProcessStates.RUNNING))
-        self.assertEqual(len(specs), 0, (prog.programs, specs))
-
-    def test_runforever_eager_notatick(self):
-        programs = {'foo':0, 'bar':0, 'baz_01':0 }
-        groups = {}
-        any = None
-        prog = self._makeOnePopulated(programs, any)
-        prog.stdin.write('eventname:NOTATICK len:0\n')
-        prog.stdin.seek(0)
-        prog.runforever(test=True)
-        self.assertEqual(prog.stderr.getvalue(), '')
-
-    def test_runforever_eager_error_on_request_some(self):
-        programs = ['foo', 'bar', 'baz_01', 'notexisting']
-        any = None
-        prog = self._makeOnePopulated(programs, any, exc=True)
-        prog.stdin.write('eventname:TICK len:0\n')
-        prog.stdin.seek(0)
-        prog.runforever(test=True)
-        lines = prog.stderr.getvalue().split('\n')
-        #self.assertEqual(len(lines), 7)
-        self.assertEqual(lines[0],
-                         ("Restarting selected processes ['foo', 'bar', "
-                          "'baz_01', 'notexisting']")
-                         )
-        self.assertEqual(lines[1], 'foo is in RUNNING state, restarting')
-        self.assertEqual(lines[2], 'foo restarted')
-        self.assertEqual(lines[3], 'bar not in RUNNING state, NOT restarting')
-        self.assertEqual(lines[4],
-                         'baz:baz_01 not in RUNNING state, NOT restarting')
-        self.assertEqual(lines[5],
-          "Programs not restarted because they did not exist: ['notexisting']")
-        mailed = prog.mailed.split('\n')
-        self.assertEqual(len(mailed), 12)
-        self.assertEqual(mailed[0], 'To: chrism@plope.com')
-        self.assertEqual(mailed[1],
-                    'Subject: httpok for http://foo/bar: bad status returned')
-
-    def test_runforever_eager_error_on_request_any(self):
-        programs = []
-        any = True
-        prog = self._makeOnePopulated(programs, any, exc=True)
-        prog.stdin.write('eventname:TICK len:0\n')
-        prog.stdin.seek(0)
-        prog.runforever(test=True)
-        lines = prog.stderr.getvalue().split('\n')
-        #self.assertEqual(len(lines), 6)
-        self.assertEqual(lines[0], 'Restarting all running processes')
-        self.assertEqual(lines[1], 'foo is in RUNNING state, restarting')
-        self.assertEqual(lines[2], 'foo restarted')
-        self.assertEqual(lines[3], 'bar not in RUNNING state, NOT restarting')
-        self.assertEqual(lines[4],
-                         'baz:baz_01 not in RUNNING state, NOT restarting')
-        mailed = prog.mailed.split('\n')
-        self.assertEqual(len(mailed), 11)
-        self.assertEqual(mailed[0], 'To: chrism@plope.com')
-        self.assertEqual(mailed[1],
-                    'Subject: httpok for http://foo/bar: bad status returned')
-
-    def test_runforever_eager_error_on_process_stop(self):
-        programs = ['FAILED']
-        any = False
-        prog = self._makeOnePopulated(programs, any, exc=True)
-        prog.rpc.supervisor.all_process_info = _FAIL
-        prog.stdin.write('eventname:TICK len:0\n')
-        prog.stdin.seek(0)
-        prog.runforever(test=True)
-        lines = prog.stderr.getvalue().split('\n')
-        #self.assertEqual(len(lines), 5)
-        self.assertEqual(lines[0], "Restarting selected processes ['FAILED']")
-        self.assertEqual(lines[1], 'foo:FAILED is in RUNNING state, restarting')
-        self.assertEqual(lines[2],
-                    "Failed to stop process foo:FAILED: <Fault 30: 'FAILED'>")
-        self.assertEqual(lines[3], 'foo:FAILED restarted')
-        mailed = prog.mailed.split('\n')
-        self.assertEqual(len(mailed), 10)
-        self.assertEqual(mailed[0], 'To: chrism@plope.com')
-        self.assertEqual(mailed[1],
-                    'Subject: httpok for http://foo/bar: bad status returned')
-
-    def test_runforever_eager_error_on_process_start(self):
-        programs = ['SPAWN_ERROR']
-        any = False
-        prog = self._makeOnePopulated(programs, any, exc=True)
-        prog.rpc.supervisor.all_process_info = _FAIL
-        prog.stdin.write('eventname:TICK len:0\n')
-        prog.stdin.seek(0)
-        prog.runforever(test=True)
-        lines = prog.stderr.getvalue().split('\n')
-        #self.assertEqual(len(lines), 4)
-        self.assertEqual(lines[0],
-                         "Restarting selected processes ['SPAWN_ERROR']")
-        self.assertEqual(lines[1],
-                         'foo:SPAWN_ERROR is in RUNNING state, restarting')
-        self.assertEqual(lines[2],
-           "Failed to start process foo:SPAWN_ERROR: <Fault 50: 'SPAWN_ERROR'>")
-        mailed = prog.mailed.split('\n')
-        self.assertEqual(len(mailed), 9)
-        self.assertEqual(mailed[0], 'To: chrism@plope.com')
-        self.assertEqual(mailed[1],
-                    'Subject: httpok for http://foo/bar: bad status returned')
-
-    def test_runforever_eager_gcore(self):
-        programs = ['foo', 'bar', 'baz_01', 'notexisting']
-        any = None
-        prog = self._makeOnePopulated(programs, any, exc=True, gcore="true",
-                                      coredir="/tmp")
-        prog.stdin.write('eventname:TICK len:0\n')
-        prog.stdin.seek(0)
-        prog.runforever(test=True)
-        lines = prog.stderr.getvalue().split('\n')
-        self.assertEqual(lines[0],
-                         ("Restarting selected processes ['foo', 'bar', "
-                          "'baz_01', 'notexisting']")
-                         )
-        self.assertEqual(lines[1], 'gcore output for foo:')
-        self.assertEqual(lines[2], '')
-        self.assertEqual(lines[3], ' ')
-        self.assertEqual(lines[4], 'foo is in RUNNING state, restarting')
-        self.assertEqual(lines[5], 'foo restarted')
-        self.assertEqual(lines[6], 'bar not in RUNNING state, NOT restarting')
-        self.assertEqual(lines[7],
-                         'baz:baz_01 not in RUNNING state, NOT restarting')
-        self.assertEqual(lines[8],
-          "Programs not restarted because they did not exist: ['notexisting']")
-        mailed = prog.mailed.split('\n')
-        self.assertEqual(len(mailed), 15)
-        self.assertEqual(mailed[0], 'To: chrism@plope.com')
-        self.assertEqual(mailed[1],
-                    'Subject: httpok for http://foo/bar: bad status returned')
-
-    def test_runforever_not_eager_none_running(self):
-        programs = ['bar', 'baz_01']
-        any = None
-        prog = self._makeOnePopulated(programs, any, exc=True, gcore="true",
-                                      coredir="/tmp", eager=False)
-        prog.stdin.write('eventname:TICK len:0\n')
-        prog.stdin.seek(0)
-        prog.runforever(test=True)
-        lines = filter(None, prog.stderr.getvalue().split('\n'))
-        self.assertEqual(len(lines), 0, lines)
-        self.failIf('mailed' in prog.__dict__)
-
-    def test_runforever_not_eager_running(self):
-        programs = ['foo', 'bar']
-        any = None
-        prog = self._makeOnePopulated(programs, any, exc=True, eager=False)
-        prog.stdin.write('eventname:TICK len:0\n')
-        prog.stdin.seek(0)
-        prog.runforever(test=True)
-        lines = filter(None, prog.stderr.getvalue().split('\n'))
-        self.assertEqual(lines[0],
-                         ("Restarting selected processes ['foo', 'bar']")
-                         )
-        self.assertEqual(lines[1], 'foo is in RUNNING state, restarting')
-        self.assertEqual(lines[2], 'foo restarted')
-        self.assertEqual(lines[3], 'bar not in RUNNING state, NOT restarting')
-        mailed = prog.mailed.split('\n')
-        self.assertEqual(len(mailed), 10)
-        self.assertEqual(mailed[0], 'To: chrism@plope.com')
-        self.assertEqual(mailed[1],
-                    'Subject: httpok for http://foo/bar: bad status returned')
-
-class CrashMailTests(unittest.TestCase):
-    def _getTargetClass(self):
-        from superlance.crashmail import CrashMail
-        return CrashMail
-    
-    def _makeOne(self, *opts):
-        return self._getTargetClass()(*opts)
-
-    def setUp(self):
-        import tempfile
-        self.tempdir = tempfile.mkdtemp()
-
-    def tearDown(self):
-        import shutil
-        shutil.rmtree(self.tempdir)
-
-    def _makeOnePopulated(self, programs, any, response=None):
-
-        import os
-        sendmail = 'cat - > %s' % os.path.join(self.tempdir, 'email.log')
-        email = 'chrism@plope.com'
-        header = '[foo]'
-        prog = self._makeOne(programs, any, email, sendmail, header)
-        prog.stdin = StringIO()
-        prog.stdout = StringIO()
-        prog.stderr = StringIO()
-        return prog
-
-    def test_runforever_not_process_state_exited(self):
-        programs = {'foo':0, 'bar':0, 'baz_01':0 }
-        groups = {}
-        any = None
-        prog = self._makeOnePopulated(programs, any)
-        prog.stdin.write('eventname:PROCESS_STATE len:0\n')
-        prog.stdin.seek(0)
-        prog.runforever(test=True)
-        self.assertEqual(prog.stderr.getvalue(), 'non-exited event\n')
-
-    def test_runforever_expected_exit(self):
-        programs = ['foo']
-        any = None
-        prog = self._makeOnePopulated(programs, any)
-        payload=('expected:1 processname:foo groupname:bar '
-                 'from_state:RUNNING pid:1')
-        prog.stdin.write(
-            'eventname:PROCESS_STATE_EXITED len:%s\n' % len(payload))
-        prog.stdin.write(payload)
-        prog.stdin.seek(0)
-        prog.runforever(test=True)
-        self.assertEqual(prog.stderr.getvalue(), 'expected exit\n')
-
-    def test_runforever_unexpected_exit(self):
-        programs = ['foo']
-        any = None
-        prog = self._makeOnePopulated(programs, any)
-        payload=('expected:0 processname:foo groupname:bar '
-                 'from_state:RUNNING pid:1')
-        prog.stdin.write(
-            'eventname:PROCESS_STATE_EXITED len:%s\n' % len(payload))
-        prog.stdin.write(payload)
-        prog.stdin.seek(0)
-        prog.runforever(test=True)
-        output = prog.stderr.getvalue()
-        lines = output.split('\n')
-        self.assertEqual(lines[0], 'unexpected exit, mailing')
-        self.assertEqual(lines[1], 'Mailed:')
-        self.assertEqual(lines[2], '')
-        self.assertEqual(lines[3], 'To: chrism@plope.com')
-        self.failUnless('Subject: [foo]: foo crashed at' in lines[4])
-        self.assertEqual(lines[5], '')
-        self.failUnless(
-            'Process foo in group bar exited unexpectedly' in lines[6])
-        import os
-        mail = open(os.path.join(self.tempdir, 'email.log'), 'r').read()
-        self.failUnless(
-            'Process foo in group bar exited unexpectedly' in mail)
-
-
-class MemmonTests(unittest.TestCase):
-    def _getTargetClass(self):
-        from superlance.memmon import Memmon
-        return Memmon
-    
-    def _makeOne(self, *opts):
-        return self._getTargetClass()(*opts)
-
-    def _makeOnePopulated(self, programs, groups, any):
-        rpc = DummyRPCServer()
-        sendmail = 'cat - > /dev/null'
-        email = 'chrism@plope.com'
-        memmon = self._makeOne(programs, groups, any, sendmail, email, rpc)
-        memmon.stdin = StringIO()
-        memmon.stdout = StringIO()
-        memmon.stderr = StringIO()
-        memmon.pscommand = 'echo 22%s'
-        return memmon
-        
-    def test_runforever_notatick(self):
-        programs = {'foo':0, 'bar':0, 'baz_01':0 }
-        groups = {}
-        any = None
-        memmon = self._makeOnePopulated(programs, groups, any)
-        memmon.stdin.write('eventname:NOTATICK len:0\n')
-        memmon.stdin.seek(0)
-        memmon.runforever(test=True)
-        self.assertEqual(memmon.stderr.getvalue(), '')
-
-    def test_runforever_tick_programs(self):
-        programs = {'foo':0, 'bar':0, 'baz_01':0 }
-        groups = {}
-        any = None
-        memmon = self._makeOnePopulated(programs, groups, any)
-        memmon.stdin.write('eventname:TICK len:0\n')
-        memmon.stdin.seek(0)
-        memmon.runforever(test=True)
-        lines = memmon.stderr.getvalue().split('\n')
-        self.assertEqual(len(lines), 8)
-        self.assertEqual(lines[0], 'Checking programs foo=0, bar=0, baz_01=0')
-        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
-        self.assertEqual(lines[2], 'Restarting foo:foo')
-        self.assertEqual(lines[3], 'RSS of bar:bar is 2265088')
-        self.assertEqual(lines[4], 'Restarting bar:bar')
-        self.assertEqual(lines[5], 'RSS of baz:baz_01 is 2265088')
-        self.assertEqual(lines[6], 'Restarting baz:baz_01')
-        self.assertEqual(lines[7], '')
-        mailed = memmon.mailed.split('\n')
-        self.assertEqual(len(mailed), 4)
-        self.assertEqual(mailed[0], 'To: chrism@plope.com')
-        self.assertEqual(mailed[1],
-                         'Subject: memmon: process baz:baz_01 restarted')
-        self.assertEqual(mailed[2], '')
-        self.failUnless(mailed[3].startswith('memmon.py restarted'))
-
-    def test_runforever_tick_groups(self):
-        programs = {}
-        groups = {'foo':0}
-        any = None
-        memmon = self._makeOnePopulated(programs, groups, any)
-        memmon.stdin.write('eventname:TICK len:0\n')
-        memmon.stdin.seek(0)
-        memmon.runforever(test=True)
-        lines = memmon.stderr.getvalue().split('\n')
-        self.assertEqual(len(lines), 4)
-        self.assertEqual(lines[0], 'Checking groups foo=0')
-        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
-        self.assertEqual(lines[2], 'Restarting foo:foo')
-        self.assertEqual(lines[3], '')
-        mailed = memmon.mailed.split('\n')
-        self.assertEqual(len(mailed), 4)
-        self.assertEqual(mailed[0], 'To: chrism@plope.com')
-        self.assertEqual(mailed[1],
-          'Subject: memmon: process foo:foo restarted')
-        self.assertEqual(mailed[2], '')
-        self.failUnless(mailed[3].startswith('memmon.py restarted'))
-
-    def test_runforever_tick_any(self):
-        programs = {}
-        groups = {}
-        any = 0
-        memmon = self._makeOnePopulated(programs, groups, any)
-        memmon.stdin.write('eventname:TICK len:0\n')
-        memmon.stdin.seek(0)
-        memmon.runforever(test=True)
-        lines = memmon.stderr.getvalue().split('\n')
-        self.assertEqual(len(lines), 8)
-        self.assertEqual(lines[0], 'Checking any=0')
-        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
-        self.assertEqual(lines[2], 'Restarting foo:foo')
-        self.assertEqual(lines[3], 'RSS of bar:bar is 2265088')
-        self.assertEqual(lines[4], 'Restarting bar:bar')
-        self.assertEqual(lines[5], 'RSS of baz:baz_01 is 2265088')
-        self.assertEqual(lines[6], 'Restarting baz:baz_01')
-        self.assertEqual(lines[7], '')
-        mailed = memmon.mailed.split('\n')
-        self.assertEqual(len(mailed), 4)
-
-    def test_runforever_tick_programs_and_groups(self):
-        programs = {'baz_01':0}
-        groups = {'foo':0}
-        any = None
-        memmon = self._makeOnePopulated(programs, groups, any)
-        memmon.stdin.write('eventname:TICK len:0\n')
-        memmon.stdin.seek(0)
-        memmon.runforever(test=True)
-        lines = memmon.stderr.getvalue().split('\n')
-        self.assertEqual(len(lines), 7)
-        self.assertEqual(lines[0], 'Checking programs baz_01=0')
-        self.assertEqual(lines[1], 'Checking groups foo=0')
-        self.assertEqual(lines[2], 'RSS of foo:foo is 2264064')
-        self.assertEqual(lines[3], 'Restarting foo:foo')
-        self.assertEqual(lines[4], 'RSS of baz:baz_01 is 2265088')
-        self.assertEqual(lines[5], 'Restarting baz:baz_01')
-        self.assertEqual(lines[6], '')
-        mailed = memmon.mailed.split('\n')
-        self.assertEqual(len(mailed), 4)
-        self.assertEqual(mailed[0], 'To: chrism@plope.com')
-        self.assertEqual(mailed[1],
-                         'Subject: memmon: process baz:baz_01 restarted')
-        self.assertEqual(mailed[2], '')
-        self.failUnless(mailed[3].startswith('memmon.py restarted'))
-
-    def test_runforever_tick_programs_norestart(self):
-        programs = {'foo': sys.maxint}
-        groups = {}
-        any = None
-        memmon = self._makeOnePopulated(programs, groups, any)
-        memmon.stdin.write('eventname:TICK len:0\n')
-        memmon.stdin.seek(0)
-        memmon.runforever(test=True)
-        lines = memmon.stderr.getvalue().split('\n')
-        self.assertEqual(len(lines), 3)
-        self.assertEqual(lines[0], 'Checking programs foo=%s' % sys.maxint)
-        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
-        self.assertEqual(lines[2], '')
-        self.assertEqual(memmon.mailed, False)
-
-    def test_stopprocess_fault_tick_programs_norestart(self):
-        programs = {'foo': sys.maxint}
-        groups = {}
-        any = None
-        memmon = self._makeOnePopulated(programs, groups, any)
-        memmon.stdin.write('eventname:TICK len:0\n')
-        memmon.stdin.seek(0)
-        memmon.runforever(test=True)
-        lines = memmon.stderr.getvalue().split('\n')
-        self.assertEqual(len(lines), 3)
-        self.assertEqual(lines[0], 'Checking programs foo=%s' % sys.maxint)
-        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
-        self.assertEqual(lines[2], '')
-        self.assertEqual(memmon.mailed, False)
-
-    def test_stopprocess_fails_to_stop(self):
-        programs = {'BAD_NAME': 0}
-        groups = {}
-        any = None
-        memmon = self._makeOnePopulated(programs, groups, any)
-        memmon.stdin.write('eventname:TICK len:0\n')
-        memmon.stdin.seek(0)
-        from supervisor.process import ProcessStates
-        memmon.rpc.supervisor.all_process_info =  [ {
-            'name':'BAD_NAME',
-            'group':'BAD_NAME',
-            'pid':11,
-            'state':ProcessStates.RUNNING,
-            'statename':'RUNNING',
-            'start':0,
-            'stop':0,
-            'spawnerr':'',
-            'now':0,
-            'description':'BAD_NAME description',
-             } ]
-        import xmlrpclib
-        self.assertRaises(xmlrpclib.Fault, memmon.runforever, True)
-        lines = memmon.stderr.getvalue().split('\n')
-        self.assertEqual(len(lines), 4)
-        self.assertEqual(lines[0], 'Checking programs BAD_NAME=%s' % 0)
-        self.assertEqual(lines[1], 'RSS of BAD_NAME:BAD_NAME is 2264064')
-        self.assertEqual(lines[2], 'Restarting BAD_NAME:BAD_NAME')
-        self.failUnless(lines[3].startswith('Failed'))
-        mailed = memmon.mailed.split('\n')
-        self.assertEqual(len(mailed), 4)
-        self.assertEqual(mailed[0], 'To: chrism@plope.com')
-        self.assertEqual(mailed[1],
-          'Subject: memmon: failed to stop process BAD_NAME:BAD_NAME, exiting')
-        self.assertEqual(mailed[2], '')
-        self.failUnless(mailed[3].startswith('Failed'))
-
-
-def make_connection(response, exc=None):
-    class TestConnection:
-        def __init__(self, hostport):
-            self.hostport = hostport
-
-        def request(self, method, path):
-            if exc:
-                raise ValueError('foo')
-            self.method = method
-            self.path = path
-
-        def getresponse(self):
-            return response
-
-    return TestConnection
-
-class DummyResponse:
-    status = 200
-    reason = 'OK'
-    body = 'OK'
-    def read(self):
-        return self.body
-
-class DummyRPCServer:
-    def __init__(self):
-        self.supervisor = DummySupervisorRPCNamespace()
-        self.system = DummySystemRPCNamespace()
-
-class DummySystemRPCNamespace:
-    pass
-
-import time
-from supervisor.process import ProcessStates
-
-_NOW = time.time()
-
-_FAIL = [ {
-        'name':'FAILED',
-        'group':'foo',
-        'pid':11,
-        'state':ProcessStates.RUNNING,
-        'statename':'RUNNING',
-        'start':_NOW - 100,
-        'stop':0,
-        'spawnerr':'',
-        'now':_NOW,
-        'description':'foo description',
-        },
-{
-        'name':'SPAWN_ERROR',
-        'group':'foo',
-        'pid':11,
-        'state':ProcessStates.RUNNING,
-        'statename':'RUNNING',
-        'start':_NOW - 100,
-        'stop':0,
-        'spawnerr':'',
-        'now':_NOW,
-        'description':'foo description',
-        },]
-
-class DummySupervisorRPCNamespace:
-    _restartable = True
-    _restarted = False
-    _shutdown = False
-    _readlog_error = False
-
-
-    all_process_info = [
-        {
-        'name':'foo',
-        'group':'foo',
-        'pid':11,
-        'state':ProcessStates.RUNNING,
-        'statename':'RUNNING',
-        'start':_NOW - 100,
-        'stop':0,
-        'spawnerr':'',
-        'now':_NOW,
-        'description':'foo description',
-        },
-        {
-        'name':'bar',
-        'group':'bar',
-        'pid':12,
-        'state':ProcessStates.FATAL,
-        'statename':'FATAL',
-        'start':_NOW - 100,
-        'stop':_NOW - 50,
-        'spawnerr':'screwed',
-        'now':_NOW,
-        'description':'bar description',
-        },
-        {
-        'name':'baz_01',
-        'group':'baz',
-        'pid':12,
-        'state':ProcessStates.STOPPED,
-        'statename':'STOPPED',
-        'start':_NOW - 100,
-        'stop':_NOW - 25,
-        'spawnerr':'',
-        'now':_NOW,
-        'description':'baz description',
-        },
-        ]
-
-    def getAllProcessInfo(self):
-        return self.all_process_info
-
-    def startProcess(self, name):
-        from supervisor import xmlrpc
-        from xmlrpclib import Fault
-        if name.endswith('SPAWN_ERROR'):
-            raise Fault(xmlrpc.Faults.SPAWN_ERROR, 'SPAWN_ERROR')
-        return True
-
-    def stopProcess(self, name):
-        from supervisor import xmlrpc
-        from xmlrpclib import Fault
-        if name == 'BAD_NAME:BAD_NAME':
-            raise Fault(xmlrpc.Faults.BAD_NAME, 'BAD_NAME:BAD_NAME') 
-        if name.endswith('FAILED'):
-            raise Fault(xmlrpc.Faults.FAILED, 'FAILED')
-        return True
-    
-
-def test_suite():
-    return unittest.findTestCases(sys.modules[__name__])
-
-if __name__ == '__main__':
-    unittest.main(defaultTest='test_suite')

trunk/superlance/tests/__init__.py

+def test_suite():
+    return unittest.findTestCases(sys.modules[__name__])

trunk/superlance/tests/crashmail_test.py

+import sys
+import unittest
+from StringIO import StringIO
+
+class CrashMailTests(unittest.TestCase):
+    def _getTargetClass(self):
+        from superlance.crashmail import CrashMail
+        return CrashMail
+    
+    def _makeOne(self, *opts):
+        return self._getTargetClass()(*opts)
+
+    def setUp(self):
+        import tempfile
+        self.tempdir = tempfile.mkdtemp()
+
+    def tearDown(self):
+        import shutil
+        shutil.rmtree(self.tempdir)
+
+    def _makeOnePopulated(self, programs, any, response=None):
+
+        import os
+        sendmail = 'cat - > %s' % os.path.join(self.tempdir, 'email.log')
+        email = 'chrism@plope.com'
+        header = '[foo]'
+        prog = self._makeOne(programs, any, email, sendmail, header)
+        prog.stdin = StringIO()
+        prog.stdout = StringIO()
+        prog.stderr = StringIO()
+        return prog
+
+    def test_runforever_not_process_state_exited(self):
+        programs = {'foo':0, 'bar':0, 'baz_01':0 }
+        groups = {}
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        prog.stdin.write('eventname:PROCESS_STATE len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        self.assertEqual(prog.stderr.getvalue(), 'non-exited event\n')
+
+    def test_runforever_expected_exit(self):
+        programs = ['foo']
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        payload=('expected:1 processname:foo groupname:bar '
+                 'from_state:RUNNING pid:1')
+        prog.stdin.write(
+            'eventname:PROCESS_STATE_EXITED len:%s\n' % len(payload))
+        prog.stdin.write(payload)
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        self.assertEqual(prog.stderr.getvalue(), 'expected exit\n')
+
+    def test_runforever_unexpected_exit(self):
+        programs = ['foo']
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        payload=('expected:0 processname:foo groupname:bar '
+                 'from_state:RUNNING pid:1')
+        prog.stdin.write(
+            'eventname:PROCESS_STATE_EXITED len:%s\n' % len(payload))
+        prog.stdin.write(payload)
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        output = prog.stderr.getvalue()
+        lines = output.split('\n')
+        self.assertEqual(lines[0], 'unexpected exit, mailing')
+        self.assertEqual(lines[1], 'Mailed:')
+        self.assertEqual(lines[2], '')
+        self.assertEqual(lines[3], 'To: chrism@plope.com')
+        self.failUnless('Subject: [foo]: foo crashed at' in lines[4])
+        self.assertEqual(lines[5], '')
+        self.failUnless(
+            'Process foo in group bar exited unexpectedly' in lines[6])
+        import os
+        mail = open(os.path.join(self.tempdir, 'email.log'), 'r').read()
+        self.failUnless(
+            'Process foo in group bar exited unexpectedly' in mail)
+
+if __name__ == '__main__':
+    unittest.main()

trunk/superlance/tests/crashmailbatch_test.py

+import unittest
+import mock
+import time
+from StringIO import StringIO
+
+class CrashMailBatchTests(unittest.TestCase):
+    fromEmail = 'testFrom@blah.com'
+    toEmail = 'testTo@blah.com'
+    subject = 'Test Alert'
+    now = 1279677400.1
+    unexpectedErrorMsg = '2010-07-20 18:56:40,099 -- Process bar:foo \
+(pid 58597) died unexpectedly'
+    
+    def _getTargetClass(self):
+        from superlance.crashmailbatch import CrashMailBatch
+        return CrashMailBatch
+        
+    def _makeOneMocked(self, **kwargs):
+        kwargs['stdin'] = StringIO()
+        kwargs['stdout'] = StringIO()
+        kwargs['stderr'] = StringIO()
+        kwargs['fromEmail'] = kwargs.get('fromEmail', self.fromEmail)
+        kwargs['toEmail'] = kwargs.get('toEmail', self.toEmail)
+        kwargs['subject'] = kwargs.get('subject', self.subject)
+        kwargs['now'] = self.now
+        
+        obj = self._getTargetClass()(**kwargs)
+        obj.sendEmail = mock.Mock()
+        return obj
+
+    def getProcessExitedEvent(self, pname, gname, expected):
+        headers = {
+            'ver': '3.0', 'poolserial': '7', 'len': '71',
+            'server': 'supervisor', 'eventname': 'PROCESS_STATE_EXITED',
+            'serial': '7', 'pool': 'checkmailbatch',
+        }
+        payload = 'processname:%s groupname:%s from_state:RUNNING expected:%d \
+pid:58597' % (pname, gname, expected)
+        return (headers, payload)
+        
+    def test_getProcessStateChangeMsg_expected(self):
+        crash = self._makeOneMocked()
+        hdrs, payload = self.getProcessExitedEvent('foo', 'bar', 1)
+        self.assertEquals(None, crash.getProcessStateChangeMsg(hdrs, payload))
+
+    def test_getProcessStateChangeMsg_unexpected(self):
+        crash = self._makeOneMocked()
+        hdrs, payload = self.getProcessExitedEvent('foo', 'bar', 0)
+        msg = crash.getProcessStateChangeMsg(hdrs, payload)
+        self.assertEquals(self.unexpectedErrorMsg, msg)
+        
+    def test_handleEvent_exit_expected(self):
+        crash = self._makeOneMocked()
+        hdrs, payload = self.getProcessExitedEvent('foo', 'bar', 1)
+        crash.handleEvent(hdrs, payload)
+        self.assertEquals([], crash.getBatchMsgs())
+        self.assertEquals('', crash.stderr.getvalue())
+
+    def test_handleEvent_exit_unexpected(self):
+        crash = self._makeOneMocked()
+        hdrs, payload = self.getProcessExitedEvent('foo', 'bar', 0)
+        crash.handleEvent(hdrs, payload)
+        self.assertEquals([self.unexpectedErrorMsg], crash.getBatchMsgs())
+        self.assertEquals('%s\n' % self.unexpectedErrorMsg, crash.stderr.getvalue())
+
+if __name__ == '__main__':
+    unittest.main()         

trunk/superlance/tests/dummy.py

+class DummyRPCServer:
+    def __init__(self):
+        self.supervisor = DummySupervisorRPCNamespace()
+        self.system = DummySystemRPCNamespace()
+
+class DummyResponse:
+    status = 200
+    reason = 'OK'
+    body = 'OK'
+    def read(self):
+        return self.body 
+        
+class DummySystemRPCNamespace:
+    pass
+
+
+import time
+from supervisor.process import ProcessStates
+
+_NOW = time.time()
+    
+class DummySupervisorRPCNamespace:
+    _restartable = True
+    _restarted = False
+    _shutdown = False
+    _readlog_error = False
+
+
+    all_process_info = [
+        {
+        'name':'foo',
+        'group':'foo',
+        'pid':11,
+        'state':ProcessStates.RUNNING,
+        'statename':'RUNNING',
+        'start':_NOW - 100,
+        'stop':0,
+        'spawnerr':'',
+        'now':_NOW,
+        'description':'foo description',
+        },
+        {
+        'name':'bar',
+        'group':'bar',
+        'pid':12,
+        'state':ProcessStates.FATAL,
+        'statename':'FATAL',
+        'start':_NOW - 100,
+        'stop':_NOW - 50,
+        'spawnerr':'screwed',
+        'now':_NOW,
+        'description':'bar description',
+        },
+        {
+        'name':'baz_01',
+        'group':'baz',
+        'pid':12,
+        'state':ProcessStates.STOPPED,
+        'statename':'STOPPED',
+        'start':_NOW - 100,
+        'stop':_NOW - 25,
+        'spawnerr':'',
+        'now':_NOW,
+        'description':'baz description',
+        },
+        ]
+
+    def getAllProcessInfo(self):
+        return self.all_process_info
+
+    def startProcess(self, name):
+        from supervisor import xmlrpc
+        from xmlrpclib import Fault
+        if name.endswith('SPAWN_ERROR'):
+            raise Fault(xmlrpc.Faults.SPAWN_ERROR, 'SPAWN_ERROR')
+        return True
+
+    def stopProcess(self, name):
+        from supervisor import xmlrpc
+        from xmlrpclib import Fault
+        if name == 'BAD_NAME:BAD_NAME':
+            raise Fault(xmlrpc.Faults.BAD_NAME, 'BAD_NAME:BAD_NAME') 
+        if name.endswith('FAILED'):
+            raise Fault(xmlrpc.Faults.FAILED, 'FAILED')
+        return True
+

trunk/superlance/tests/fatalmailbatch_test.py

+import unittest
+import mock
+import time
+from StringIO import StringIO
+
+class FatalMailBatchTests(unittest.TestCase):
+    fromEmail = 'testFrom@blah.com'
+    toEmail = 'testTo@blah.com'
+    subject = 'Test Alert'
+    now = 1279677400.1
+    unexpectedErrorMsg = '2010-07-20 18:56:40,099 -- Process bar:foo \
+failed to start too many times'
+    
+    def _getTargetClass(self):
+        from superlance.fatalmailbatch import FatalMailBatch
+        return FatalMailBatch
+        
+    def _makeOneMocked(self, **kwargs):
+        kwargs['stdin'] = StringIO()
+        kwargs['stdout'] = StringIO()
+        kwargs['stderr'] = StringIO()
+        kwargs['fromEmail'] = kwargs.get('fromEmail', self.fromEmail)
+        kwargs['toEmail'] = kwargs.get('toEmail', self.toEmail)
+        kwargs['subject'] = kwargs.get('subject', self.subject)
+        kwargs['now'] = self.now
+        
+        obj = self._getTargetClass()(**kwargs)
+        obj.sendEmail = mock.Mock()
+        return obj
+
+    def getProcessFatalEvent(self, pname, gname):
+        headers = {
+            'ver': '3.0', 'poolserial': '7', 'len': '71',
+            'server': 'supervisor', 'eventname': 'PROCESS_STATE_FATAL',
+            'serial': '7', 'pool': 'checkmailbatch',
+        }
+        payload = 'processname:%s groupname:%s from_state:BACKOFF' \
+                % (pname, gname)
+        return (headers, payload)
+        
+    def test_getProcessStateChangeMsg(self):
+        crash = self._makeOneMocked()
+        hdrs, payload = self.getProcessFatalEvent('foo', 'bar')
+        msg = crash.getProcessStateChangeMsg(hdrs, payload)
+        self.assertEquals(self.unexpectedErrorMsg, msg)
+        
+if __name__ == '__main__':
+    unittest.main()

trunk/superlance/tests/httpok_test.py

+import sys
+import time
+import unittest
+from StringIO import StringIO
+from supervisor.process import ProcessStates
+from superlance.tests.dummy import *
+
+_NOW = time.time()
+
+_FAIL = [ {
+        'name':'FAILED',
+        'group':'foo',
+        'pid':11,
+        'state':ProcessStates.RUNNING,
+        'statename':'RUNNING',
+        'start':_NOW - 100,
+        'stop':0,
+        'spawnerr':'',
+        'now':_NOW,
+        'description':'foo description',
+        },
+{
+        'name':'SPAWN_ERROR',
+        'group':'foo',
+        'pid':11,
+        'state':ProcessStates.RUNNING,
+        'statename':'RUNNING',
+        'start':_NOW - 100,
+        'stop':0,
+        'spawnerr':'',
+        'now':_NOW,
+        'description':'foo description',
+        },]
+
+def make_connection(response, exc=None):
+    class TestConnection:
+        def __init__(self, hostport):
+            self.hostport = hostport
+
+        def request(self, method, path):
+            if exc:
+                raise ValueError('foo')
+            self.method = method
+            self.path = path
+
+        def getresponse(self):
+            return response
+
+    return TestConnection
+
+class HTTPOkTests(unittest.TestCase):
+    def _getTargetClass(self):
+        from superlance.httpok import HTTPOk
+        return HTTPOk
+    
+    def _makeOne(self, *opts):
+        return self._getTargetClass()(*opts)
+
+    def _makeOnePopulated(self, programs, any, response=None, exc=None,
+                          gcore=None, coredir=None, eager=True):
+        if response is None:
+            response = DummyResponse()
+        rpc = DummyRPCServer()
+        sendmail = 'cat - > /dev/null'
+        email = 'chrism@plope.com'
+        url = 'http://foo/bar'
+        timeout = 10
+        status = '200'
+        inbody = None
+        gcore = gcore
+        coredir = coredir
+        prog = self._makeOne(rpc, programs, any, url, timeout, status,
+                             inbody, email, sendmail, coredir, gcore, eager)
+        prog.stdin = StringIO()
+        prog.stdout = StringIO()
+        prog.stderr = StringIO()
+        prog.connclass = make_connection(response, exc=exc)
+        return prog
+
+    def test_listProcesses_no_programs(self):
+        programs = []
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        specs = list(prog.listProcesses())
+        self.assertEqual(len(specs), 0)
+
+    def test_listProcesses_w_RUNNING_programs_default_state(self):
+        programs = ['foo']
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        specs = list(prog.listProcesses())
+        self.assertEqual(len(specs), 1)
+        self.assertEqual(specs[0],
+                         DummySupervisorRPCNamespace.all_process_info[0])
+
+    def test_listProcesses_w_nonRUNNING_programs_default_state(self):
+        programs = ['bar']
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        specs = list(prog.listProcesses())
+        self.assertEqual(len(specs), 1)
+        self.assertEqual(specs[0],
+                         DummySupervisorRPCNamespace.all_process_info[1])
+
+    def test_listProcesses_w_nonRUNNING_programs_RUNNING_state(self):
+        programs = ['bar']
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        specs = list(prog.listProcesses(ProcessStates.RUNNING))
+        self.assertEqual(len(specs), 0, (prog.programs, specs))
+
+    def test_runforever_eager_notatick(self):
+        programs = {'foo':0, 'bar':0, 'baz_01':0 }
+        groups = {}
+        any = None
+        prog = self._makeOnePopulated(programs, any)
+        prog.stdin.write('eventname:NOTATICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        self.assertEqual(prog.stderr.getvalue(), '')
+
+    def test_runforever_eager_error_on_request_some(self):
+        programs = ['foo', 'bar', 'baz_01', 'notexisting']
+        any = None
+        prog = self._makeOnePopulated(programs, any, exc=True)
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = prog.stderr.getvalue().split('\n')
+        #self.assertEqual(len(lines), 7)
+        self.assertEqual(lines[0],
+                         ("Restarting selected processes ['foo', 'bar', "
+                          "'baz_01', 'notexisting']")
+                         )
+        self.assertEqual(lines[1], 'foo is in RUNNING state, restarting')
+        self.assertEqual(lines[2], 'foo restarted')
+        self.assertEqual(lines[3], 'bar not in RUNNING state, NOT restarting')
+        self.assertEqual(lines[4],
+                         'baz:baz_01 not in RUNNING state, NOT restarting')
+        self.assertEqual(lines[5],
+          "Programs not restarted because they did not exist: ['notexisting']")
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 12)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+    def test_runforever_eager_error_on_request_any(self):
+        programs = []
+        any = True
+        prog = self._makeOnePopulated(programs, any, exc=True)
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = prog.stderr.getvalue().split('\n')
+        #self.assertEqual(len(lines), 6)
+        self.assertEqual(lines[0], 'Restarting all running processes')
+        self.assertEqual(lines[1], 'foo is in RUNNING state, restarting')
+        self.assertEqual(lines[2], 'foo restarted')
+        self.assertEqual(lines[3], 'bar not in RUNNING state, NOT restarting')
+        self.assertEqual(lines[4],
+                         'baz:baz_01 not in RUNNING state, NOT restarting')
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 11)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+    def test_runforever_eager_error_on_process_stop(self):
+        programs = ['FAILED']
+        any = False
+        prog = self._makeOnePopulated(programs, any, exc=True)
+        prog.rpc.supervisor.all_process_info = _FAIL
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = prog.stderr.getvalue().split('\n')
+        #self.assertEqual(len(lines), 5)
+        self.assertEqual(lines[0], "Restarting selected processes ['FAILED']")
+        self.assertEqual(lines[1], 'foo:FAILED is in RUNNING state, restarting')
+        self.assertEqual(lines[2],
+                    "Failed to stop process foo:FAILED: <Fault 30: 'FAILED'>")
+        self.assertEqual(lines[3], 'foo:FAILED restarted')
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 10)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+    def test_runforever_eager_error_on_process_start(self):
+        programs = ['SPAWN_ERROR']
+        any = False
+        prog = self._makeOnePopulated(programs, any, exc=True)
+        prog.rpc.supervisor.all_process_info = _FAIL
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = prog.stderr.getvalue().split('\n')
+        #self.assertEqual(len(lines), 4)
+        self.assertEqual(lines[0],
+                         "Restarting selected processes ['SPAWN_ERROR']")
+        self.assertEqual(lines[1],
+                         'foo:SPAWN_ERROR is in RUNNING state, restarting')
+        self.assertEqual(lines[2],
+           "Failed to start process foo:SPAWN_ERROR: <Fault 50: 'SPAWN_ERROR'>")
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 9)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+    def test_runforever_eager_gcore(self):
+        programs = ['foo', 'bar', 'baz_01', 'notexisting']
+        any = None
+        prog = self._makeOnePopulated(programs, any, exc=True, gcore="true",
+                                      coredir="/tmp")
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = prog.stderr.getvalue().split('\n')
+        self.assertEqual(lines[0],
+                         ("Restarting selected processes ['foo', 'bar', "
+                          "'baz_01', 'notexisting']")
+                         )
+        self.assertEqual(lines[1], 'gcore output for foo:')
+        self.assertEqual(lines[2], '')
+        self.assertEqual(lines[3], ' ')
+        self.assertEqual(lines[4], 'foo is in RUNNING state, restarting')
+        self.assertEqual(lines[5], 'foo restarted')
+        self.assertEqual(lines[6], 'bar not in RUNNING state, NOT restarting')
+        self.assertEqual(lines[7],
+                         'baz:baz_01 not in RUNNING state, NOT restarting')
+        self.assertEqual(lines[8],
+          "Programs not restarted because they did not exist: ['notexisting']")
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 15)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+    def test_runforever_not_eager_none_running(self):
+        programs = ['bar', 'baz_01']
+        any = None
+        prog = self._makeOnePopulated(programs, any, exc=True, gcore="true",
+                                      coredir="/tmp", eager=False)
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = filter(None, prog.stderr.getvalue().split('\n'))
+        self.assertEqual(len(lines), 0, lines)
+        self.failIf('mailed' in prog.__dict__)
+
+    def test_runforever_not_eager_running(self):
+        programs = ['foo', 'bar']
+        any = None
+        prog = self._makeOnePopulated(programs, any, exc=True, eager=False)
+        prog.stdin.write('eventname:TICK len:0\n')
+        prog.stdin.seek(0)
+        prog.runforever(test=True)
+        lines = filter(None, prog.stderr.getvalue().split('\n'))
+        self.assertEqual(lines[0],
+                         ("Restarting selected processes ['foo', 'bar']")
+                         )
+        self.assertEqual(lines[1], 'foo is in RUNNING state, restarting')
+        self.assertEqual(lines[2], 'foo restarted')
+        self.assertEqual(lines[3], 'bar not in RUNNING state, NOT restarting')
+        mailed = prog.mailed.split('\n')
+        self.assertEqual(len(mailed), 10)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                    'Subject: httpok for http://foo/bar: bad status returned')
+
+if __name__ == '__main__':
+    unittest.main()

trunk/superlance/tests/memmon_test.py

+import sys
+import unittest
+from StringIO import StringIO
+from superlance.tests.dummy import *
+
+class MemmonTests(unittest.TestCase):
+    def _getTargetClass(self):
+        from superlance.memmon import Memmon
+        return Memmon
+    
+    def _makeOne(self, *opts):
+        return self._getTargetClass()(*opts)
+
+    def _makeOnePopulated(self, programs, groups, any):
+        rpc = DummyRPCServer()
+        sendmail = 'cat - > /dev/null'
+        email = 'chrism@plope.com'
+        memmon = self._makeOne(programs, groups, any, sendmail, email, rpc)
+        memmon.stdin = StringIO()
+        memmon.stdout = StringIO()
+        memmon.stderr = StringIO()
+        memmon.pscommand = 'echo 22%s'
+        return memmon
+        
+    def test_runforever_notatick(self):
+        programs = {'foo':0, 'bar':0, 'baz_01':0 }
+        groups = {}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:NOTATICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        self.assertEqual(memmon.stderr.getvalue(), '')
+
+    def test_runforever_tick_programs(self):
+        programs = {'foo':0, 'bar':0, 'baz_01':0 }
+        groups = {}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 8)
+        self.assertEqual(lines[0], 'Checking programs foo=0, bar=0, baz_01=0')
+        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[2], 'Restarting foo:foo')
+        self.assertEqual(lines[3], 'RSS of bar:bar is 2265088')
+        self.assertEqual(lines[4], 'Restarting bar:bar')
+        self.assertEqual(lines[5], 'RSS of baz:baz_01 is 2265088')
+        self.assertEqual(lines[6], 'Restarting baz:baz_01')
+        self.assertEqual(lines[7], '')
+        mailed = memmon.mailed.split('\n')
+        self.assertEqual(len(mailed), 4)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                         'Subject: memmon: process baz:baz_01 restarted')
+        self.assertEqual(mailed[2], '')
+        self.failUnless(mailed[3].startswith('memmon.py restarted'))
+
+    def test_runforever_tick_groups(self):
+        programs = {}
+        groups = {'foo':0}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 4)
+        self.assertEqual(lines[0], 'Checking groups foo=0')
+        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[2], 'Restarting foo:foo')
+        self.assertEqual(lines[3], '')
+        mailed = memmon.mailed.split('\n')
+        self.assertEqual(len(mailed), 4)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+          'Subject: memmon: process foo:foo restarted')
+        self.assertEqual(mailed[2], '')
+        self.failUnless(mailed[3].startswith('memmon.py restarted'))
+
+    def test_runforever_tick_any(self):
+        programs = {}
+        groups = {}
+        any = 0
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 8)
+        self.assertEqual(lines[0], 'Checking any=0')
+        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[2], 'Restarting foo:foo')
+        self.assertEqual(lines[3], 'RSS of bar:bar is 2265088')
+        self.assertEqual(lines[4], 'Restarting bar:bar')
+        self.assertEqual(lines[5], 'RSS of baz:baz_01 is 2265088')
+        self.assertEqual(lines[6], 'Restarting baz:baz_01')
+        self.assertEqual(lines[7], '')
+        mailed = memmon.mailed.split('\n')
+        self.assertEqual(len(mailed), 4)
+
+    def test_runforever_tick_programs_and_groups(self):
+        programs = {'baz_01':0}
+        groups = {'foo':0}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 7)
+        self.assertEqual(lines[0], 'Checking programs baz_01=0')
+        self.assertEqual(lines[1], 'Checking groups foo=0')
+        self.assertEqual(lines[2], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[3], 'Restarting foo:foo')
+        self.assertEqual(lines[4], 'RSS of baz:baz_01 is 2265088')
+        self.assertEqual(lines[5], 'Restarting baz:baz_01')
+        self.assertEqual(lines[6], '')
+        mailed = memmon.mailed.split('\n')
+        self.assertEqual(len(mailed), 4)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+                         'Subject: memmon: process baz:baz_01 restarted')
+        self.assertEqual(mailed[2], '')
+        self.failUnless(mailed[3].startswith('memmon.py restarted'))
+
+    def test_runforever_tick_programs_norestart(self):
+        programs = {'foo': sys.maxint}
+        groups = {}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 3)
+        self.assertEqual(lines[0], 'Checking programs foo=%s' % sys.maxint)
+        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[2], '')
+        self.assertEqual(memmon.mailed, False)
+
+    def test_stopprocess_fault_tick_programs_norestart(self):
+        programs = {'foo': sys.maxint}
+        groups = {}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        memmon.runforever(test=True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 3)
+        self.assertEqual(lines[0], 'Checking programs foo=%s' % sys.maxint)
+        self.assertEqual(lines[1], 'RSS of foo:foo is 2264064')
+        self.assertEqual(lines[2], '')
+        self.assertEqual(memmon.mailed, False)
+
+    def test_stopprocess_fails_to_stop(self):
+        programs = {'BAD_NAME': 0}
+        groups = {}
+        any = None
+        memmon = self._makeOnePopulated(programs, groups, any)
+        memmon.stdin.write('eventname:TICK len:0\n')
+        memmon.stdin.seek(0)
+        from supervisor.process import ProcessStates
+        memmon.rpc.supervisor.all_process_info =  [ {
+            'name':'BAD_NAME',
+            'group':'BAD_NAME',
+            'pid':11,
+            'state':ProcessStates.RUNNING,
+            'statename':'RUNNING',
+            'start':0,
+            'stop':0,
+            'spawnerr':'',
+            'now':0,
+            'description':'BAD_NAME description',
+             } ]
+        import xmlrpclib
+        self.assertRaises(xmlrpclib.Fault, memmon.runforever, True)
+        lines = memmon.stderr.getvalue().split('\n')
+        self.assertEqual(len(lines), 4)
+        self.assertEqual(lines[0], 'Checking programs BAD_NAME=%s' % 0)
+        self.assertEqual(lines[1], 'RSS of BAD_NAME:BAD_NAME is 2264064')
+        self.assertEqual(lines[2], 'Restarting BAD_NAME:BAD_NAME')
+        self.failUnless(lines[3].startswith('Failed'))
+        mailed = memmon.mailed.split('\n')
+        self.assertEqual(len(mailed), 4)
+        self.assertEqual(mailed[0], 'To: chrism@plope.com')
+        self.assertEqual(mailed[1],
+          'Subject: memmon: failed to stop process BAD_NAME:BAD_NAME, exiting')
+        self.assertEqual(mailed[2], '')
+        self.failUnless(mailed[3].startswith('Failed'))
+        
+if __name__ == '__main__':
+    unittest.main()  

trunk/superlance/tests/process_state_email_monitor_test.py

+import unittest
+import mock
+import time
+from StringIO import StringIO
+
+class ProcessStateEmailMonitorTests(unittest.TestCase):
+    fromEmail = 'testFrom@blah.com'
+    toEmail = 'testTo@blah.com'
+    subject = 'Test Alert'
+    
+    def _getTargetClass(self):
+        from superlance.process_state_email_monitor \
+        import ProcessStateEmailMonitor
+        return ProcessStateEmailMonitor
+        
+    def _makeOneMocked(self, **kwargs):
+        kwargs['stdin'] = StringIO()
+        kwargs['stdout'] = StringIO()
+        kwargs['stderr'] = StringIO()
+        kwargs['fromEmail'] = kwargs.get('fromEmail', self.fromEmail)
+        kwargs['toEmail'] = kwargs.get('toEmail', self.toEmail)
+        kwargs['subject'] = kwargs.get('subject', self.subject)
+        
+        obj = self._getTargetClass()(**kwargs)
+        obj.sendEmail = mock.Mock()
+        return obj
+    
+    def test_sendBatchNotification(self):
+        testMsgs = ['msg1', 'msg2']
+        monitor = self._makeOneMocked()
+        monitor.batchMsgs = testMsgs
+        monitor.sendBatchNotification()
+        
+        #Test that email was sent
+        self.assertEquals(1, monitor.sendEmail.call_count)
+        emailCallArgs = monitor.sendEmail.call_args[0]
+        self.assertEquals(1, len(emailCallArgs))
+        expected = {
+            'body': 'msg1\nmsg2',
+            'to': 'testTo@blah.com',
+            'from': 'testFrom@blah.com',
+            'subject': 'Test Alert',
+        }
+        self.assertEquals(expected, emailCallArgs[0])
+        
+        #Test that email was logged
+        self.assertEquals("""Sending notification email:
+To: testTo@blah.com
+From: testFrom@blah.com
+Subject: Test Alert
+Body:
+msg1
+msg2
+""", monitor.stderr.getvalue())
+        
+    def test_logEmail_with_body_digest(self):
+        monitor = self._makeOneMocked()
+        email = {
+            'to': 'you@fubar.com',
+            'from': 'me@fubar.com',
+            'subject': 'yo yo',
+            'body': 'a' * 30,
+        }
+        monitor.logEmail(email)
+        self.assertEquals("""Sending notification email:
+To: you@fubar.com
+From: me@fubar.com
+Subject: yo yo
+Body:
+aaaaaaaaaaaaaaaaaaaa...
+""", monitor.stderr.getvalue())
+        self.assertEquals('a' * 30, email['body'])
+
+    def test_logEmail_without_body_digest(self):
+        monitor = self._makeOneMocked()
+        email = {
+            'to': 'you@fubar.com',
+            'from': 'me@fubar.com',
+            'subject': 'yo yo',
+            'body': 'a' * 20,
+        }
+        monitor.logEmail(email)
+        self.assertEquals("""Sending notification email:
+To: you@fubar.com
+From: me@fubar.com
+Subject: yo yo
+Body:
+aaaaaaaaaaaaaaaaaaaa
+""", monitor.stderr.getvalue())
+
+if __name__ == '__main__':
+    unittest.main()

trunk/superlance/tests/process_state_monitor_test.py

+import unittest
+import mock
+import time