Anonymous avatar Anonymous committed 68434fc

Added explicit LICENSE file, and PyPI classifiers

Comments (0)

Files changed (3)

+
+                                      LICENSE
+
+
+ classh is released under the PSF (Python Software Foundation License) as 
+ included below. The full text from which this was derived is at:
+
+        http://www.python.org/download/releases/2.6.2/license
+
+ In layman's language, here are the primary features of this license. 
+ The following description is not legal advice; read the full text of 
+ the license (below) and consult qualified professional counsel for an 
+ interpretation of the license terms as they apply to you.
+
+    * classh is absolutely free, even for commercial use (including resale). 
+      You can sell a product including classh or a product that imports, 
+      embeds or otherwise integrates it without incurring any licensing fees.
+
+    * The Open Source Initiative has certified the Python license as 
+      Open Source, and includes it on their list of open source licenses.
+
+    * There is no GPL-like "copyleft" restriction. There is no requirement
+      to release any of your source code.
+
+    * You cannot remove the copyright notice from either the source code
+      nor any resulting binary distribution (egg, RPM, dpkg, etc) packages.
+
+ This software is copyrighted (c) 2009 under United States and international 
+ law by James T. Dennis.
+
+
+  ============================================================================
+
+    PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
+    --------------------------------------------
+
+    1. This LICENSE AGREEMENT is between James T. Dennis (hereafter referred
+       to simply as Jim, and the Individual or Organization ("Licensee") 
+       accessing and otherwise using this software ("classh") in source or 
+       binary form and its associated documentation.
+
+    2. Subject to the terms and conditions of this License Agreement, Jim
+       hereby grants Licensee a nonexclusive, royalty-free, world-wide
+       license to reproduce, analyze, test, perform and/or display publicly,
+       prepare derivative works, distribute, and otherwise use Python
+       alone or in any derivative version, provided, however, that Jim's
+       License Agreement and notice of copyright, i.e., "Copyright (c)
+       2009, James T. Dennis, All Rights Reserved" are retained in classh
+       itself and any derivative work thereof.
+
+    3. In the event Licensee prepares a derivative work that is based on
+       or incorporates classh or any part thereof, and wants to make
+       the derivative work available to others as provided herein, then
+       Licensee hereby agrees to include in any such work a brief summary of
+       the changes made to classh.
+
+    4. Jim is making Python available to Licensee on an "AS IS"
+       basis.  Jim MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
+       IMPLIED.  BY WAY OF EXAMPLE, BUT NOT LIMITATION, Jim MAKES NO AND
+       DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR 
+       FITNESS FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF classh WILL 
+       NOT INFRINGE ANY THIRD PARTY RIGHTS.
+
+    5. Jim SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
+       FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
+       A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON,
+       OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY 
+       THEREOF.
+
+    6. This License Agreement will automatically terminate upon a 
+       material breach of its terms and conditions.
+
+    7. Nothing in this License Agreement shall be deemed to create any
+       relationship of agency, partnership, or joint venture between Jim and
+       Licensee.  This License Agreement does not grant permission to use Jim's
+       name, image, nor trademarks to endorse or promote products or 
+       services of Licensee, or any third party.
+
+    8. By copying, installing or otherwise using classh, Licensee
+       agrees to be bound by the terms and conditions of this License
+       Agreement.
+
+ 
+
+
+#!/usr/bin/env python
+##############################################################################
+'''Provide an easy way to concurrently run ssh jobs on a large number of 
+   targets, gather output, error messages and results from each and handle 
+   timeouts.
+'''
+
+__author__='''Jim Dennis <answrguy@gmail.com>'''
+__url__   ='''http://bitbucket.org/jimd/classh/'''
+__license___='''BSD'''
+
+
+from subprocess import Popen, PIPE
+from time import sleep, time
+import sys, os, string, signal
+## from signal import alarm
+
+
+def debug(str):   # TODO: use logging package?
+    if 'DEBUG' in os.environ:
+        print >> sys.stderr, str
+
+class SSHJobMan(object):
+    '''SSH Job Manager
+       Given a command and a list of targets (hostnames), concurrently
+       run the command on some number (poolsize) of the targets,
+       gathering all the results (output, error messages and exit values)
+       and handling any timeouts (jobtimeout)
+       
+       Maintains a dictionary of results, each item of which is a dictionary
+       containing the specified types fo results.
+    '''
+
+
+    def __init__(self, hostlist=None, cmd=None, **opts):
+        '''
+        '''
+        self.started = False
+        self.poolsize = 50
+        self.pool = dict()         # for maintain host/process data
+        self.results = dict()  
+        self.jobtimeout = 300
+        self.ssh = [ '/usr/bin/ssh' ] # TODO: dynamically find this
+        self.ssh_args = [ '-oBatchMode=yes'] #TODO: add other batch friendly options?
+
+        self.targets = hostlist[:]
+        self.cmd = cmd 
+
+        self.__dict__.update(opts)  # Ugly hack?
+        
+
+    def start(self):
+        '''Set instance started flag
+           Prime the pool with new jobs
+
+           This is deferred from initialization to
+           allow the caller to hook self.poll() into
+           any signal handling (SIGCHILD, SIGALARM), etc.
+        '''
+
+        debug ('Starting')
+        self.add_jobs()
+        self.started = time()
+        
+
+    def add_jobs(self):
+        '''Called from start and from poll to fill the pool with
+           subprocesses.  While the start method primes the pool, 
+           poll method keeps it topped off until all targets have been
+           handled.
+
+           Jobs are added to the pool and to the results with a unique
+           key.  The poll method removes completed jobs from the pool
+           calling the gather method to save their results.  The key
+           is used to keep track of each job even if multiple jobs
+           go to the same target.  (If there are duplicate targets than
+           the additional keys will be of the form: hostname:XX)
+        '''
+
+        while self.targets and len(self.pool.keys()) < self.poolsize:
+            debug('adding jobs')
+            key   = self.targets.pop()
+            host  = key
+            if key in self.results:
+                x = 0 
+                while key + ':' + str(x) in self.results:  # Unique-ify a key
+                    x += 1
+                key = key + ':' + str(x) 
+            self.results[key] = {   # Placeholder  + start time
+              'Job:': host,
+              'Start': time() }
+            proc = Popen(self.ssh + self.ssh_args + [host] + [self.cmd], 
+	             stdout=PIPE,stderr=PIPE, close_fds=True)
+            self.pool[key] = (host, proc, time()) # start time for timeouts
+            debug('... added job %s' % key)
+
+    def poll(self):
+        '''Scan pool for completed jobs, 
+           call gather() for any of those
+           remove completed jobs from pool
+           call add_jobs() to top off pool
+           return list of completed job keys
+        '''
+        
+        debug ('polling')
+        reaped = list() 
+        for key, job in self.pool.items():
+            (host, proc, starttime) = job
+            rv = proc.poll()
+            if rv is not None:
+                debug('job %s is done' % key)
+                self.gather(key, host, rv, proc)
+                del self.pool[key]
+                reaped.append(key)
+            # Handle timeouts:
+            elapsed = time() - starttime
+            if elapsed > self.jobtimeout:
+                debug('job %s timed out after %s seconds' % (key, elapsed))
+                self.kill(proc)
+                self.gather(key, host, proc.poll(), proc)
+                del self.pool[key]
+                reaped.append(key)
+        debug ('reaped %s jobs' % len(reaped))
+        self.add_jobs()
+        return reaped
+
+    def gather(self, key, hostname, exitval, proc):
+        '''Gather results from a subprocess
+           These are stored as a dictionary of dictionaries
+        '''
+        debug ('gathering')
+        (out,err) = proc.communicate()
+        self.results[key] = {
+              'Job': hostname,
+              'Ret': exitval,
+              'Out': out,
+              'Err': err,
+              'Start': self.results[key]['Start'],
+              'End': time()
+              }
+            
+    def kill(self, proc):
+        '''Kill a subprocess which has exceeded its timeout
+           called by poll()
+        '''
+        debug('Killing %s' % proc.pid)
+        try:
+            os.kill(proc.pid, signal.SIGTERM)
+        except OSError:
+            debug('Trying SIGKILL!')
+            try:
+                os.kill(proc.pid, signal.SIGKILL)
+            except OSError:
+                debug('Ignoring os.kill OSError')
+
+    def done(self):
+        '''We're done when we have been started and
+           we have zero remaining jobs in the pool
+        '''
+        debug ('done called with %s remaining jobs' % len(self.pool.keys()))
+        return self.started and not len(self.pool.keys())
+
+
+# Use in print results to munge non-printable chars
+filter = string.maketrans(
+           string.translate(string.maketrans('',''),
+           string.maketrans('',''),string.printable[:-5]),
+           '.'*len(string.translate(string.maketrans('',''),
+           string.maketrans('',''),string.printable[:-5])))
+
+
+def print_results(key, res):
+    errs = ' '.join(res['Err'].split('\n'))
+    outs = ' '.join(res['Out'].split('\n')) 
+    errs = errs[:60].translate(filter)
+    outs = outs[:60].translate(filter)
+
+    pfix = ''.join(["%-48s" % key, "%-5s" % res['Ret'], 
+        "(%s)" % ("%0.2f" % (res['End'] - res['Start']))])
+    if res['Ret']:
+        print pfix, "\tErrors: ", errs[:40]
+    print pfix, "\tOutput: ", outs[:40]
+            
+def summarize_results(res):
+    success = 0
+    errs = 0
+    timeouts = 0
+    for i in res.values():
+        if    i['Ret'] == 0: success += 1
+        elif  i['Ret'] >  0: errs += 1
+        elif  i['Ret'] <  0: timeouts += 1
+    print "\n\n\tSuccessful: %s\tErrors: %s\tSignaled(timeouts): %s" \
+      % (success, errs, timeouts)
+
+
+def readfile(fname):
+    results = []
+    try: 
+        f = open(fname)
+    except EnvironmentError, e:
+        print >> sys.stderr, "Error reading %s, %s" % (fname, e)
+    for line in f:
+        if line.endswith('\n'):
+            line = line[:-1].strip()
+        results.append(line)
+    return results
+
+if __name__ == "__main__":
+    '''Stupid simple test code and wrapper:
+       first argument is command, rest are hosts
+       Dispatch jobs to hosts and incrementally print
+       results
+    '''
+    start = time()
+    if len(sys.argv[1:]) < 2:
+        print >> sys.stderr, "Must specify a command and a list of hosts"
+        sys.exit(1)
+
+    cmd   = sys.argv[1]
+    hosts = list()
+
+    for arg in sys.argv[2:]:
+        if '/' in arg:
+            # it's a filename so:
+            hosts.extend(readfile(arg))
+        else:
+            ## TODO: expand host range expressions
+            hosts.append(arg)
+
+    print >> sys.stderr, "About to run '%s' on %s hosts...\n\n" \
+        % (cmd, len(hosts))
+
+
+    job = SSHJobMan(hosts, cmd)
+    job.start()
+
+    completed = None
+    while not job.done():
+        completed = job.poll()
+        sleep(0.2)
+        if completed:
+            for each in completed:
+                print_results(each, job.results[each])
+        
+    summarize_results(job.results)
+    print "Completed in %s seconds" % (time() - start)
+       
     version='0.01',
     author='James T. Dennis',
     author_email='answrguy@gmail.com',
+    license='PSF',
+    url='http://bitbucket.org/jimd/classh/',
+    packages=['classh'],
+    classifiers=[
+        "Development Status :: 3 - Alpha",
+        "Environment :: Console",
+        "Intended Audience :: Information Technology",
+        "Intended Audience :: System Administrators",
+        "License :: OSI Approved :: Python Software Foundation License",
+        "Natural Language :: English",
+        "Operating System :: POSIX",
+        "Operating System :: Unix",
+        "Programming Language :: Python",
+        "Topic :: System :: Clustering",
+        "Topic :: System :: Systems Administration",
+        "Topic :: Utilities"
+    ],
     description="Cluster Administrators' ssh Wrapper",
-    url='http://bitbucket.org/jimd/classh/',
-    packages=['classh']
+    long_description='''
+
+    classh is yet another wrapper around ssh for running jobs on multiple 
+    targets.  It can support tens of thousands of targets (tested on over 
+    25,000 at once) and runs a configurable number of jobs concurrently. 
+    It separately gathers results, output and error messages, displaying 
+    summary/status information it comes in (asynchronously) and more 
+    detailed data after all jobs have completed.
+
+    classh provides an SSHJobMan class which can be imported into your own
+    Python code and easily used to handle specialized display or other 
+    disposition of results.  (For example the names of all hosts on which
+    the job succeeded can be fed into another process while various failure
+    modes can be tested and fed into other processes).
+
+    '''
     )
  
 
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.