Unstable/inconsistent behaviour of progress.py example code

In want to use the sarge library to execute a simple cmd1 | cmd2 shell command from my Python script an grab it's output (both stdout and stderr) live while the shell command executes. I was previously using subprocess.Popen() + subprocess.PIPE + communicate() but the output was buffered.

As a working base I tested the progress.py code detailed in the tutorial.

File "test_sarge_live.py":

import optparse # because of 2.6 support
import sys
import threading
import time
import logging

from sarge import capture_stdout

logging.basicConfig(level=logging.DEBUG, filename='/tmp/test_sarge_live.log',
                    filemode='w', format='%(asctime)s %(threadName)-10s %(name)-15s %(lineno)4d %(message)s')

def progress(capture, options):
    lines_seen = 0
    messages = {
        'line 25\n': 'Getting going ...\n',
        'line 50\n': 'Well on the way ...\n',
        'line 75\n': 'Almost there ...\n',
    }
    while True:
        s = capture.readline()
        if not s and lines_seen:
            break
        if options.dots:
            sys.stderr.write('.')
        else:
            msg = messages.get(s)
            if msg:
                sys.stderr.write(msg)
        lines_seen += 1
    if options.dots:
        sys.stderr.write('\n')
    sys.stderr.write('Done - %d lines seen.\n' % lines_seen)

def main():
    parser = optparse.OptionParser()
    parser.add_option('-n', '--no-dots', dest='dots', default=True,
                      action='store_false', help='Show dots for progress')
    options, args = parser.parse_args()

    #~ p = capture_stdout('ncat -k -l -p 42421', async=True)
    p = capture_stdout('python lister.py -d 0.1 -c 100', async=True)

    t = threading.Thread(target=progress, args=(p.stdout, options))
    t.start()

    while(p.returncodes[0] is None):
        # We could do other useful work here. If we have no useful
        # work to do here, we can call readline() and process it
        # directly in this loop, instead of creating a thread to do it in.
        p.commands[0].poll()
        time.sleep(0.05)
    t.join()

if __name__ == '__main__':
    sys.exit(main())

But running it gives very inconsistent output:

On Ubuntu v16.04 using Python v2.7.12 and sarge v0.1.4 I get theses (at random):

This ('NoneType' object has no attribute 'returncode' + sys.excepthook is missing)

Traceback (most recent call last):
File "test_sarge_live.py", line 55, in <module>
  sys.exit(main())
File "test_sarge_live.py", line 46, in main
  while(p.returncodes[0] is None):
File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1072, in returncodes
  result = [c.process.returncode for c in self.commands]
AttributeError: 'NoneType' object has no attribute 'returncode'
..
Done - 2 lines seen.
close failed in file object destructor:                                                                                                                                                                                                                         
sys.excepthook is missing
lost sys.stderr

Or also this (no 'NoneType' object has no attribute 'returncode')

close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

Or even this:

Traceback (most recent call last):
.  File "test_sarge_live.py", line 55, in <module>

sys.exit(main())Done - 1 lines seen.

File "test_sarge_live.py", line 46, in main
  while(p.returncodes[0] is None):
IndexError: list index out of range
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
File "/usr/lib/python2.7/threading.py", line 754, in run
File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1136, in run_node
File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1282, in run_command_node
File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 656, in run
File "/usr/lib/python2.7/threading.py", line 585, in set
File "/usr/lib/python2.7/threading.py", line 407, in notifyAll
<type 'exceptions.TypeError'>: 'NoneType' object is not callable

And sometimes this no failure output:

.
Done - 1 lines seen.

I also tested on Docker container Debian v9.4 using Python v2.7.13 and sarge v0.1.4: I get the same outputs.

When I kill (via [Ctrl]+[C]) one instance and immediately run the script again I get:

.Traceback (most recent call last):
  File "test_sarge_live.py", line 55, in <module>
    sys.exit(main())
  File "test_sarge_live.py", line 46, in main
    while(p.returncodes[0] is None):
IndexError: list index out of range

Done - 1 lines seen.
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
  File "/usr/lib/python2.7/threading.py", line 754, in run
  File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1136, in run_node
  File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 1282, in run_command_node
  File "/usr/local/lib/python2.7/dist-packages/sarge/__init__.py", line 656, in run
  File "/usr/lib/python2.7/threading.py", line 585, in set
  File "/usr/lib/python2.7/threading.py", line 407, in notifyAll
<type 'exceptions.TypeError'>: 'NoneType' object is not callable
root@1ad9fddcff28:/app# close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

but I think this is normal.

Is the tutorial code still accurate?

Comments (8)