Capturing output from an infinite cycle

[Dmitry Malinovsky]

Hi,

I tried to run a simple script: run_forever.py

import time

i = 0
while True:
    time.sleep(1)
    print(i)
    i += 1

with sarge as described in the docs:

In [1]: import sarge

In [2]: p = sarge.run('python run_forever.py', async=True, stdout=sarge.Capture())

However, I can't capture the output:

In [3]: p.stdout.readline()
Out[3]: b''

In [4]: p.commands[0].stdout.readline()
Out[4]: b''

Am I doing it wrong? How can I achieve that?

[Vinay Sajip]

I think it's a buffering issue. With your run_forever script, the following script illustrates what you need to do:

# read_forever.py

import datetime
import sys

import sarge

c = sarge.Capture()
p = sarge.run('python -u run_forever.py', async=True, stdout=c)
try:
    while True:
        s = c.readline()
        if s:
            now = datetime.datetime.now().strftime('%H:%M:%S')
            sys.stdout.write('%s: %s' % (now, s))
finally:
    p.commands[0].process.terminate()

Note the use of -u to run Python unbuffered in the child process.

When I run this, I get the following output:

$ python read_forever.py 
11:08:20: 0
11:08:21: 1
11:08:22: 2
11:08:23: 3
11:08:24: 4
11:08:25: 5
 (and so on, indefinitely)

[Vinay Sajip]

You can dispense with the -u if you insert a sys.stdout.flush() after the print() in run_forever.py. So, it's not a bug in sarge.

[Dmitry Malinovsky]

Thanks for the clarification!

[Vinay Sajip]

No problem. See the docs about buffering.

[Dmitry Malinovsky]

Just want to add 2 cents - I have to call python -u read_forever.py or add sys.stdout.flush() after sys.stdout.write in read_forever.py to get the output (I'm using python 3.3.4).

[Vinay Sajip]

I'm surprised, though you do have to add a .decode('utf-8') for Python 3 on the results of c.readline() because that is a byte-string. It works fine (on Python 3.4 - I don't have 3.3 installed) with just the decode() added, and using sys.executable to invoke run_forever with the same interpreter as read_forever.

$ python3.4 read_forever.py
running: /usr/local/bin/python3.4 run_forever.py
11:42:58: 0
11:42:59: 1
11:43:00: 2
11:43:01: 3
11:43:02: 4