Inconsistent behavior with async=True

[Dmitry Malinovsky]

Hi,

I found that I can't specify sleep n call before any other command (including another sleep n call) without waiting for it using async=True. Is it expected behavior?

This works as expected:

In [1]: from sarge import run

In [2]: p = run('echo abc | cat && sleep 5', async=True)
# Returns immediately
abc

In [3]: p.commands[-1].poll()

In [4]: p.commands[-1].poll()
Out[4]: 0

This is not:

In [5]: p = run('sleep 5 && echo abc | cat', async=True)
# Hangs for 5 seconds
abc

[Vinay Sajip]

The && makes that part of the pipeline synchronous: the decision on whether or not to run echo can't be taken until sleep returns.

I'll investigate why it doesn't return immediately when I get a chance.

[Dmitry Malinovsky]

Unfortunately, behavior is the same when using a semicolon:

In [1]: from sarge import run

In [2]: p = run('sleep 5; echo abc | cat', async=True)
# Hangs for 5 seconds
abc

[Vinay Sajip]

Unfortunately, behavior is the same when using a semicolon

Yes, for the same reason - the ; makes that part of the pipeline synchronous. The following script:

import logging
import sys
import time

from sarge import run

try:
    raw_input   # for Python 3.x
except NameError:
    raw_input = input

logging.basicConfig(level=logging.DEBUG, filename='test_sleep.log',
                    filemode='w', format='%(asctime)s %(threadName)-10s %(name)-15s %(lineno)4d %(message)s')
start = time.time()
p = run('sleep 5 & echo abc', async=True)
elapsed = time.time() - start
print('Elapsed: %.1f secs.' % elapsed)
# This is there to stop the program exiting prematurely (note that sarge
# threads are daemon threads).
raw_input('Press enter:')

Uses & rather than && or ;, and so will run asynchronously (returns immediately from the run() call). Partial log:

2014-07-09 21:36:30,092 MainThread sarge           1301 started: ListNode(parts=[CommandNode(command=['sleep', '5'] redirects={}), SyncNode(sync=&), CommandNode(command=['echo', 'abc'] redirects={})]), None, True
2014-07-09 21:36:30,092 MainThread sarge           1334 thread Thread-1 started to run node: CommandNode(command=['sleep', '5'] redirects={})
2014-07-09 21:36:30,092 MainThread sarge           1334 thread Thread-2 started to run node: CommandNode(command=['echo', 'abc'] redirects={})
2014-07-09 21:36:30,092 Thread-1   sarge           1215 started: CommandNode(command=['sleep', '5'] redirects={}), None, False
2014-07-09 21:36:30,092 Thread-1   sarge            585 Command('sleep 5') created
2014-07-09 21:36:30,092 Thread-1   sarge            621 About to call Popen: ['sleep', '5'], {'stdin': None, 'stderr': None, 'stdout': None}
2014-07-09 21:36:30,099 Thread-1   sarge            624 Popen: Command('sleep 5'), {'stdin': None, 'stderr': None, 'stdout': None} -> {'_child_created': True, 'returncode': None, 'stdout': None, 'stdin': None, 'pid': 2050, 'stderr': None, 'universal_newlines': False}
2014-07-09 21:36:30,100 Thread-1   sarge            643 about to wait for process
2014-07-09 21:36:30,101 Thread-2   sarge           1215 started: CommandNode(command=['echo', 'abc'] redirects={}), None, False
2014-07-09 21:36:30,101 Thread-2   sarge            585 Command('echo abc') created
2014-07-09 21:36:30,101 Thread-2   sarge            621 About to call Popen: ['echo', 'abc'], {'stdin': None, 'stderr': None, 'stdout': None}
2014-07-09 21:36:30,109 Thread-2   sarge            624 Popen: Command('echo abc'), {'stdin': None, 'stderr': None, 'stdout': None} -> {'_child_created': True, 'returncode': None, 'stdout': None, 'stdin': None, 'pid': 2051, 'stderr': None, 'universal_newlines': False}
2014-07-09 21:36:30,110 Thread-2   sarge            643 about to wait for process
2014-07-09 21:36:30,112 Thread-2   sarge            647 returning Command('echo abc') (Popen(returncode=0 stdin=None stdout=None stderr=None))
2014-07-09 21:36:35,106 Thread-1   sarge            647 returning Command('sleep 5') (Popen(returncode=0 stdin=None stdout=None stderr=None))

[Dmitry Malinovsky]

I do not understand why a semicolon makes the call to be synchronous, since it is just a delimeter, and a command after a semicolon will always run, despite of the result of previous command.

[Vinay Sajip]

That's just how bash works - see for example the first answer on this page.

[Dmitry Malinovsky]

I see, the order matters. Thanks for the tips!

[Vinay Sajip]

Closes #20: Run pipeline in separate thread if async.

→ <<cset 0b19f818aa7a>>