Issue #3 open

Add real-world example to the documentation

Sven Marnach
created an issue

I read your call to download and evaluate this module on python-ideas.

After reading the documentation, It's still unclear to me what I would use this module for. I do a lot of shell-script-like stuff in Python, but I usually don't want to invoke the shell, partly because of the quoting hell I'd get into, but partly because I chose to use Python because I want to write Python, not shell. I can see how this module alleviates the quoting hell -- in current stable Python there is not even a library function to quote for the shell, except for the undocumented pipes.quote().

I certainly wouldn't invoke the shell for any of the examples given in the documentation. The most complex example given in the docs is {{{ for line in iter_shell_output("ls -l .py | tee {}", "example file.txt"): print(line) }}} This doesn't seem exactly "real-world" to me, and if I'd need something like this, it would involve iterating over glob.iglob(".py") instead of invoking the shell.

I appreciate that my personal preferences are not everyone's preferences, and that there are areas where the shell is simply more convenient than Python. It would be nice to have examples for those areas in the docs, though. This would make it easier to judge whether the module is actually an appropriate solution, or if it would be better to improve Python's capabilities in the areas where it is inferior. The only hint I can find in the current docs is that the idea is "[to use] the underlying system shell normally for command invocation and pipeline manipulation." Command invocation can be done with '', which would leave pipeline manipulation. This might indeed be an area where Python is lacking an appropriate library (I'm not counting the 'pipes' module as an appropriate solution).

Don't get this wrong as some kind of fundamental criticism -- I'm honestly struggling to grasp the scope of this module.

Comments (4)

  1. Nick Coghlan repo owner
    • changed status to open

    Not a problem - I have my own concerns about the suitability of this approach (which is why it's a module on PyPI, rather than already in subprocess).

    In Python 3, shlex.quote() is an officially supported and documented API to complement shlex.split(). When it's available, shell-command actually uses that (otherwise it uses a copy-and-paste version taken from 3.2).

    Currently, getting shell-style invocation in Python (without third party library support) means using: - os.path.expandvars for environment variable references - subprocess.Popen directly to get pipelining - the glob module to get globbing - shlex.quote together with str.format (or another formatting mechanism) to handle whitespace and shell metacharacters correctly

    If you use "shell=True" in the subprocess APIs you get access to the first three of those for free (using platform native syntax), but still have to deal with quoting manually.

    Shell command aims to tackle *all four* of those problems, by merging string interpolation with implicit quoting of the interpolated values, all executed via subprocess with "shell=True". That means Windows admins can use Windows shell syntax and *nix admins can use POSIX shell syntax, and the necessary code to achieve it is barely a couple of dozen lines long.

    I thought I mentioned these benefits in the shell command docs, but apparently not. It must have been in a mailing list post at some point.

    Now, there's still an open question as to whether "just use the system shell" is the right approach, or whether a custom POSIX-based mini-shell like that in Julia would be a better idea. That's one of the reasons I've backed off from the idea of adding these to subprocess for 3.3.

  2. Sven Marnach reporter

    Thanks for the clarification.

    The documentation already mentions pipelines, and there's an example using globbing. Maybe these two points could be made a bit more explicit. Environment variable expansion isn't mentioned at all, but certainly something that is much more convenient via the shell.

    On a side note, shlex.quote() isn't in any stable Python release yet. It's new in version 3.3.

  3. Log in to comment