Need Commandline Functional Test Runner For Commandline tools

Create issue
Issue #5 open
Noah Gift created an issue

I work in Film, and we make a lot of commandline tools, and I mean a lot! I am looking for something that can perform a functional test to automatically test the commandline tools. I think this could fill a very huge need in the Python community that hasn't really been tackled.

This would be my wish list, or "cli test pony":

  1. Be able to test all options and error conditions.

  2. Easily mock operations that take too long to test.

  3. Would be nice if it could be run easily in a nightly test.

  4. It would be nice to have a "frameworkish" tool that builds the boilerplate for the test, perhaps based on some convention or config file. For example, many places often start to offer the same options for every tool that make.

  5. Be able to run tests under many different shell environments. We often have thousands of different environmental variable configurations, so it would be nice to test potentially against all of them. I know, I know, this is a pain.

Comments (5)

  1. Holger Krekel repo owner
    • changed status to open

    Hi Noah,

    thanks for the submission. I think i understand the need and i'd like to take some steps towards these stories you hint at. Could you help by writing down some example test scenarios for your 1. and 2. points?

    I am particularly interested in what you mean with "mock operations that take too long to test" - as we are talking about running external shell scripts, how do you envision this to get specified and executed?

    take your time, i am away for vacation until end april, after which i'd like to at least factor out and document the current support that py.test has for testing command line tools. For reference i include the pointer again to the current functional/acceptance tests that py.test performs on its own tools: (But i think you saw that when we twittered already).

    cheers, holger

  2. Log in to comment