Issue #357 resolved

Selecting tests with -k doesn't work as expected

Brianna Laugher avatarBrianna Laugher created an issue

Using platform linux2 -- Python 2.7.5 -- pytest-2.3.5

I suspect this is related to the changes in d16cc2d which went into 2.3.4. Basically I find it a lot harder now to select tests, especially parametrised ones.

Here are some examples:

$py.test tests/system/combined/test_marineWindWarnings.py --collectonly
=============== test session starts ===============
platform linux2 -- Python 2.7.5 -- pytest-2.3.5
plugins: xdist, cov
collected 6 items 
<Module 'tests/system/combined/test_marineWindWarnings.py'>
  <Function 'test_windSpeedsTreatedConsistently[25.4-<None>-25]'>
  <Function 'test_windSpeedsTreatedConsistently[25.5-CW1.W:STR-30]'>
  <Function 'test_windSpeedsTreatedConsistently[33.4-CW1.W:STR-30]'>
  <Function 'test_windSpeedsTreatedConsistently[33.5-CW1.W:GALE-35]'>
  <Function 'test_windSpeedsTreatedConsistently[63.4-CW1.W:STO-60]'>
  <Function 'test_windSpeedsTreatedConsistently[63.5-CW1.W:HURR-65]'>
  • If I run -k 'None', I get no tests selected (rather than 1).
  • If I run -k '<None>', I get a SyntaxError!
  • If I run -k '33.4', I get 6 tests run! No idea why. Same with '25.4'. And even '25'. I have no idea how it is handling numbers.
  • If I run -k 'test_windSpeedsTreatedConsistently[33.4-CW1.W:STR-30]' (which is what I used to do) I get another SyntaxError that shows that my string has been changed to test_windSpeedsTreatedConsistently[33.4not CW1.W:STRnot 30].

It seems to me at the moment there is no way I can select only this first test.

To be honest, I really prefer the old behaviour of -k. Being able to run collectonly and then copy a test name into -k was useful.

Related, I can't find an explanation in the documentation about which characters/words are "special" and not treated as a flat string. Am I writing a regexp? I can see "-" is converted to "not"; what about other punctuation? The usage docs say

py.test -k string     # only run tests whose names contain a string

which is basically not true right now.

Comments (9)

  1. holger krekel

    Most people like the new -k and -m syntax ASFAIK but it is more limited with matching against param values, indeed, as you analyze. Not sure atm how to remedy it -- we could fall back to the old behaviour if the expression is a syntax error or introduce a specific parameter match option. Like --testmatch p 'part_of_name_or_param' which would do a string match. Or -t for short.

  2. holger krekel

    actually have a better idea (maybe): if the -k expression contains no spaces, revert back to the old behaviour of a simple string match and "-TEXT" as negating. This should have avoided your surprises, right?

  3. Brianna Laugher

    Hmm, I don't really like the idea of having it behave differently depending on what you specify. That would just be more confusing for the user. "-TEXT" will appear in all names of parametrized tests (with at least 2 parameters). There is often spaces in my test names too as the parametrization might contain a string with spaces.

    I think a clearly separate option is better than overloading -k with multiple behaviours. If no one beats me to it I will look at adding it as a plugin or patch.

    BTW I still don't really understand why searching for numbers like '33.4' or '25' would match all my tests?

  4. holger krekel

    hum, your "behaves differently depending on what you specify" criticism makes sense. On the other hand i am not sure it would be a practical problem. I am fine with a separate option and guess it should go into core.

    As to why 33.4 matches: internally the -k expression is evaluated as a python expression. 33.4 evalues to True so independently of test names it matches. Apart form parametrization people cannot have float numbers appear as part of the test name which explains why it hasn't come up yet. We could also think about making the option match parametrization specifically, maybe. Note also that we are considering long-term to have test ids not be generated based on the values of the parameters because it disallows parametrization like [value, value], i.e. repeating tests. However, more thoughts on this topic are warranted.

  5. holger krekel

    fix issue357 - special case "-k" expressions to allow for filtering with simple strings that are not valid python expressions. Examples: "-k 1.3" matches all tests parametrized with 1.3. "-k None" filters all tests that have "None" in their name and conversely "-k 'not None'". Previously these examples would raise syntax errors.

    Also add a note to the docs about what is allowed.

    → <<cset 4951204afadc>>

  6. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.