Allow `setup.cfg` as an alternative configuration file

Issue #297 on hold
Peter Bittner created an issue

(This issue comes from a comment in issue #185.) I'd love to see the tox configuration living in setup.cfg too in general, as an alternative.

It would allow a more elegant Python package configuration for developers: You could put configuration of all your tools into a single file. The ini-style allows for separate sections. I've described some considerations in a comment on issue 13.

A few projects already look into setup.cfg. flake8, behave, py.test, among others. Pylint is a prominent exception (its maintainer considers this idea harmful; declined PR; discussion).

Comments (23)

  1. Florian Bruhin

    I personally think this is a bad idea as well, especially for tools which typically require multiple sections.

    setup.cfg was originally intended as a configuration file for distutils/setuptools - why dump all the other unrelated stuff in there? What's wrong with having different files as separation for different tools? WIth the same rationale, you could argue it's better to have all your code in a single file, even if it's completely unrelated.

  2. Peter Bittner reporter

    Doing this is a bit tricky, though. It requires giving up some elegance in config.py where tox.ini is defined as a default for the -c command line argument. Respecting "several defaults" requires some spaghetti code later in the process. Probably on line 218. Not very elegant. 😧

  3. Peter Bittner reporter

    @The-Compiler The "all-code-in-one-file" allegory is not correct.

    Configuration for testing in a single location, especially when several different tools are used on a single project, has a distict advantage: You can compare the settings of the different tools in use. You have all your test-related settings in a single place.

    I don't think this file will ever have more than 20 lines (i.e. half a screen page), even with a couple of tools using.

  4. Florian Bruhin

    Something about our configurations is clearly different then ;)

       18 .coveragerc
       65 .pylintrc
       40 pytest.ini
      243 tox.ini
      366 total
    
  5. Peter Bittner reporter

    Can you share what you have in a 243 lines tox.ini file? And 40 lines of pytest.ini (which you can put in tox.ini, IIRC). Pylint configuration is borked, agreed, everybody knows that; that's why projects like Prospector pop up.

    Here is a working sample configuration for a bunch of test tools. 27 lines total; note that I have symlinks for .coveragerc, pylintrc, tox.ini pointing to setup.cfg. What features do I miss out that require more configuration?

    Please keep in mind that I'm suggesting an alternative to the current approach, not a replacement.

  6. Peter Bittner reporter

    Maybe I should give some more background information, so people can understand my motivation (and the one of those that share my opinion):

    When you look at a typical Web development project you have tons of configuration files, the majority of them from frontend technologies (in agency projects I have participated with). The project repository root is polluted by files, which is just "noise" that makes your project less maintainable:

    bower.json
    config.rb
    .coveragerc
    .csscomb.json
    Dockerfile | Vagrantfile
    .eslintrc
    Gemfile
    Gemfile.lock
    gulpfile.js
    lint.yml
    package.json
    pylintrc
    scss.mustache
    setup.cfg
    spritesmith-retina-mixins.template.mustache
    tox.ini
    

    I believe we "backend people" should know better. In projects that I do at least .coveragerc, pylintrc and tox.ini are only symlinks to setup.cfg. Some tools (behave BDD testing, flake8 and pytest) read their configuration from setup.cfg already.

    Yes, setup.cfg was intended as a configuration file for distutils/setuptools, and it still is. And the good news is tox also integrates nicely into setuptools.

    This way you can run your tests with python setup.py test, the standard approach. And tests run the way you want your project to be tested (e.g. using tox). No need to train new developers, no need to write extensive test instructions. It's just obvious, because the project follows an established standard approach.

    So, what if in a typical project you have only a single file that you look into to find out what the test setup is like? That could be appealing to a couple of folks out there, myself included. For excessive use, like in Florian's qutebrowser project, you can still have this split up, which makes perfectly sense when you have such a huge amount of configuration data.

  7. Holger Krekel repo owner

    not sure i follow. Isn't tox currently looking into setup.cfg already? I think you can define [tox] and [testenv] sections just fine there but maybe i misremember.

  8. Holger Krekel repo owner

    ah. in any case i agree it's a good idea to have a single file configuring packaging and testing. Ronny why do you think it's a bad idea to have tox config in setup.cfg? I guess we should at some point make it more official, probably by starting to discuss on distutils-sig.

  9. Ronny Pfannschmidt

    i think its a considerable dirty and problematic solution to just drop random things into the namespace of a config file of a different tool,

    i already find i fairly problematic that pytest/flake8 configuration inside tox.init has no prefix for the configuration key

    a much more preferable solution would be a standardized folder that tools can choose adapt or a per tool convention on adding other tools to the configfile

  10. Florian Bruhin

    Some other thoughts:

    Yesterday I helped @warsaw debug an issue where a pytest test failed when packaging for Debian. If I understood the issue correctly, Debian doesn't use tox to run the tests when packaging, so they didn't have the tox.ini around, which made the tests fail because of a missing option. Principle of least surprise, IMHO - you wouldn't search pytest configuration in tox.ini (if you don't know), just like you wouldn't search tox configuration in setup.cfg.

    Also from my experience contributing to, say, about 20 python projects (quite possibly more), I see setup.py test used very rarely, and tox used extensively. If I see a tox.ini, I immediately know "hey, I can run tox to test stuff without having to see how things work for this project". If the tox configuration is hidden in setup.cfg, again, I'll have to search for it in places I wouldn't expect it to be.

  11. Barry Warsaw

    @The-Compiler A couple of minor details in a moment, but I think you've nailed the essential bits. I do the same thing - when I see a tox.ini in an upstream project, there's no guessing as to how to run the test suite, and that is a huge win for Python in general. I've long pined for a consistent interface to test discovery, and there's more to be done with the newer metadata formats (i.e. it would be nice to have a declarative way to know), but as tox becomes more prevalent, it's becoming less mysterious how to run the tests. I used to thing setup.py test would be that interface, but for lots of reasons, it won't and probably shouldn't be. Can you tell I'm a huge fan of tox? :)

    A few details: our source package git branch does include the full upstream source, and during package build we also have the full source available, albeit with any Debian patches already applied (though with pytest 2.8.5 we have only one such patch left). However, we can't use tox for technical reasons in our build tools, something I plan on working to fix at some point. (Our recommended Python build helper supports tox, but it needs to plumb through additional argument passing, e.g. --sitepackages). So for now, what I generally do is pluck out the [testenv]commands bit and put that in our build rules file. That's what I did in the pytest case. I just missed that the [pytest] section of the tox.ini was relevant. You're spot on for that point.

    Unrelated to the issue we discussed (and for you help... thanks!) was that we cannot run the tests from the top-level source tree. I was doing that originally and got three mysterious failures in test_genscripts. I was about to give up on those tests (as I understand it, genscripts is deprecated anyway), but someone else in Debian helped debug this issue and now we cd to the temp directory when running the tests and they now pass.

    One last thing, I've noticed that the pytest suite doesn't clean up all its tempdirs. Normally, our builds run in overlays so can't mess with the host filesystem, but this isn't the case for /tmp. For this reason I mktemp a separate tempdir and export $TMPDIR to run the test suite. It still leaves cruft, but it's easier to clean up after the fact.

    Thanks for all your help! We now have 2.8.5 in Debian.

  12. Holger Krekel repo owner

    I understand that seing a "tox.ini" and "pytest.ini" signals useful information. On the other hand i'd like to see python move to a single config file for packaging and testing because i dislike the proliferation of config files. If we today say we want to have separate files we'll have setup.cfg, setup.py, pytest.ini, tox.ini to just distribute a single-module and a single-test module. Isn't this a bit much overhead/cruft? Somehow having a, say, setup.cfg which incorporates all the information seems leaner. Given some practise it means you will check for a "tox" section in that file instead of looking for the filename. Or just issuing "tox -l" and see. This can be automated. And to use setup.cfg is just a practical decision because we can't change the usage of this file easily and i always prefer incremental solutions instead of "everyone has to change".

  13. Ronny Pfannschmidt

    this will drag on a massive legacy, and now that pypa.json will come there will be another source of truth and setup.cfg might become obsolete

  14. Peter Bittner reporter

    Under the light of "move to a single config file for packaging and testing" (Holger): Allowing setup.cfg to house tox configuration should make sense, and should not be seen as "personal preference" (and would match existing py.test behavior as additional benefit).

    We must respect other arguments in this discussion, but as long as they are personal preference and clash with future benefits of python we should be able to agree on a common ground. (I know this is difficult, because the whole discussion is about personal preference.)

  15. Florian Bruhin

    FWIW, in pytest abusing setup.cfg causes various kinds of pain (examples: 1311, 567), and there is some consensus to deprecate and drop setup.cfg support.

    This seems very much a technical reason and not just a personal preference to not go down that rabbit hole again.

  16. Ronny Pfannschmidt

    I'm fine with a single file, but setup.cfg very certainly is not that file due to legacy and breakage potential

  17. Peter Bittner reporter

    Would you guys prefer including all other test tools via testenv:'s in tox.ini instead? So, instead of a setup.cfg we would use a tox.ini as done in the behave-django project, for doctests, flake8, behave, etc.?

  18. Log in to comment