Support conda envs when using Miniconda

Issue #273 on hold
Michael Nazario created an issue

Miniconda provides conda envs to create isolated Python environments similar to virtualenv. Its main advantage is that conda environments also isolate the binary shared library dependencies from the rest of your machine. Unfortunately, conda does not support virtualenvs because of the difference in how they manage shared libraries.

Since Miniconda isolates its shared library dependencies from the rest of your machine, virtualenvs which are created by a Miniconda-based environment fail during creation time. The virtualenvs will attempt to use shared libraries in the system-level location, but not the Miniconda location.

NOTE: If you try to create virtualenvs using Miniconda in an environment which already has the correct shared libraries in the system-level location, then your virtualenv will seem to succeed. However, any shared libraries which were only installed using conda will fail at runtime. Because of this, testing this feature should be done in an isolated environment.

I believe that the first implementation of this feature has these requirements:

  • Create conda environments instead of virtualenvs based on a single miniconda Python version
  • Install pip using conda by default
  • Add dependencies using pip (like virtualenvs)

Further features which are necessary for complete conda env supports are:

  • Create conda environments for multiple miniconda Python versions
  • Allow the installation of conda packages along with pip packages

There is a similar project called ctox (https://github.com/hayd/ctox) which tries to implement this. However, it doesn't have any guarantees on reliability. This project was actually started around the time there were comments on this ticket (https://bitbucket.org/hpk42/tox/issues/167/add-an-option-to-run-commands-after) about conda.

I would like to work with the tox team on specifying any additions necessary for conda support in Tox.

Comments (11)

  1. Holger Krekel repo owner

    Sorry it took a while. Me and i think @flub would also like to think further on supporting non-virtualenv virtualization through hooks and a tox-conda or tox-miniconda plugin. If you'd like to help make this happen you are welcome to discuss, and also to join and/or contribute to https://www.indiegogo.com/projects/python-testing-sprint-mid-2016/x/4034848#/ -- i'd be happy if some more people, companies fund the sprint with a particular focus on tox which needs more activity and community.

  2. Michael Nazario reporter

    In December I made a prototype to get this to work with only Conda (and not virtualenv). I haven't quite gotten the chance to make it play nicely with all of the tests. I'll go ahead and get back to working on making it fit into Tox better.

  3. Oliver Bestwalter

    We started working out a concept to make things like this possible. It is also very likely that conda will be proof of concept to show the viabilty of this approach. See #338

  4. Miguel Sánchez de León Peque

    @obestwalter Please, let us know if you have something testable (I would be glad to try it out). 😊

  5. Oliver Bestwalter

    The draft mentioned in #338 is the outcome of us sitting together at the sprint and sketching out how this should look like from the user perspective and make sure it would be general enough to make it work with other scenarios (like no virtualenv or pyenv, etc.). If someone is interested in starting to work on the implementation this can act as the starting point. I am not using conda myself, so I am not a good candidate to implement the conda specific part.

  6. Kale Franz

    I'm the lead for the conda project, and I just want to chime in to say I'm extremely supportive of conda and tox working together. Conda envs and python virtualenvs don't currently "play nice" together, but there's no technical barrier to why they couldn't/shouldn't. If they did, tox would work just fine for python projects within a conda env. However, making a conda plugin for tox is a more powerful concept: it would allow tox to create environment matrices for any language ecosystem.

    I don't currently have the bandwidth to do this work myself, which is why in other places I've asked for community help. However, if there are flags or features that conda is missing to make everything work, I'm eager to help make it happen.

    Related:

  7. Bruno Oliveira

    Hey @kalefranz,

    Thanks for chiming in. I was also involved in the discussion which lead to the extend-envs-and-packagebuilds.md document. That draft was the result of brainstorming and looking at the current implementation to figure out what was feasible/doable, so it should be a good starting point if someone wants to champion it; both the section about how each plugin can handle different packages as well as the hooks needed for the plugin system to be generic were written with the thought of supporting both virtualenv and conda as plugins (with virtualenv being a core plugin).

    Unfortunately I'm also short on time to work on this right now although I would love to.

  8. Log in to comment