Issue #28 resolved

environment templates

Doug Hellmann avatarDoug Hellmann created an issue

Brent O'Conner's project django-environment (http://github.com/epicserve/django-environment) lead me to think about creating a template system for defining new environments. This probably reproduces a lot of work by tools such as buildout recipes, so we'll want to do some research before developing anything.

Comments (18)

  1. Doug Hellmann

    Early implementation thoughts:

    Using a pip requirements file would be an easy way to trigger installation of specific apps. An option to mkvirtualenv could be used to specify the requirements file.

    Installing the virtualenvwrapper hooks could be a little more challenging. If the requirements file and hooks were all in the same directory, much like Brent has in django_env/bin, then rather than specifying the requirements file the option to mkvirtualenv could just point to that directory.

    So the user would do something like:

    $ mkvirtualenv -t django my_new_django_project

    After creating and activiating the new virtualenv, mkvirtualenv would look for a "django" template somewhere. It would install the packages listed in the requirements.txt file within the template, then copy any hook scripts from the template into the environment (or simply add calls to them at the end of the default hooks, as you do with your install script).

    One trick would be finding the templates. It would be nice to be able to install new templates with pip, of course. We could use setuptools entry points registry to define "virtualenvwrapper.template" plugins and pkg_resources to locate/retrieve their contents. A single command line tool could do that work:

    $ virtualenvwrapper django postmkvirtualenv

    would print the location of the postmkvirtualenv hook from the "django" template so mkvirtualenv could run it, for example.

    Similarly:

    $ virtualenvwrapper django requirements.txt

    would dump the contents of the requirements file. mkvirtualenv would probably use a temporary file to hold them, then call pip to install everything.

  2. Brent O'Connor

    I like your idea of being able to specify a command line argument for different framework templates or just custom templates in general.

    We probably need to define a plugin api for templates in virtualenvwrapper. We would need to think of a standard location for plugin files and how that would work with the pip installer and easy_install. It would be nice if we could figure out a way to run "pip install django-enviroment" and then have the installer install virtualenv and virtualenvwrapper if they aren't installed already.

    Another thing I would like to think about and add would be a virtualenvwrapper command plugin that allows you to have a base command like pyenv, etc. that you could run and would have tab completion for all the commands that are available. I made a command shortcut called runserver to easily start the django development server and it would be nice to tie that into a standard wrapper command so you could run "pyenv runserver" for example.

    And on a side note, working on this project has made me hate writing bash scripting code! I'm thinking about porting all my code over to python. I think you can set environment variables for the bash shell from within python.

  3. Matthew Wilson

    This is vaguely related to the postmkvirtualenv hook I started writing today:

    # If $HOME/checkouts/$envname doesn't exist, then 
    # mkdir $HOME/checkouts/$envname
    
    # If $WORKON_HOME/postactivate exists, copy it into $WORKON_HOME/$envname
    # cp $WORKON_HOME/postactivate $WORKON_HOME/$envname
    
    # If $WORKON_HOME/.screenrc-envname exists, 
    # cp $WORKON_HOME/.screenrc-envname $HOME/.screenrc-$envname
    

    It would be nice (perhaps for me only though) if this kind of stuff was built in. I want to copy some files into my new $WORKON_HOME/$envname, and I want to copy some other files into $HOME.

  4. Doug Hellmann

    Replying to comment 2:

    We would need to think of a stand location for plugin files and how that would work with the pip installer and

    Using setuptools/distribute entry points in the way I describe eliminates the need for any pre-defined file locations. There would probably be a class defined that template implementors could use to specify the file locations, with the default class using the obvious filenames. But the files themselves would be tucked away inside the template package somewhere. I need to put together a sample implementation to really illustrate what I'm talking about.

    easy_install. It would be nice if we could figure out a way to run "pip install django-enviroment" and then have the installer install virtualenv and virtualenvwrapper if they aren't installed already.

    pip honors the prereqs defined in a setup.py file, so getting virtualenv and virtualenvwrapper that way would be easy.

    Another thing I would like to think about and add would be a virtualenvwrapper command plugin that allows you to have a base command like pyenv, etc. that you could run and would have tab completion for all the commands that are available. I made a command shortcut called runserver to easily start the django development server and it would be nice to tie that into a standard wrapper command so you could run "pyenv runserver" for example.

    That's an interesting idea. So it would do tab-completion based on the commands in $VIRTUAL_ENV/bin?

    And on a side note, working on this project has made me hate writing bash scripting code! I'm thinking about porting all code over to python. I think you can set environment variables for the bash shell from within python.

    You can, but not for the parent process, so most of the virtualenv wrapper work needs to be done in shell.

  5. Doug Hellmann

    In reply to mw44118 comment 3:

    I could see a system like this being used to define plugins for all of the hooks, too. In addition to running any local hooks defined in the current way, the plugins would be scanned for anything providing "global" hooks. The behavior you describe could be packaged as plugins so the user would only need to "pip install mattshooks".

    If entry points are used for the plugins, then the set of plugins visible will depend on which python interpreter is used for the search (the global interpreter or the one in the virtualenv). Each approach has its benefits and drawbacks. Installing globally makes them easy to find, but requires global permissions and the wrapper has to use the correct python interpreter to see them. Installing them into the virtualenv requires separate copies and an installation delay but makes finding them easy. Thoughts?

  6. Jorge Vargas

    Ups sorry #10 was me. I forgot to login.

    As for Paste I actually started some code for this. I haven't made it public because I haven't found the time to manage the influx of code regarding it.

    It is basically a very simple implementation of the paster create command with only one dependency (Tempita) I originally planned to use that for making "quickstart" packages for distutils, distribute, etc.

  7. Doug Hellmann
    • changed status to open

    I experimented with setuptools entry points this morning, and I think I've come up with a way to make it easy to add plugins to virtualenvwrapper. Those plugins could then add any desired behavior, such as templates for new environments. I'm in the process of re-implementing the existing user script hooks using a plugin as an example.

  8. Jorge Vargas

    The only pb I have with entrypoints is that they depend on distribute/setuptools which means we may have a bootstrap problem, although most people familar with venv already have something like that going so it ain't that bad.

  9. Doug Hellmann

    I'm definitely working with distribute, not the older setuptools. Adding the dependency isn't optimal, but I think most virtualenv users are going to have pip anyway, and it will bring in distribute or setuptools.

  10. Doug Hellmann

    I pushed some changes this morning that make it possible to create plugins for virtualenvwrapper, so everything doesn't have to be tied to the user's customization scripts. I've tried to make everything backwards-compatible, but want more local testing time before I cut an official release. If anyone else wants to give it a try, grab the source and let me know if you run into problems.

  11. Doug Hellmann

    The official release of 2.0 with the new plugin system went out last night. Under the current system, the best way to implement this feature is probably through a new command, since there is no mechanism for adding command line options to the mkvirtualenv command directly. I'm still open to suggestions for adding option processing, though.

  12. Doug Hellmann

    I've been considering how to implement this, and have come up with 3 options. All involve new hooks to enable the template feature, in one form or another.

    1. Go ahead and add command line argument processing to mkvirtualenv. For example, the user would run something like "mkvirtualenv -t templatename ... venvname".

    2. Create a new "mkproject" command. It would process arguments to identify the template, then pass the rest of the args to mkvirtualenv. The core command could call new hooks (pre_mkproject, post_mkproject) implemented by each template plugin.

    3. Let each template plugin provide its own new command ("mkdjangoproject", "mkturbogearsproject", etc.). This avoids the need for option processing in the existing commands and allows each template to define its own arguments.

    For option 1 or 2, the question of a "default" template comes up. The default should probably be no template, but it also seems like it would be useful to let the user specify some other default via an environment variable.

    There is also the question of who creates the project directory. Under option 1 and 3, the template would be responsible for creating a it, but we would probably standardize on a variable name to tell where the root directory for those projects should go. With option 2 the core mkproject command could create the directory before invoking the template plugin. With option 3 we could also have the new commands run a mkproject command that provides some core project management features without any options.

    With a core command like mkproject, the user could also specify multiple templates which could be applied in order. For example "mkproject -t hg -t django my_project" could initialize a mercurial repository, install Django, then run "django-admin.py startproject" to populate the project directory.

    In all cases, only the template(s) specified by the user would be invoked, so template plugins would work differently from other plugins.

    Paster's template support looks useful, but I don't think I want to specify that level of implementation detail for virtualenvwrapper plugins. It would make sense to use paster to *implement* project templates, but there may be other tools better suited for different situations.

    I'm leaning towards option 2 or 3 at this point, mostly because I don't want to overload mkvirtualenv with too much. I like the idea of chaining multiple templates together, and that works better with a central command for creating projects (option 2).

    Does anyone else have an opinion?

  13. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.