Issues

Issue #38 resolved

virtualenvwrapper.sh is noticeably slower than virtualenvwrapper_bashrc

Anonymous created an issue

Upgraded virtualenvwrapper today and updated my .bash_profile. Wow, things are a LOT slower when starting a shell now! I did some investigating and I think that hook running may be to blame. Please see this screen capture:

http://www.brentwoodruff.com/files/virtualenvwrapper.mp4

Comments (36)

  1. Doug Hellmann repo owner
    • changed status to open

    You definitely win for Most Awesome Way to Submit a Bug Report. ;-)

    I have the wrappers loaded via my .bashrc instead of .bash_profile. I'm not sure if that would make any difference or not. Theoretically if you load the hooks one time when you login to the Mac you don't need to reload them in each shell (the parent environment should be inherited).

    I ran a few timings on my system:

    host28:dhellmann:~:507 $ time bash -c /usr/bin/true
    
    real    0m0.025s
    user    0m0.002s
    sys     0m0.005s
    host28:dhellmann:~:508 $ time bash --login -c /usr/bin/true
    
    real    0m0.877s
    user    0m0.531s
    sys     0m0.322s
    

    Those times are with virtualenvwrapper, virtualenvwrapper.project, and virtualenvwrapper.bitbucket installed. What hooks do you have set up? What are they doing?

    Could you provide similar output for yours so we can see if it's something to do with your config that is exposing a performance issue?

  2. fprimex

    Went ahead and set up an account so I can keep up with tickets. I've not set up any hooks for virtualenvwrapper, so I'm not sure what I should move around or to where. It seems as though new Terminal windows and tabs insist on reading .bash_profile. At any rate, I've always used .bash_profile and login shells. Here's some timing information.

    Baseline bash - no login shell:

    $time bash -c /usr/bin/true
    
    real	0m0.007s
    user	0m0.002s
    sys	0m0.005s
    

    No virtualenvwrapper in .bash_profile:

    $time bash --login -c /usr/bin/true
    
    real	0m0.015s
    user	0m0.006s
    sys	0m0.009s
    

    Using virtualenv_bashrc in .bash_profile:

    $time bash --login -c /usr/bin/true
    
    real	0m0.057s
    user	0m0.027s
    sys	0m0.028s
    

    Using virtualenvwrapper.sh in .bash_profile as shown in the video:

    $time bash --login -c /usr/bin/true
    
    real	0m2.152s
    user	0m1.568s
    sys	0m0.498s
    

    Same as above, with initialize, pre and post_activate hooks commented out:

    $time bash --login -c /usr/bin/true
    
    real	0m0.319s
    user	0m0.226s
    sys	0m0.086s
    
  3. pjv

    i noticed the same long lag and based on doug's note above, went on a quest to find a place to load the wrappers into my environment on a level where any terminal window or tab (always a new login shell) would inherit it. i couldn't figure out how to do it. i have a feeling that modifying the system-wide bashrc (/etc/bashrc) would probably do it, but i don't want to mess with os x guts.

    i think you could also make something work with "screen" virtual terminals, but i am already addicted to tabs in terminal.app.

    if anyone knows a way to source virtualenvwrapper.sh so that subsequently opened terminal windows/tabs would inherit, please spill.

  4. David Wolever

    I've found the same thing. With pre-2.0, everything was nice and fast… But with post-2.0, almost every operation is really, really slow:

    ~ time workon myenv 
    
    real	0m1.025s
    user	0m0.664s
    sys	0m0.349s
    (myenv)~/myenv
    

    Looking through the code, I suspect it's because Python is being started a whole truckload of times (I count 4 or 5 calls each time virtualenvwrapper.sh is executed).

    In my local version, I've simply commented out the calls (the first `WORKON_HOME=$(...)` and the last call to `virtualenvwrapper_initialize`), which at least makes startup fast… But I haven't figured out how to speed up workon yet.

    I'll see if I can speed things up tonight without sacrificing too much functionality...

  5. David Wolever

    Doug Fritz: sorry, I'm not sure I understand how that will help… Won't that mean virtualenv wrapper is only started if a VIRTUALENVWRAPPER_PYTHON is defined? And how would that help make it any faster?

  6. Doug Hellmann repo owner

    David, no the "-z" means "if zero length" so it will avoid re-loading virtualenvwrapper.sh if it is already loaded.

    You should be able to load virtualenvwrapper.sh one time in your root login shell. It shouldn't need to be loaded for each new shell you start.

  7. David Wolever

    D'oh! Sorry, I'm an idiot.

    However, I'm not sure it's true that you only need to load it once… The exported shell variables will persist, but to the best of my knowledge, the functions won't (at least, not when I test it).

    Moreover, if one logs in via, eg, SSH, they will be starting a login shell fairly often (I do it a bunch of times a day), so that won't be a solution for them.

    Anyway, I've started going through the code, seeing where I can make things faster - would you be free for a quick chat on IRC? Or would you rather the discussion stayed here? I'm wolever on freenode, or David Wolever.

  8. Doug Hellmann repo owner

    I'm surprised the shell functions don't persist. I'll have to test that to see what I can do.

    I don't have time to chat tonight, but I'm interested in what you find.

    FWIW, I don't see the same performance problems on my system so there must be something else at play here in addition to the initialization code.

  9. Doug Hellmann repo owner

    Use of the tempfile program isn't portable across all of the supported systems, that's why I moved to Python's tempfile module.

    You probably want to redirect the grep output in the tests to /dev/null to avoid printing the matches to stdout, right?

    I like the idea of doing the run and source operations at the same time, although maybe we can come up with a shorter option name? :-) That should probably just be the default behavior, so maybe the --source option can be replaced with something like "--script $filename".

  10. David Wolever
    • Use of the tempfile program isn't portable across all of the supported systems, that's why I moved to Python's tempfile module.
      Dang, I thought tempfile was pretty safe, but I guess I was wrong. Out of curiosity, which platforms don't support it? Anyway, do you think it would be acceptable to use a shell function (eg, "figure out a unique name, touch, chmod 600") in leu of a call to Python?
    • You probably want to redirect the grep output in the tests to /dev/null to avoid printing the matches to stdout, right?
      Right. In that case, there is at least one other, similar, test which should probably be fixed too...
    • I like the idea of doing the run and source operations at the same time, although maybe we can come up with a shorter option name? :-) That should probably just be the default behavior, so maybe the --source option can be replaced with something like "--script $filename".
      Sure, that sounds good. I wasn't sure how this was being used, hence verbosity.
  11. Doug Hellmann repo owner

    There's no tempfile on OS X Leopard, which is what I'm running on my laptop.

    The only place the hook loader is used is through that shell function. That's the way it *should* be invoked, so we can give the option any reasonably meaningful name.

  12. David Wolever
    • There's no tempfile on OS X Leopard, which is what I'm running on my laptop.

    Dang, so there isn't. I run 10.5 too, but I hadn't realized that my tempfile comes from fink. I'll update the code to fix that.

    • The only place the hook loader is used is through that shell function. That's the way it *should* be invoked, so we can give the option any reasonably meaningful name.

    Awesome. Will update too.

  13. David Wolever

    Also, if backwards compatibility isn't important, this line could be removed entirely:

        WORKON_HOME=$("$VIRTUALENVWRAPPER_PYTHON" -c "import os; print os.path.expandvars(os.path.expanduser(\"$WORKON_HOME\"))")
    
  14. Anonymous

    Just a small patch: first, it turns out that `mktemp` doesn't default to putting the temp files in, you know, the temp directory… So it fixes that (the -t flag). Also, I've found that I had a lot of old tempfiles lying around (eg, because I killed the script in the middle), so I've added a `trap` that will rm the tempfiles:

    diff --git a/virtualenvwrapper.sh b/virtualenvwrapper.sh
    --- a/virtualenvwrapper.sh
    +++ b/virtualenvwrapper.sh
    @@ -92,7 +92,9 @@
     # Expects 1 argument, the suffix for the new file.
     virtualenvwrapper_tempfile () {
         # Note: the 'X's must come last
    -    mktemp "virtualenvwrapper-$1-XXXXXX"
    +    file="`mktemp -t "virtualenvwrapper-$1-XXXXXX"`"
    +    trap "rm '$file' &>/dev/null" EXIT
    +    echo $file
     }
     
     # Run the hooks
    
  15. Anonymous

    Could you please add a "-f" to the rm in the trap line? trap "rm -f '$file' &>/dev/null" EXIT

  16. Dave Abrahams

    Not sure this should really be considered resolved. It takes like 9 seconds on a very fast machine for me to open a new terminal window if I source this in my .zshenv

    I take it back, sorry. My numbers are fluctuating wildly.

  17. Anonymous

    David Salter: If you run `time source virtualenvwrapper.sh` (or the zsh equivalent) a couple of times, what is the result? Also, can you run `time python -c "pass"` a couple of times too?

  18. Anonymous

    I removed the source out of my .bash_profile and ran the above time tests after restarting my machine.

    vbabiy@Vitaly-Babiys-MacBook-Pro ~ ❯❯ time source virtualenvwrapper.sh
    
    real	0m2.459s
    user	0m0.184s
    sys	0m0.152s
    vbabiy@Vitaly-Babiys-MacBook-Pro ~ ❯❯ time source virtualenvwrapper.sh
    
    real	0m0.143s
    user	0m0.107s
    sys	0m0.036s
    vbabiy@Vitaly-Babiys-MacBook-Pro ~ ❯❯ time source virtualenvwrapper.sh
    
    real	0m0.144s
    user	0m0.109s
    sys	0m0.034s
    

    Testing Python:

    vbabiy@Vitaly-Babiys-MacBook-Pro ~ ❯❯ time python -c "pass"
    
    real	0m0.054s
    user	0m0.010s
    sys	0m0.008s
    vbabiy@Vitaly-Babiys-MacBook-Pro ~ ❯❯ time python -c "pass"
    
    real	0m0.022s
    user	0m0.012s
    sys	0m0.009s
    vbabiy@Vitaly-Babiys-MacBook-Pro ~ ❯❯ time python -c "pass"
    
    real	0m0.021s
    user	0m0.011s
    sys	0m0.009s
    
  19. Doug Hellmann repo owner

    I'm sure the source of the slowdown is the fact that there are now initialization hooks run each time virtualenvwrapper.sh is sourced so that plugins can add their own functions. This is handled automatically, instead of requiring the user to add a separate "source" statement for every plugin they install.

  20. Log in to comment