Virtual Environments Demystified

Lukas Waymann2018-02-13

Here’s a non-exhaustive list of programs that are all meant to help create or manage virtual environments in some way:

Hatch, VirtualEnvManager, autoenv, fades, inve, pew, pipenv, pyenv-virtualenv, pyenv-virtualenvwrapper, pyenv, pyvenv, rvirtualenv, tox, v, venv, vex, virtual-python, virtualenv-burrito, virtualenv-mv, virtualenv, virtualenvwrapper-win, virtualenvwrapper, workingenv

Clearly, this stuff must be really hard to get right. I also must be a moron, since, after having written some thousand lines of Python, I don’t even know what problem we are trying to solve here, and the abundance of relevant programs with subtly different names has deterred me from reading up on it so far.

So what is a virtual environment? The official docs’ tutorial describes it as

a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.

A directory with a Python interpreter? Easy enough.

$ mkdir virtual_env
$ cp /bin/python3 virtual_env/

Let’s see. Directory? Check. Contains a Python installation? Check. Contains a number of additional packages? Zero is a number! (Check.) Particular version? Um…

$ cd virtual_env/
$ ./python3 --version
Python 3.6.3

I think that will do. Is it self-contained, though? It doesn’t contain itself…

Jokes aside, there are only two things missing to actually make our directory a virtual environment as specified by PEP 405, the proposal that integrated a standard mechanism for virtual environments with Python.¹

A file named pyvenv.cfg containing the line home = /usr/bin
A lib/python3.6/site-packages subdirectory

(Both paths are subject to the OS and the second one also to the Python version used.)

$ echo 'home = /usr/bin' > pyvenv.cfg
$ mkdir -p lib/python3.6/site-packages

I will also move the Python binary into a bin subdirectory.²

$ mkdir bin && mv python3 bin/

Fair. We have a directory that formally qualifies as a virtual environment:

$ tree --noreport
.
├── bin
│   └── python3
├── lib
│   └── python3.6
│       └── site-packages
└── pyvenv.cfg

This leads us to the next question.

What’s the point?

When we run our copy of the Python binary, the pyvenv.cfg file changes what happens during startup: the presence of the home key tells Python the binary belongs to a virtual environment, the key’s value (/usr/bin) tells it where to find a complete Python installation that includes the standard library.

The bottom line is that ./lib/python3.6/site-packages becomes part of the module search path. The point is that we can now install packages to that location, in particular, specific versions that may conflict with the dependencies of another Python program on the same system.³

For example, if your project needs exactly version 0.0.3 of left-pad:

$ pip3 install -t lib/python3.6/site-packages/ left-pad==0.0.3

Now this will work:

$ ./bin/python3 -c 'import left_pad'

While this should raise ModuleNotFoundError, as desired:

$ python3 -c 'import left_pad'

Another project on the same system could have a different version of left-pad in its own virtual environment, without interfering with this one.

The standard tool for creating virtual environments

In practice, one does not simply create virtual environments by hand, which brings us back to the dauntingly long list of tools above. Fortunately, one of them is not like the others, because it ships with Python as part of the standard library: venv.⁴

In its simplest form, venv is used to create a virtual environment like so:

$ python3 -m venv virtual_env

This creates the virtual_env directory and also copies or symlinks the Python interpreter:

$ cd virtual_env
$ find -name python3
./bin/python3

It also copies a bunch of other stuff: I get 650 files in 89 subdirectories amounting to about 10 MiB in total. One of those files is the pip binary, and we can use it to install packages into the virtual environment without passing extra command-line arguments:

$ ./bin/pip install left-pad

You can read more about using venv and optional magic like “activate” scripts in the Python tutorial or venv’s documentation—this post is only meant to boil down what a virtual environment actually is.

Summary

A virtual environment is a directory containing a Python interpreter, a special pyvenv.cfg file that affects startup of the interpreter, and some third-party Python packages. Python packages installed into a virtual environment will not interfere with other Python applications on the same system. The “standard tool for creating virtual environments” is venv.

Appendix: timeline

I think Ian Bicking’s non_root_python.py qualifies as the first tool for creating virtual environments. Based on that, virtual-python.py was added to EasyInstall in version 0.6a6 in October 2005. Here’s a timeline summarizing some main events.

2005-10-17: virtual-python.py is added to EasyInstall.
2006-03-08: Ian Bicking publishes a blog post about improving virtual-python.py titled “Working Environment Brainstorm”.
2006-03-15: Ian Bicking announces working-env.py.
2006-04-26: Ian Bicking announces an improved version of working-env.py called workingenv.
2007-09-14: virtualenv’s first commit
2007-10-10: Ian Bicking announces virtualenv: “Workingenv is dead, long live Virtualenv!”
2009-10-24: virtual-python.py is removed from EasyInstall.
2011-06-13: PEP 405 is created.
2012-05-25: PEP 405 is accepted for inclusion in Python 3.3.
2012-09-29: Python 3.3 is released and venv and pyvenv become part of the standard library.
2014-03-16: Python 3.4 is released and venv “defaults to installing pip into all created virtual environments” now.
2015-09-13: Python 3.5 is released. “The use of venv is now recommended for creating virtual environments.”
2016-12-23: Python 3.6 is released; “pyvenv was the recommended tool for creating virtual environments for Python 3.3 and 3.4, and is deprecated in Python 3.6.”

Notes

The “Virtual Boy” image is used with permission from James Rolfe.
If you found this article helpful or otherwise worthwhile and want to say thanks, one way you can do so is by buying me a coffee.

Before PEP 405 was accepted, virtual environments were purely the domain of third-party tools with no direct support from the language itself. ↩
I think this should not be necessary. But, because of what I assume to be a bug in CPython, it is. A bin/ subdirectory certainly is the conventional location for the binary, though. ↩
Be aware that we only get Python-level isolation. ↩
pyvenv also ships with Python, but was deprecated in version 3.6. Both venv and pyvenv were added to Python in version 3.3. ↩