Working with Virtual Environments

For every non-standard package installed in a system Python, the gods kill a kitten

—me

Why Virtual Environments?

As a professional developer, your projects will often require packages that are not part of Python’s standard library. And as the number of projects you work on increases, so does the likelihood that you require different versions of the same library in different projects. Version conflicts are the stuff of which nightmares are made. Therefore, it is considered best practice to always isolate your projects using a virtual environment.

There are a number of different approaches to solving this problem. I’ll list some others later, but for this course we will focus on using the venv module module.

Installing Virtual Environments

As of Python 3.3, it is no longer required to install an additional package to create a virtual environment. The venv module ships with Python as a part of the standard library.

Some linux distributions (notably Ubuntu) have taken steps to un-bundle the module for some reason. If you are using linux, be aware that the following examples may not work at first. If they do not, then you will have to install an additional package using your system package manager. For Ubuntu 16.04, that package is python3-venv.

Using Virtual Environments

To create a new virtual environment, you can use the following command:

$ python3 -m venv demoenv

That command invokes your Python 3 executable (python3). It is used to run the venv module (-m venv). The module is asked to create a named virtual environment (demoenv).

The result of this command is to create a folder called demoenv in your current working directory:

$ ls .
demoenv

In general, the form of this command is as follows. <ENV> is the name of the environment you want to create. If you are already in a folder and want to create an environment there, use ./. The available options are well documented. You can use the -h option to get some help at the command line.

$ virtualenv [options] <ENV>

What Happened?

When you ran that command, a couple of things took place:

  • A new directory with your requested name was created.
  • A new Python executable was created in <ENV>/bin (<ENV>/Scripts on Windows).
  • The new Python was cloned from your Python 3.
  • The new Python was isolated from any libraries installed in the original Python.
  • Pip and setuptools were installed so you can install additional packages.

You can take a peek to see the resulting structure of the demoenv directory:

demoenv/
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── easy_install
│   ├── easy_install-3.6
│   ├── pip
│   ├── pip3
│   ├── pip3.6
│   ├── python -> python3
│   └── python3 -> /usr/bin/python3
├── include
├── lib
│   └── python3.6
├── lib64 -> lib
├── pyvenv.cfg
└── share
    └── python-wheels

Activation

The virtual environment you just created, demoenv contains an executable Python command. If you do a quick check to see which Python executable is found by your terminal, you’ll see that it is not the new one:

$ which python
/usr/bin/python

You can execute the new Python by explicitly pointing to it:

$ ./demoenv/bin/python -V
Python 3.6.1

That’s tedious and hard to remember. Instead, you can activate your virtualenv using the bash source command:

$ source demoenv/bin/activate
(demoenv)$ which python
/Users/cewing/demoenv/bin/python

There. That’s better. Now whenever you run the python command, the executable that will be used will be the new one in your demoenv.

Notice also that the your shell prompt has changed. It indicates which virtualenv is currently active. Little clues like that really help you to keep things straight when you’ve got a lot of projects going on. It’s nice the makers of virtualenv thought of it.

Installing Packages

With an active virtual environment, you also have pip and easy_install. These Python packaging tools allow you to install packages in your virtual environment. The installed packages will not be available in your wider Python installation. They are isolated and safe.

(demoenv)$ which pip
/Users/cewing/demoenv/bin/pip
(demoenv)$ which easy_install
/Users/cewing/demoenv/bin/easy_install

Let’s see this in action. We’ll install a package called docutils. It provides support for converting ReStructuredText documents into other formats. This document you are reading is built using tools from that package.

(demoenv)$ pip install docutils
Collecting docutils
  Using cached docutils-0.13.1-py3-none-any.whl
Installing collected packages: docutils
Successfully installed docutils-0.13.1

And now, when we fire up our Python interpreter, the docutils package is available to us:

(demoenv)$ python
Python 3.6.1 (v3.6.1:69c0db5050, Mar 21 2017, 01:21:04)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import docutils
>>> docutils.__path__
['/Users/Nick/demoenv/lib/python3.6/site-packages/docutils']
>>> exit()
(demoenv)$

There’s one other interesting side-effect of installing software with venv. The docutils package provides a number of executable scripts when it is installed: rst2html.py, rst2latex.py and so on. You can see them in the bin directory inside your demoenv. These scripts are set up to execute using the Python with which they were built. What this means is that running these scripts will use the Python executable in your virtualenv, even if that virtualenv is not active!

Deactivation

So you’ve got a virtual environment created. And you’ve activated it so that you can install packages and use them. Eventually you’ll need to move on to some other project. This likely means that you’ll need to stop working with this demoenv and switch to another. It’s a good idea to keep a separate virtual environment for every project you work on.

When a venv is active, you use the deactivate command to turn it off:

(demoenv)$ deactivate
$ which python
/usr/bin/python

Note that your shell prompt returns to normal. The executable Python found when you check which python is the system one again.

Cleaning Up

There is one more great advantage that venv confers on you as a developer. The ability to easily remove a batch of installed Python software from your system. Consider a situation where you installed a library that breaks your Python (it happens). If you are working in your system Python, you now have to figure out what that package installed, where, and go clean it out manually. With venv the process is as simple as removing the directory that was created when you started out.

Let’s do that with our demoenv:

$ rm -rf demoenv

And that’s it. The entire environment and all the packages you installed into it are now gone. There’s no traces left to pollute your world.

Other Options

The venv module is part of the Python standard library starting with Python 3.3. But what if you are forced to use Python 2? What if you are on a system where venv is un-bundled and hopelessly borked? There are other options available.

The virtualenv package is one such option. It works for Python 2.x (and for Python 3 as well). In fact, it is the predecessor of the venv module.

The virtualenv package can be supplemented by virtualenvwrapper. It provides a number of command-line tools that can really add power to your workflow with virtualenv.

Another option is the conda packaging system. Conda is part of the Anaconda Python distribution. It is widely used in the data science and scientific computing communities. We will not use it in this class. Do Not install it on your machine for the duration of class, as it has side effects that can break our working patterns.

Managing Multiple Pythons

Sometimes you find yourself needing to install many different versions of Python. The more versions you have, the more challenging it can be to keep them cleanly managed. The pyenv project provides tools that can help.

You can identify different Python installations available in your system. You can choose which of them to treat as your global Python. You can change global versions at will. You can set different versions to be active in different directories. And much more.

It’s well beyond the scope of this course to dive into this tool. It is mentioned here as a reference, in case you find it useful at some point in your future career.

A Note

It can be easy to confuse the virtual environments provided by the venv module with OS virtualization (like virtualbox or vagrant). Be aware that venv only isolates Python and Python packages. It is not responsible for isolating any system packages. Packages like openssl, apache, nginx and so on are never managed by venv.

Wrap-Up

In this lecture we’ve:

  • learned why we might want to use a virtual environment in Python
  • learned to use the venv module in Python 3 to create such an environment
  • learned how to activate and deactive a virtual environment
  • learned how to install packages in a virtual environment using pip
  • learned that virtual environments can be disposed of

That’s enough for now.