Python Virtual Environments
A python virtual environment is a cooperatively isolated runtime environment that allows users and to install and upgrade Python distribution packages without interfering with the behaviour of other Python applications running on the same system.
They essentially allow you to create a “virtual” isolated Python installation and install packages into that virtual installation. When you switch projects, you can simply create a new virtual environment and not have to worry about breaking the packages installed in the other environments. It is always recommended to use a virtual environment while developing Python applications.
Python includes a module specifically created for creating lightweight virtual environments called venv (virtualenv in python2 versions).
You can create Python virtual environments with venv by generally following these steps:
Note that this process assumes you are working in the directory /home/[username] if you are working in a directory other than your home directory please replace that directory path where /home/[username] is referrenced in the steps below
1) Load a Python module. If your workflow requires a specific version of Python, it is important that you load this specific version at this time. There are many differences between Python2 and Python3, so be sure which version your workflow requires. If you are unsure, Python3 is the currently supported and updated version, with the latest version available on the cluster being Python/3.8.3.
To load a python module, use the command module add python/[version]
For example, module add python/3.8.3
2) Create the environment. To create the python virtual environment with a Python2 version, you will use the command python -m virtualenv [env_name]
To create the python virtual environment with a Python3 version, you will use the command python3 -m venv [env_name]
For example, to create a virtual environment with Python3 called “test_env”, the command would be:
python3 -m venv test_env
3) Activate the environment. Before you can start installing or using packages in your virtual environment you’ll need to activate it. Activating a virtual environment will put the virtual environment-specific python and pip executables into your shell’s PATH.
To activate the environment, use the command:
source /home/[username]/[env_name]/bin/activate
4) Confirming you’re in the virtual environment. You can confirm you’re in the virtual environment by checking the location of your Python interpreter, it should point to the env directory.
Use the following command to check which Python interpreter your shell is currently using with the command:
which python
Which should give the following output:
/home/[username]/[env_name]/bin/python
5) Installing packages. To install packages you can use the pip command, similar to how you would install packages on the cluster without using a python environment. An example of a pip command to install the ‘numpy’ module is:
pip install numpy
Inside your python virtual environment you do not need to include the –user flag with the pip command like you do when installing pip modules on the cluster outside of a virtual environment. This is because the Python package installed with pip will only be installed in this virtual environment and will not be able to be used from outside of the virtual environment, such as modules that are installed outside of virtual environments with the –user command are.
6) Deactivating the environment. To leave the virtual environment, you can use the command deactivate while inside the virtual environment. This will remove you from the virtual environment but will not delete it or the files inside of it. To re-enter the virtual environment, simply use the ‘source’ command as you did in step #3 of these instructions and you will re-enter the virtual environment where you last left it, and the files and packages contained inside it will be there.
For more information on installing packages inside a virtual environment, please see the resource here.
If you have any questions or problems using python virtual environments, please email orcd-help-engaging@mit.edu