Working with Third-Party Packages

Length: 00:14:59

Lesson Summary:

The standard library is great, but the vast quantity of third-party packages in the Python ecosystem is also at our disposal. In this lesson, we'll learn how to install Python packages and separate our dependencies using a virtualenv.

Documentation For This Video

Using pip to Install Packages

As a language with strong open-source roots, Python has a very large repository of open-source packages that can be installed for our use. Thankfully, this repository is easy for us to use, and when we installed Python we were even given the tool to install packages. The simplest tool that we have is pip. Since we have more than one Python installation, we need to make sure that we're using the version of pip that corresponds to the version of Python that we would like to install the package for. With Python 3.7, we'll use pip3.7. Let's install our first package, the popular AWS client library boto3:

$ pip3.7 install boto3
...
Installing collected packages: docutils, jmespath, urllib3, six, python-dateutil, botocore, s3transfer, boto3
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python3.7/site-packages/docutils'
Consider using the `--user` option or check the permissions.
$

There's an error because we don't have permissions to install a package globally unless we use sudo. If we do use sudo, then any other user on the system that could use our Python 3.7 install would also have access to boto3. An alternative approach is to install the package into a directory for packages only for our user using the --user flag when installing. Let's install the package locally to our user:

$ pip3.7 install --user boto3
...
Installing collected packages: urllib3, jmespath, six, python-dateutil, docutils, botocore, s3transfer, boto3
Successfully installed boto3-1.9.93 botocore-1.12.93 docutils-0.14 jmespath-0.9.3 python-dateutil-2.8.0 s3transfer-0.2.0 six-1.12.0 urllib3-1.24.1
$

The boto3 package has some dependencies, so pip also installed those as part of the installation process.

Viewing Installed Packages

If we want to view the packages that are already installed we'll also use the pip for that using the pip freeze command:

$ pip3.7 freeze
boto3==1.9.93
botocore==1.12.93
certifi==2018.11.29
Click==7.0
docutils==0.14
Flask==1.0.2
itsdangerous==1.1.0
Jinja2==2.10
jmespath==0.9.3
MarkupSafe==1.1.0
pipenv==2018.11.26
python-dateutil==2.8.0
s3transfer==0.2.0
six==1.12.0
urllib3==1.24.1
virtualenv==16.2.0
virtualenv-clone==0.4.0
Werkzeug==0.14.1

Since we installed boto3 with the --user flag, we'll still see it in this list. But a different user would not. The freeze subcommand gives us the information in a format that puts into a file, and then that file could be used to install packages with the specific version. Here's what that would look like:

$ pip3.7 freeze > requirements.txt
$ pip3.7 install --user -r requirements.txt
Requirement already satisfied: boto3==1.9.93 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (1.9.93)
Requirement already satisfied: botocore==1.12.93 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.12.93)
Requirement already satisfied: certifi==2018.11.29 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (2018.11.29)
Requirement already satisfied: Click==7.0 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (7.0)
Requirement already satisfied: docutils==0.14 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (0.14)
Requirement already satisfied: Flask==1.0.2 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.0.2)
Requirement already satisfied: itsdangerous==1.1.0 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (1.1.0)
Requirement already satisfied: Jinja2==2.10 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (2.10)
Requirement already satisfied: jmespath==0.9.3 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (0.9.3)
Requirement already satisfied: MarkupSafe==1.1.0 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (1.1.0)
Requirement already satisfied: pipenv==2018.11.26 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (2018.11.26)
Requirement already satisfied: python-dateutil==2.8.0 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (2.8.0)
Requirement already satisfied: s3transfer==0.2.0 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (0.2.0)
Requirement already satisfied: six==1.12.0 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.12.0)
Requirement already satisfied: urllib3==1.24.1 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 15)) (1.24.1)
Requirement already satisfied: virtualenv==16.2.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (16.2.0)
Requirement already satisfied: virtualenv-clone==0.4.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (0.4.0)
Requirement already satisfied: Werkzeug==0.14.1 in ./.local/lib/python3.7/site-packages (from -r requirements.txt (line 18)) (0.14.1)
Requirement already satisfied: pip>=9.0.1 in /usr/local/lib/python3.7/site-packages (from pipenv==2018.11.26->-r requirements.txt (line 11)) (18.1)
Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.7/site-packages (from pipenv==2018.11.26->-r requirements.txt (line 11))
(40.6.2)

Creating a virtualenv

If you're working on multiple packages that have varying dependency requirements, you can run into issues if you're installing packages either globally or localized to a user. Python's solution to this is what's known as a "virtualenv" (for "virtual environment"). A virtualenv is a localized Python install with its own packages, and it can be activated/deactivated. The Python module for creating a virtualenv is called venv, and we can use it by loading the module and providing a path to where we would like to place the virtualenv. Let's create a virtualenv where we can install the package psycopg2:

$ mkdir ~/venvs
$ python3.7 -m venv ~/venvs/pg

Now we have a virtualenv, but we need to "activate" it by running a script that was created within the virtualenv's bin directory.

$ source ~/venvs/pg/bin/activate
(pg) $ python --version
Python 3.7.2

The (pg) at the front of our prompt is to indicate to us which virtualenv we currently have active. While this virtualenv is active, the only python in our path is the Python 3.7 that we used to generate it, and pip will install packages for that Python (so we don't need to use pip3.7). Let's install the psycopg2 package:

(pg) $ pip install psycopg2
Collecting psycopg2
  Downloading https://files.pythonhosted.org/packages/0c/ba/e521b9dfae78dc88d3e88be99c8d6f8737a69b65114c5e4979ca1209c99f/psycopg2-2.7.7-cp37-cp37m-manylinux1_x86_64.whl (2.7MB)
    100% |????????????????????????????????| 2.7MB 14.4MB/s
Installing collected packages: psycopg2
Successfully installed psycopg2-2.7.7

To deactivate our virtualenv, we can use the deactivate executable that was put into our $PATH:

(pg) $ deactivate
$

Using pipenv

The last tool that we're going to look at is a newer tool, pipenv, built to handle both creating and working with a virtualenv, and also managing dependencies. Let's install pipenv and see what the workflow looks like:

$ pip3.7 install --user pipenv
...

Rather than putting all of the virtualenvs in a directory that we have to navigate to, Pipenv would have us create a project and then initialize a virtualenv for it. Let's create a database project and then initialize a virtualenv for it using Pipenv:

$ mkdir ~/database
$ cd ~/database
$ pipenv --python python3.7
Creating a virtualenv for this project…
Pipfile: /home/cloud_user/database/Pipfile
Using /usr/local/bin/python3.7 (3.7.2) to create virtualenv…
? Creating virtual environment...Already using interpreter /usr/local/bin/python3.7
Using base prefix '/usr/local'
New python executable in /home/cloud_user/.local/share/virtualenvs/database-OGMn9Yao/bin/python3.7
Also creating executable in /home/cloud_user/.local/share/virtualenvs/database-OGMn9Yao/bin/python
Installing setuptools, pip, wheel...
done.

? Successfully created virtual environment!
Virtualenv location: /home/cloud_user/.local/share/virtualenvs/database-OGMn9Yao

A few things to note here:

  1. Pipenv manages a virtualenv for each project that we set up within ~/.local/share/virtualenvs
  2. We now have a Pipfile which is a more customizable file for us to manage our dependencies in than the requirements.txt file that we generated earlier.

To activate our virtualenv, we can use a new command from within our project directory:

$ pipenv shell
Launching subshell in virtual environment…
$  . /home/cloud_user/.local/share/virtualenvs/database-OGMn9Yao/bin/activate
(database) $

When we want to add a dependency to our project we also use pipenv:

(database) $ pipenv install psycopg2
Installing psycopg2…
Adding psycopg2 to Pipfile's [packages]…
? Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
? Success!
Updated Pipfile.lock (59b6f6)!
Installing dependencies from Pipfile.lock (59b6f6)…
  ???????????????????????????????? 1/1 — 00:00:00

The big difference between this and just installing the package using pip is that Pipenv will work through a dependency graph to make sure that if two of our dependencies have a common dependency, then we install a version that is compatible with both of our explicit dependencies.

The last thing to note is that to exit our virtualenv, we can simply use exit and it will drop us back to our previous session without the virtualenv loaded.


This lesson is only available to Linux Academy members.

Sign Up To View This Lesson

Or Log In

Looking For Team Training?

Learn More