⇦

2021-01-03

Python virtualenv s for Node devs

You've started doing some Python while working with another team, and they've told you to do everything in virtualenv s as it's the right way. You've also been told Python's packaging system is a bit of a mess, let's try tame it somewhat.

We're going to compare npm to the "modern classic" Python equivalent. I'm going to ignore more recent developments like poetry and friends.

Let's install a python version with brew:

brew install python@3.9

OK, so we've got a Python version that seems sane, let's set up a virtualenv with the equivalent of npm init .

$(brew --prefix python@3.9)/libexec/bin/python -m venv .env

We now have a new folder in our current directory called .env that contains symlinks to the Python that we just ran:

.env
├── bin
│   ├── Activate.ps1
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── easy_install
│   ├── easy_install-3.9
│   ├── pip
│   ├── pip3
│   ├── pip3.9
│   ├── python -> python3
│   ├── python3 -> /opt/homebrew/opt/python@3.11/bin/python3.11
│   └── python3.9 -> python3
├── include
├── lib
│   └── python3.9
│       └── site-packages
│           ├── __pycache__
│           ├── easy_install.py
│           ├── pip
│           ├── pip-20.2.3.dist-info
│           ├── pkg_resources
│           ├── setuptools
│           └── setuptools-49.2.1.dist-info
└── pyvenv.cfg

We could always run .env/bin/python or .env/bin/pip install flask or whatever, but virtualenv s come with a special trick file to add .env/bin to bash 's PATH. Let's run it:

source .env/bin/activate

Now everything should be set up. Let's try:

(.env) ➜ echo $PATH
/Users/user/src/my-special-project/.env/bin:/usr/local/bin:/usr/bin:/bin
(.env) ➜ which python
/Users/user/src/my-special-project/.env/bin/python
(.env) ➜ which pip
/Users/user/src/my-special-project/.env/bin/pip
(.env) ➜ python --version
Python 3.9.1

The main difference from npm here is we don't have to run anything with npm run ... as we've hacked with the PATH.

Installing stuff

Let's install flask :

pip install flask

Now our directory should look like:

.env
├── bin
│   ├── python -> python3
│   ├── flask
│   └── ...
├── lib
│   └── python3.9
│       └── site-packages
│           ├── click
│           ├── flask
│           └── ...

As you can see, .env/lib/python3.9/site-packages is a lot like node_modules . We can see all the library's files:

(.env) ➜ ls .env/lib/python3.9/site-packages/flask
__init__.py     _compat.py      cli.py          debughelpers.py json            signals.py      views.py
__main__.py     app.py          config.py       globals.py      logging.py      templating.py   wrappers.py
__pycache__     blueprints.py   ctx.py          helpers.py      sessions.py     testing.py

Now, what is the equivalent to the following in a package.json ?

{
    ...
    "dependencies": {
      "express": "^4.17.1"
    }
}

Python is a fair bit different from Node in that only one version of a package can be installed in a virtualenv at a time. This means you tend to have abstract requirements (that is to say with very limited version pinning) listed in setup.py|requirements.in|pyproject.toml and pinned requirements (for testing/deployent if you're working on a service) in a requirements.txt .

You can install a set of requirements with:

pip install -r requirements.txt

Or whatever file it is you want to install from. This is the equivalent of npm install .

In terms of pinning requirements in your own project a la yarn lock or whatever the flavour of the month is, I've always been a fan of pip-tools - using this means you can safely ignore the last few years of Python packaging excitement and get on with your life.