A Python Environment Setup

I’ve been told that python package management is bad. I have seen some really bad practices online, asking you to run commands here and there without an understanding of the bigger picture, what they do and sometimes with escalated privileges.

Along the years, I have compiled a list of practices I follow, and a list of tools I use. I hope to be able to share some of the knowledge I’ve acquired and show you a different way of doing things. You might learn about a new tool, or a new use for a tool. Come along for the ride !

Python

As most know, Python is an interpreted programming language. I am not going to go into the details of the language in this post, I will only talk about management.

If you want to develop in Python, you need to install libraries. You can find some in your package manager but let’s face it pip is your way.

The majority of Linux distributions will have Python installed as a lot of system packages now rely on it, even some package managers.

Okay, this is the last time I actually use the system’s Python. What ? Why ? You ask !

pyenv

I introduce you to pyenv. Pyenv is a Python version management tool, it allows you to install and manage different versions of Python as a user.

Beautiful, music to my ears.

Let’s get it from the package manager, this is a great use of the package manager if it offers an up to date version of the package.

sudo pacman -S pyenv

If you’re not using an Archlinux based distribution follow the instructions on their webpage.

Alright ! Now that we’ve got ourselves pyenv, let’s configure it real quickly.

Following the docs, I created ~/.config/fish/config.d/pyenv.fish and in it I put the following.

# Add pyenv executable to PATH by running
# the following interactively:

set -Ux PYENV_ROOT $HOME/.pyenv
set -U fish_user_paths $PYENV_ROOT/bin $fish_user_paths

# Load pyenv automatically by appending
# the following to ~/.config/fish/config.fish:

status is-login; and pyenv init --path | source

Open a new shell and you’re all ready to continue along, you’re all locked, loaded and ready to go!

Setup the environment

This is the first building block of my environment. We first start by querying for Python versions available for us.

pyenv install --list

Then, we install the latest Python version. Yes, even if it’s an upgrade, I’ll handle the upgrade, as well, as we go along.

Set everything up to use the new installed version.

First, we set the global Python version for our user.

pyenv global 3.9.5

Then, we switch our current shell’s Python version, instead of opening a new shell.

pyenv shell 3.9.5

That was easy. We test that everything works as expected by checking the version.

pyenv version

Now, if you do a which on the python executable, you will find that it is in the pyenv shims' directory.

Upgrade

In the future, the upgrade path is exactly the same as the setup path shown above. You query for the list of Python versions available, choose the latest and move on from there. Very easy, very simple.

pip

pip is the package installer for Python.

At this stage, you have to understand that you are using a Python version installed by pyenv as your user. The pip provided, if you do a which, is also in the same shims directory.

Using pip at this stage as a user is better than running it as root but it is also not touching your system; just your user. We can do one better. I’m going to use pip as a user once !

I know, you will have a lot of questions at this point as to why. You will see, patience is a virtue.

pipx

Meet pipx, this tool is the amazing companion for a DevOps, and developer alike. Why ? You would ask.

It, basically, creates Python virtual environments for packages you want to have access to globally. For example, I’d like to have access to a Python LSP server on the go. This way my text editor has access to it too and, of course, can make use of it freely. Anyway, let’s cut this short and show you. You will understand better.

Let’s use the only pip command as a user to install pipx.

pip install --user pipx

LSP Server

As I gave the LSP server as an example, let’s go ahead and install it with some other Python packages needed for global things like emacs.

pipx install black
pipx install ipython
pipx install isort
pipx install nose
pipx install pytest
pipx install python-lsp-server

Now each one is in it’s own happy little virtual environment separated from any other dependency but its own. Isn’t that lovely ?

If you try to run ipython, you will see that it will actually work. If you look deeper at it, you will see that it is pointing to ~/.local/bin/ipython which is a symlink to the actual package in a pipx virtual environment.

Upgrade

After you set a new Python version with pyenv, you simply reinstall everything.

pipx reinstall-all

And like magic, everything get recreated using the new version of Python newly set.

virtualfish

Now that pipx is installed, let’s go head and install something to manage our Python virtual environments on-demand, for use whenever we need to, for targeted projects.

Some popular choices people use are Pipenv, Poetry, virtualenv and plain and simple python with the venv module. You’re welcome to play with all of them. Considering I use fish as my default shell, I like to use virtualfish.

Let’s install it.

pipx install virtualfish

This offers me a new command; vf. With vf, I can create Python virtual environments and they will all be saved in a directory of my choosing.

Setup

Let’s create one for Ansible.

vf new ansible

This should activate it. Then, we install Ansible.

pip install ansible molecule docker

At this stage, you will notice that you have ansible installed. You will also notice that all the pipx packages are also still available.

If you want to tie virtualfish to a specific directory, use vf connect.

Upgrade

To upgrade the Python version of all of our virtual environments, virtualfish makes it as easy as

vf upgrade

And we’re done !

Workflow

At this stage, you have an idea about the tools I use and where their scope falls. I like them because they are limited to their own scope, each has its own little domain where it reigns.

• I use pyenv to install and manage different versions of Python for testing purposes while I stay on the latest.
• I use pipx for the commands that I need access to globally as a user.
• I use virtualfish to create one or more virtual environment per project I work on.

With this setup, I can test with different versions of Python by creating different virtual environments with different version each, or two versions of the tool you’re testing as you keep the Python version static. It could also be different versions of a library, testing forward compatibility for example.

At each step, I have an upgrade path to keep all my environments running the latest versions. I also have a lot of flexibility by using requirements.txt files and others for development sometimes or even testing.

Conclusion

As you can see, with a little bit of knowledge and by standing on the shoulders of giants, you can easily manage a Python environment entirely as a user. You have full access to a wide array of Python distributions to play with. Endless different versions of packages, globally and locally installed. If you create virtual environments for each of your projects, you won’t fall in the common pitfalls of versioning hell. Keep your virtual environments numerous and dedicated to projects, small sets, and you won’t face any major problems with keeping your system clean yet up to date.