Python development environment basics

Installing

Installing Python on Linux

On Linux we can install Python from the package manager. It is probably already installed. The details depend on your particular Linux distribution. On Debian and Debian-derivatives, we can install programs using apt.

With root rights (or sudo):

apt install python3

After installing, we should be able to run python, python3 as a command in the commandline.

Installing an alternative version of Python on Linux

Sometimes we can't want to use the system-wide version of Python for a specific project or software. I did use pyenv for this in the past, but as of Juli 2024 I encounter a number of bugs that I have not been able to work around.

In this example we will install Python version 3.7.8, don't forget to adjust the example if you want to install a different version. libssl-dev will need to be installed before compiling Python, otherwise we won't have SSL support.

apt install -y libssl-dev
wget https://www.python.org/ftp/python/3.7.8/Python-3.7.8.tgz
tar xzvf Python-3.7.8.tgz
cd Python-3.7.8
./configure --enable-optimizations
sudo make altinstall

Here we build this python version as an altinstall, meaning "alternative installation" as opposed to overwriting the system installation. It can now be called as python3.7. The binaries can be found in /usr/local/bin:

$ ls -al /usr/local/bin
...
-rwxr-xr-x  1 root root      229 Jul 17 08:35 pip3.7
-rwxr-xr-x  2 root root 11906976 Jul 17 08:35 python3.7
...

Dependencies will need to be installed separately for this version. The Python package manager pip can be called as a module of this python version, or we can use the binary:

python3.7 -m pip
pip3.7

In my situation the installed packages end up in ~/.local/lib/python3.7/site-packages/.

To make this all work with the software we installed this Python version for, one will probably need to change some setup or install scripts, or wrap them with another script that sets up any aliases you need. Important: do not set system-wide aliases because this will likelu break many parts of your system.

Installing Python on Windows

On Windows we can use the Microsoft Store and install the latest Python version. This comes with pip. We may find that we have to call Python with py rather than python or python3.

Virtual environments

We want to create a so-called virtual environment for each project. What that means is that although we use the same physical environment (our computer) for each project, we want to have a separation between what we use for our different project. We don't want the tools and dependencies for one project to mess with our other projects.

The virtual environment is for the development environment. For production environments there are other things we need to consider.

We can set up a virtual environment by running the following in the project's root directory:

python3 -m venv .venv

On Linux and Linux-a-likes such as macOS, MinGW, WSL and Git Bash we can then use our environment by running:

source .venv/bin/activate

On the Windows commandline we can use it by running:

.venv\Scripts\activate.bat

On Windows powershell we can use:

.venv\Scripts\Activate.ps1

We don't want to commit our environment directory to git; it will have to be created on each different computer. Let's add the following to our .gitignore file:

**/.venv

Dependency management / Installing packages

pip

Pip is the Package Installer for Python.

We may need to install the python package manager pip separately. On Debian we can do that with: apt install python3-pip.

After installing, we should be able to run pip as a command in the commandline. Note that on some systems we can not just run a python module like pip, we have to explicitly run it as a python module instead:

python3 -m pip install pelican[markdown]

Installing a package

If we know the name of the package we can install it right away

python3 -m pip install pelican[markdown]

Note that the dependencies for each package will be installed automatically. We can also install multiple packages in one go:

pip install pelican[markdown] pelican-feed-filter

The full package name is "pelican[markdown]". The "[]" notation is used to indicate that there are multiple pelican packages, and we want the markdown version. In the case of pelican there are three versions: rst, markdown, and asciidoc. These are all different markup languages. We want to use markdown, but with rst being the default we need to specify that during install time. There is nothing special about the "package-name[something-specific]" notation, it is only a convention. "pelican[markdown]" is the name of a specific package, and "pelican" is not the same package. If we called python3 -m pip install pelican markdown instead, we would be trying to install the wrong "pelican" package, as well as a separate package called "markdown". We would not be installing the pelican version that is specifically configured to us markdown.

Installing a package directly from a git repository

It is possible to install directly from a git repository:

python3 -m pip install git+https://github.com/getpelican/pelican.git
python3 -m pip install git+ssh://git@github.com:getpelican/pelican.git

Dependency management

The requirements file

Projects tend to list their dependencies in a file called requirements.txt, with lines like this:

pyparsing>=2.0.1

Here we see that the pyparsing library is a requirement, and the minimum required version is 2.0.1.

The requirements can then be installed like this:

pip install -r requirements.txt

The -r flag indicates to pip that the next argument is the path to a requirements file that we want to use.

Generating a requirements file.

The requirements file can be generated automatically by using pip's freeze function, python3 -m pip freeze.

Let's say we start a new project, with a new empty (virtual) environment. If we run pip freeze we will not have any output because nothing is installed in our new empty environment. Let's install some packages: pip install pelican[markdown]. Afterwards we will get output from the pip freeze command:

>python3 -m pip freeze
blinker==1.5
commonmark==0.9.1
docutils==0.19
feedgenerator==2.0.0
importlib-metadata==4.12.0
Jinja2==3.1.2
Markdown==3.4.1
MarkupSafe==2.1.1
pelican==4.8.0
Pygments==2.13.0
python-dateutil==2.8.2
pytz==2022.2.1
rich==12.5.1
six==1.16.0
Unidecode==1.3.4
zipp==3.8.1

We can route this to a file. Let's give it the standard name, requirements.txt, to be consistent with other projects.

>pip freeze > requirements.txt

Style guide

The official style guide can be found here.

How to manually update or patch an official Python package

Sometimes we might need to do a bugfix on a Python package, or we need to use the latest version from the repository that is not yet packaged for our platform. How can we install this new version? This is very simple, we only need to copy the source files.

For example, if we want to update our system version of a package 'anyplugin' that lives in the 'pelican plugins' namespace, we simply overwrite the system version with our source files.

cp -r anyplugin /usr/local/lib/python3.8/site-packages/pelican/plugins/

The directory we copy is the actual python source file directory, this is not necessarily the directory root.

How to combine incoming byte chunks

I asked about this on StackOverflow, and then I had to look up the answer again on another occasion. So I'll list it here :). To safe you some clicks, there are two options for the byte storage:

• Instantiating an empty byte string, b'', and appending / concatenating. This incurs a bit of string copy overhead.
• Instantiating a new empty byte array, and then extending it (appending to it).

The code might look as follows:

def process_incoming_connection(conn, addr):
    byte_array = bytearray()
    while 1:
        chunk = conn.recv(1024)
        if not chunk:
            break
        byte_array.extend(chunk)
    message = byte_array.decode(encoding='utf-8');
    conn.close()
    # Now we can do something with the message