Python development environment basics
Installing
Installing Python on Linux
On Linux we can install Python from the package manager. It is probably already installed. The details depend on your particular Linux distribution. On Debian and Debian-derivatives, we can install programs using apt.
With root rights (or sudo):
apt install python3
After installing, we should be able to run python
, python3
as a command in the commandline.
Installing an alternative version of Python on Linux
Sometimes we can't want to use the system-wide version of Python for a specific project or software.
I did use pyenv
for this in the past, but as of Juli 2024 I encounter a number of bugs
that I have not been able to work around.
In this example we will install Python version 3.7.8, don't forget to adjust the example if you want to install a different version. libssl-dev will need to be installed before compiling Python, otherwise we won't have SSL support.
apt install -y libssl-dev
wget https://www.python.org/ftp/python/3.7.8/Python-3.7.8.tgz
tar xzvf Python-3.7.8.tgz
cd Python-3.7.8
./configure --enable-optimizations
sudo make altinstall
Here we build this python version as an altinstall, meaning "alternative installation" as opposed to
overwriting the system installation. It can now be called as python3.7
. The binaries can be found in
/usr/local/bin
:
$ ls -al /usr/local/bin
...
-rwxr-xr-x 1 root root 229 Jul 17 08:35 pip3.7
-rwxr-xr-x 2 root root 11906976 Jul 17 08:35 python3.7
...
Dependencies will need to be installed separately for this version. The Python package manager pip
can be called
as a module of this python version, or we can use the binary:
python3.7 -m pip
pip3.7
In my situation the installed packages end up in ~/.local/lib/python3.7/site-packages/
.
To make this all work with the software we installed this Python version for, one will probably need to change some setup or install scripts, or wrap them with another script that sets up any aliases you need. Important: do not set system-wide aliases because this will likelu break many parts of your system.
Installing Python on Windows
On Windows we can use the Microsoft Store and install the latest Python version. This comes with pip.
We may find that we have to call Python with py
rather than python
or python3
.
Virtual environments
We want to create a so-called virtual environment for each project. What that means is that although we use the same physical environment (our computer) for each project, we want to have a separation between what we use for our different project. We don't want the tools and dependencies for one project to mess with our other projects.
The virtual environment is for the development environment. For production environments there are other things we need to consider.
We can set up a virtual environment by running the following in the project's root directory:
python3 -m venv .venv
On Linux and Linux-a-likes such as macOS, MinGW, WSL and Git Bash we can then use our environment by running:
source .venv/bin/activate
On the Windows commandline we can use it by running:
.venv\Scripts\activate.bat
On Windows powershell we can use:
.venv\Scripts\Activate.ps1
We don't want to commit our environment directory to git; it will have to be created on each different computer. Let's add the following to our .gitignore file:
**/.venv
Dependency management / Installing packages
pip
Pip is the Package Installer for Python.
We may need to install the python package manager pip
separately.
On Debian we can do that with: apt install python3-pip
.
After installing, we should be able to run pip
as a command in the commandline.
Note that on some systems we can not just run a python module like pip
,
we have to explicitly run it as a python module instead:
python3 -m pip install pelican[markdown]
Installing a package
If we know the name of the package we can install it right away
python3 -m pip install pelican[markdown]
Note that the dependencies for each package will be installed automatically. We can also install multiple packages in one go:
pip install pelican[markdown] pelican-feed-filter
The full package name is "pelican[markdown]". The "[]" notation is used to indicate
that there are multiple pelican packages, and we want the markdown version.
In the case of pelican there are three versions: rst, markdown, and asciidoc. These are all different markup languages.
We want to use markdown, but with rst being the default we need to specify that during install time.
There is nothing special about the "package-name[something-specific]" notation, it is only a convention.
"pelican[markdown]" is the name of a specific package, and "pelican" is not the same package.
If we called python3 -m pip install pelican markdown
instead, we would be trying to install the wrong "pelican"
package, as well as a separate package called "markdown". We would not be installing the pelican version that
is specifically configured to us markdown.
Installing a package directly from a git repository
It is possible to install directly from a git repository:
python3 -m pip install git+https://github.com/getpelican/pelican.git
python3 -m pip install git+ssh://git@github.com:getpelican/pelican.git
Dependency management
The requirements file
Projects tend to list their dependencies in a file called requirements.txt, with lines like this:
pyparsing>=2.0.1
Here we see that the pyparsing library is a requirement, and the minimum required version is 2.0.1.
The requirements can then be installed like this:
pip install -r requirements.txt
The -r
flag indicates to pip that the next argument is the path to a requirements file that we want to use.
Generating a requirements file.
The requirements file can be generated automatically by using pip's freeze function,
python3 -m pip freeze
.
Let's say we start a new project, with a new empty (virtual) environment.
If we run pip freeze
we will not have any output because nothing is installed in our new empty environment.
Let's install some packages: pip install pelican[markdown]
.
Afterwards we will get output from the pip freeze command:
>python3 -m pip freeze
blinker==1.5
commonmark==0.9.1
docutils==0.19
feedgenerator==2.0.0
importlib-metadata==4.12.0
Jinja2==3.1.2
Markdown==3.4.1
MarkupSafe==2.1.1
pelican==4.8.0
Pygments==2.13.0
python-dateutil==2.8.2
pytz==2022.2.1
rich==12.5.1
six==1.16.0
Unidecode==1.3.4
zipp==3.8.1
We can route this to a file. Let's give it the standard name, requirements.txt, to be consistent with other projects.
>pip freeze > requirements.txt
Style guide
The official style guide can be found here.
How to manually update or patch an official Python package
Sometimes we might need to do a bugfix on a Python package, or we need to use the latest version from the repository that is not yet packaged for our platform. How can we install this new version? This is very simple, we only need to copy the source files.
For example, if we want to update our system version of a package 'anyplugin' that lives in the 'pelican plugins' namespace, we simply overwrite the system version with our source files.
cp -r anyplugin /usr/local/lib/python3.8/site-packages/pelican/plugins/
The directory we copy is the actual python source file directory, this is not necessarily the directory root.
How to combine incoming byte chunks
I asked about this on StackOverflow, and then I had to look up the answer again on another occasion. So I'll list it here :). To safe you some clicks, there are two options for the byte storage:
- Instantiating an empty byte string,
b''
, and appending / concatenating. This incurs a bit of string copy overhead. - Instantiating a new empty byte array, and then extending it (appending to it).
The code might look as follows:
def process_incoming_connection(conn, addr):
byte_array = bytearray()
while 1:
chunk = conn.recv(1024)
if not chunk:
break
byte_array.extend(chunk)
message = byte_array.decode(encoding='utf-8');
conn.close()
# Now we can do something with the message