wiki/wiki/programming_language/python.md

Python

Python is an interpreted general-purpose programming language.

Setup

You can install python using various ways.

Installation From Package Managers

Using various package managers the current python version can be easily installed. Additionally, a pyenv, uv or manual installation can be done to get a specific older version for projects.

pyenv Installation

With pyenv you can easily switch between different versions. Install pyenv and pyenv-virtualenv and proceed with adding

export PATH=${HOME}/.pyenv/bin:$PATH
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

to your ~/.profile. You can then set and install your preferred version of python globally with pyenv install <version> and pyenv global <version>.

Analog to managing python versions, pipenv can manage pip and package versions. A guide and description of the usage can be found on gioele.io.

Manual Installation

This section is based on a guide by Linuxize.

On Linux-based systems Python can be easily installed using make. The following shows a process of installing Python 3.9.1. When another Python version is needed the version number has to be adapted accordingly.

First make sure all the dependencies are installed. On a Ubuntu system or other systems using the apt package manager this can look like the following

sudo apt update
sudo apt install build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev libsqlite3-dev wget libbz2-dev

For Tk programs (like matplotlib) to work correctly the following dependencies need to be installed aswell.

sudo apt install tk-dev tcl-dev

Afterward the python source code can be downloaded.

wget https://www.python.org/ftp/python/3.9.1/Python-3.9.1.tgz
tar -xf Python-3.9.1.tgz
cd Python-3.9.1

Afterward if OpenSSL should be enabled for Python uncomment all the corresponding lines in Modules/Setup. This part of the guide is taken from a StackOverflow post by Ironman.

Then proceed with the installation.

./configure --enable-optimizations
make
sudo make altinstall

Afterward you can check if the Pyhton version is installed using the following command.

python3.9 --version

Usage

This section addresses the usage of Python.

Executable Python Scripts

Python scripts can be made directly executable without explicitly calling the Python executable itself every time. For general information about shebangs, executable scripts and the chmod +x command, refer to the corresponding section in the Shell article.

For Python, it is recommended to use the following shebang.

#!/usr/bin/env python

Using /usr/bin/env python makes the script compatible with virtual environments, uv projects and other Python environment managers.

Convert .ipynb Files to .py Files and Back

Jupyter notebooks in the .ipynb format can easily be converted to normal Python files using jupytext. This can sometimes be useful when trying to avoid using notebooks.

The following command will convert the file <file>.ipynb to a normal Python file. <input-file> describes the path without the file extension.

jupytext --to py <input-file>.ipynb

The same also works the other way around.

jupytext --to ipynb <input-file>.py

Using md in the --to option the notebook can also be converted to a markdown file and back.

Alternatively notedown can also convert to markdown using the following commands. <output-file> is the path to the output file again without the extension.

notedown <input-file>.ipynb --to markdown > <output-file>.md

Create a requirements file

To automatically create a requirements.txt of your current project, navigate to it and run pipreqs (install it if not already done).

Using Virtual Environments

There are various options to use virtual environments. The first and most commonly used option is venv. A more modern and arguably better approach is using uv.

venv Virtual Environments

venv can be used to create a virtual environment. It usually uses the standard pip manager for installing libraries.

python -m venv <project-path>

When inside the project folder the virtual environment can then be acivated by running the following command.

source ./bin/activate

And it can be disabled by running the following.

deactivate

If pyenv is installed as described in the setup section pyenv can be used to manage virtual environments. However pyenv won't create environment directories like venv does. To fix this the following command can be used to simply link them both as described on StackOverflow by Jakob Guldberg Aaes. For this to work a virtual environment already has to be set up using venv as described before. <local-venv> is the full path of the local virtual environment just created and <venv-name> the name the venv should have in pyenv.

ln -s <local-venv> ~/.pyenv/versions/<venv-name>

Using this setup the python version will automatically change when navigating into the project directory.

Package Management

Python packages can be installed using package managers. For new projects it is generally recommended to use uv, while pip is available by default in most Python installations and is supported by virtually all Python projects.

pip

The pip package manager is the standard Python package manager and can be used to install, upgrade and remove Python packages. It is recommended to use pip inside a virtual environment.

A package called <package> can be installed using

pip install <package>

To upgrade an already installed package run

pip install --upgrade <package>

Packages can be removed again using

pip uninstall <package>

To display all currently installed packages use

pip list

uv

For new projects it is generally recommended to use uv, which provides a modern replacement for pip, venv and several other Python tools. The corresponding article describes installation, virtual environments, dependency management and further usage in detail.

Modules

Python modules can be installed using package managers such as pip or uv.

For most projects it is recommended to use a virtual environment. For global installation the system package managers can be used.

Modules

Python modules can be installed using package managers such as pip or uv.

For most projects it is recommended to use a virtual environment. For global installation the system package managers can be used. Alternatively pip and uv can also handle global installation.

scikit-learn

scikit-learn is a free and open-source library that provides various machine learning algorithms. By default, scikit-learn does only utilize the CPU. This, however, can be easily changed to also utilize the GPU and speed up calculation as explained in the corresponding section

Run scikit-learn Algorithms on the GPU

Using cuml the GPU can be utilized with only two lines to add to existing code. As explained in the cuml guide on zero code change acceleration only the following two lines have to be added to run the scikit-learn algorithms on the GPU. Additionally, cuml has to be installed using a Python package manager like uv. It is important that these lines are put before importing any scikit-learn packages.

import cuml
cuml.accel.install()

Afterward all possible scikit-learn algorithms will run on the GPU instead of the CPU.

PyTorch

This section addresses the PyTorch module. PyTorch is a machine learning resource which is often used for neural networks.

Setup PyTorch with CUDA for GPU usage

CUDA is also only available for Nvidia GPUs. For AMD GPUs refer to the ROCm section.

If you are on Arch Linux or a distribution based on it install python-pytorch-cuda via pacman -S python-pytorch-cuda.

After that visit the official PyTorch website and install PyTorch for your custom configuration.

After that try to run the following python script. This should give back True if the setup was successful and the GPU is available.

import torch

torch.cuda.is_available()

Please note that according to various sources CUDA 13 does not support nVidia Pascal GPUs. In this case an earlier version of CUDA has to be used.

Setup PyTorch with ROCm for GPU usage

ROCm is also only available for AMD GPUs. For Nvidia GPUs refer to the CUDA section.

For ROCm to work some system packages have to be installed. For Linux refer to the official Linux guide. For Arch Linux although it is not listed the required AUR packages include rocm-core, rocminfo, roctracer and rccl. For Windows refer to the official Windows guide.

After the systemwide setup the project environment can be setup. It is recommended to use virtual environments. The easiest way to achieve ROCm support is by using pip. As explained in the official PyTorch guide the following example can be used inside the environment to install all needed ROCm packages. In this example the version 7.2 is installed. Adjustments to the command may have to be done in case another version should be installed.

wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torch-2.9.1%2Brocm7.2.0.lw.git7e1940d4-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torchvision-0.24.0%2Brocm7.2.0.gitb919bd0c-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/triton-3.5.1%2Brocm7.2.0.gita272dfa8-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torchaudio-2.9.0%2Brocm7.2.0.gite3c6ee2b-cp312-cp312-linux_x86_64.whl
pip install \
torch-2.9.1+rocm7.2.0.lw.git7e1940d4-cp312-cp312-linux_x86_64.whl \
torchvision-0.24.0+rocm7.2.0.gitb919bd0c-cp312-cp312-linux_x86_64.whl \
torchaudio-2.9.0+rocm7.2.0.gite3c6ee2b-cp312-cp312-linux_x86_64.whl \
triton-3.5.1+rocm7.2.0.gita272dfa8-cp312-cp312-linux_x86_64.whl

If old versions of torch, torchvision, torchaudio or triton are installed inside the environment they may need to be removed.

After this installation for some GPUs – especially integrated GPUs like the Radeon 660M – an additional step has to be taken. In this case the following global shell variable has to be set.

export HSA_OVERRIDE_GFX_VERSION=10.3.0

After that try to run the following python script. Since ROCm uses a bridge to access CUDA it should give back True if the setup was successful and the GPU is available.

import torch

torch.cuda.is_available()

Hailo

The package for Hailo chips has to be downloaded from the official website. Additional setup may be required as explained in the Hailo article.

Hailo chips can be used to run converted TensorFlow models. The conversion process is explained in the Hailo article.

To run the inference using Python on ARM boards like the Raspberry Pi AI Hat + zlodeibaal's article in Medium can be referenced.

TensorFlow

This section addresses the TensorFlow module. Tensorflos is a machine learning resource which is often used for neural networks.

Apart from package managers and virtual environments it can also be set up from source as explained on the official website. This may especially be useful if specific configurations are needed such as vendor specific GPU support.

Basic Usage of TensorFlow

The basic usage of TensorFlow is described in the official guide. Additionally, it is noted that the dataset may have to be shuffled manually as described in a comment by Y. Luo on StackOverflow. Finally, a Medium blog post describes how to easily change the learning rate.

Combining Models

Models that are normally run in sequence but trained and saved separately can be easily be combined into a single model. This can have some advantages, for example when using inteference for deep learning on Edge TPUs like the Hailo chips or the EPS32S3.

A simple example for the combination of two models (model1 and model2) into a new model (combined_model) is the following code.

output = model2(model1.output)
combined_model = tf.keras.models.Model(inputs=model1.input, outputs=output)

matplotlib

The matplotlib module is a plotting library for Python.

Qt Backend Not Found

When using Wayland, matplotlib may fail with an error similar to the following.

ImportError: Failed to import any of the following Qt binding modules:
PyQt6, PySide6, PyQt5, PySide2

This happens because matplotlib uses the Qt backend (QtAgg) for displaying interactive windows, but no Qt Python bindings are installed.

To fix the issue simply install PyQt6 inside the current Python environment. This may look like the following example where uv is used but other environments such as pip can also be used.

uv add PyQt6

After installing PyQt6, interactive matplotlib windows should work correctly under Wayland.