# Python

[Python](https://www.python.org) is an interpreted general-purpose programming
language.

## Setup

You can install python using various ways.

### Installation From Package Managers

Using various [package managers](/wiki/linux/package_manager.md) the current python version can be
easily installed.
Additionally, a [pyenv](#pyenv-installation), [uv](/wiki/programming_language/python/uv.md) or
[manual installation](#manual-installation) can be done to get a specific older version for projects.

### pyenv Installation

With [pyenv](https://github.com/pyenv/pyenv) you can easily switch between different versions.
Install `pyenv` and `pyenv-virtualenv` and proceed with adding

```txt
export PATH=${HOME}/.pyenv/bin:$PATH
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
```

to your `~/.profile`.
You can then set and install your preferred version of python globally with
`pyenv install <version>` and `pyenv global <version>`.

Analog to managing python versions, `pipenv` can manage [pip](#pip) and
[package versions](#package-management).
A guide and description of the usage can be found on [gioele.io](https://gioele.io/pyenv-pipenv).

### Manual Installation

This section is based on a guide by [Linuxize](https://linuxize.com/post/how-to-install-python-3-9-on-ubuntu-20-04/).

On [Linux-based systems](/wiki/linux.md) Python can be easily installed using make.
The following shows a process of installing Python `3.9.1`.
When another Python version is needed the version number has to be adapted accordingly.

First make sure all the dependencies are installed.
On a [Ubuntu](/wiki/linux.md#distributions) system or other systems using the apt
[package manager](/wiki/linux/package_manager.md) this can look like the following

```sh
sudo apt update
sudo apt install build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev libsqlite3-dev wget libbz2-dev
```

For Tk programs (like matplotlib) to work correctly the following dependencies need to be installed
aswell.

```sh
sudo apt install tk-dev tcl-dev
```

Afterward the python source code can be downloaded.

```sh
wget https://www.python.org/ftp/python/3.9.1/Python-3.9.1.tgz
tar -xf Python-3.9.1.tgz
cd Python-3.9.1
```

Afterward if OpenSSL should be enabled for Python uncomment all the corresponding lines in
`Modules/Setup`.
This part of the guide is taken from a
[StackOverflow post by Ironman](https://stackoverflow.com/questions/58309485/modulenotfounderror-no-module-named-ssl).

Then proceed with the installation.

```sh
./configure --enable-optimizations
make
sudo make altinstall
```

Afterward you can check if the Pyhton version is installed using the following command.

```sh
python3.9 --version
```

## Usage

This section addresses the usage of Python.

### Executable Python Scripts

Python scripts can be made directly executable without explicitly calling the Python executable
itself every time.
For general information about shebangs, executable scripts and the `chmod +x` command, refer to the
corresponding section in the [Shell article](/wiki/linux/shell.md#shebangs).

For Python, it is recommended to use the following shebang.

```py
#!/usr/bin/env python
```

Using `/usr/bin/env python` makes the script compatible with
[virtual environments](#using-virtual-environments), [uv](/wiki/programming_language/python/uv.md)
projects and other Python environment managers.

### Convert `.ipynb` Files to `.py` Files and Back

Jupyter notebooks in the `.ipynb` format can easily be converted to normal Python files using
[jupytext](https://jupytext.readthedocs.io/en/latest/).
This can sometimes be useful when trying to avoid using notebooks.

The following command will convert the file `<file>.ipynb` to a normal Python file.
`<input-file>` describes the path without the file extension.

```sh
jupytext --to py <input-file>.ipynb
```

The same also works the other way around.

```sh
jupytext --to ipynb <input-file>.py
```

Using `md` in the `--to` option the notebook can also be converted to a
[markdown](/wiki/markup_language.md) file and back.

Alternatively [notedown](https://pypi.org/project/notedown/) can also convert to markdown using the
following commands.
`<output-file>` is the path to the output file again without the extension.

```sh
notedown <input-file>.ipynb --to markdown > <output-file>.md
```

### Create a requirements file

To automatically create a `requirements.txt` of your current project, navigate
to it and run `pipreqs` (install it if not already done).

### Using Virtual Environments

There are various options to use virtual environments.
The first and most commonly used option is [venv](#venv-virtual-environments).
A more modern and arguably better approach is using
[uv](/wiki/programming_language/python/uv.md#virtual-environments).

#### venv Virtual Environments

[venv](https://docs.python.org/3/library/venv.html) can be used to create a virtual environment.
It usually uses the standard [pip](#pip) manager for installing libraries.

```
python -m venv <project-path>
```

When inside the project folder the virtual environment can then be acivated by running the
following command.

```sh
source ./bin/activate
```

And it can be disabled by running the following.

```sh
deactivate
```

If pyenv is installed as described in [the setup section](#pyenv-installation) pyenv can be used to
manage virtual environments.
However pyenv won't create environment directories like venv does.
To fix this the following command can be used to simply link them both as described
[on StackOverflow by Jakob Guldberg Aaes](https://stackoverflow.com/questions/30407446/pyenv-choose-virtualenv-directory).
For this to work a virtual environment already has to be set up using venv as described before.
`<local-venv>` is the full path of the local virtual environment just created and `<venv-name>` the
name the venv should have in pyenv.

```sh
ln -s <local-venv> ~/.pyenv/versions/<venv-name>
```

Using this setup the python version will automatically change when navigating into the project
directory.

## Package Management

Python packages can be installed using package managers.
For new projects it is generally recommended to use [uv](#uv), while [pip](#pip) is available by
default in most Python installations and is supported by virtually all Python projects.

### pip

The `pip` package manager is the standard Python package manager and can be used to install, upgrade
and remove Python packages.
It is recommended to use `pip` inside a [virtual environment](#using-virtual-environments).

A package called `<package>` can be installed using

```sh
pip install <package>
```

To upgrade an already installed package run

```sh
pip install --upgrade <package>
```

Packages can be removed again using

```sh
pip uninstall <package>
```

To display all currently installed packages use

```sh
pip list
```

### uv

For new projects it is generally recommended to use [uv](/wiki/programming_language/python/uv.md),
which provides a modern replacement for [pip](#pip), [venv](#venv-virtual-environments) and several
other Python tools.
The corresponding article describes installation, virtual environments, dependency management and
further usage in detail.

## Modules

Python modules can be installed using package managers such as
[pip](/wiki/programming_language/python/pip.md) or
[uv](/wiki/programming_language/python/uv.md).

For most projects it is recommended to use a
[virtual environment](#using-virtual-environments).
For global installation the
[system package managers](/wiki/linux/package_manager.md) can be used.

## Modules

Python modules can be installed using package managers such as
[pip](/wiki/programming_language/python/pip.md) or [uv](/wiki/programming_language/python/uv.md).

For most projects it is recommended to use a [virtual environment](#using-virtual-environments).
For global installation the [system package managers](/wiki/linux/package_manager.md) can be used.
Alternatively pip and uv can also handle global installation.

### scikit-learn

[scikit-learn](https://scikit-learn.org/stable/index.html) is a free and open-source library that
provides various machine learning algorithms.
By default, [scikit-learn](https://scikit-learn.org/stable/index.html) does only utilize the CPU.
This, however, can be easily changed to also utilize the GPU and speed up calculation as explained
in the [corresponding section](#run-scikit-learn-algorithms-on-the-gpu)

#### Run scikit-learn Algorithms on the GPU

Using [cuml](https://docs.rapids.ai/api/cuml/latest/) the GPU can be utilized with only two lines
to add to existing code.
As explained in the
[cuml guide on zero code change acceleration](https://docs.rapids.ai/api/cuml/latest/cuml-accel/)
only the following two lines have to be added to run the scikit-learn algorithms on the GPU.
Additionally, cuml has to be installed using a [Python package manager](#using-virtual-environments)
like [uv](/wiki/programming_language/python/uv.md#installing-packages).
It is important that these lines are put before importing any scikit-learn packages.

```py
import cuml
cuml.accel.install()
```

Afterward all possible scikit-learn algorithms will run on the GPU instead of the CPU.

### PyTorch

This section addresses the [PyTorch module](https://pytorch.org/).
PyTorch is a machine learning resource which is often used for
[neural networks](/wiki/neural_network.md).

#### Setup PyTorch with CUDA for GPU usage

CUDA is also only available for Nvidia GPUs.
For AMD GPUs refer to [the ROCm section](#setup-pytorch-with-rocm-for-gpu-usage).

If you are on Arch Linux or a distribution based on it install
`python-pytorch-cuda` via `pacman -S python-pytorch-cuda`.

After that visit
[the official PyTorch website](https://pytorch.org/get-started/locally) and
install PyTorch for your custom configuration.

After that try to run the following python script.
This should give back `True` if the setup was successful and the GPU is available.

```python
import torch

torch.cuda.is_available()
```

Please note that according to
[various sources](https://www.reddit.com/r/archlinux/comments/1nxipcu/nvidia_pascal/gpu_not_supporting_cuda_13_can_i)
CUDA 13 does not support nVidia Pascal GPUs.
In this case an earlier version of CUDA has to be used.

#### Setup PyTorch with ROCm for GPU usage

ROCm is also only available for AMD GPUs.
For Nvidia GPUs refer to [the CUDA section](#setup-pytorch-with-cuda-for-gpu-usage).

For ROCm to work some system packages have to be installed.
For Linux refer to the
[official Linux guide](https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installrad/native_linux/install-radeon.html).
For [Arch Linux](/wiki/linux/arch-linux.md) although it is not listed the required
[AUR packages](/wiki/linux/package_manager.md) include `rocm-core`, `rocminfo`, `roctracer` and
`rccl`.
For Windows refer to the
[official Windows guide](https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installrad/windows/howto_windows.html).

After the systemwide setup the project environment can be setup.
It is recommended to use [virtual environments](#using-virtual-environments).
The easiest way to achieve ROCm support is by using [pip](#modules).
As explained in the
[official PyTorch guide](https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installrad/native_linux/install-pytorch.html)
the following example can be used inside the environment to install all needed ROCm packages.
In this example the version `7.2` is installed.
Adjustments to the command may have to be done in case another version should be installed.

```sh
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torch-2.9.1%2Brocm7.2.0.lw.git7e1940d4-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torchvision-0.24.0%2Brocm7.2.0.gitb919bd0c-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/triton-3.5.1%2Brocm7.2.0.gita272dfa8-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torchaudio-2.9.0%2Brocm7.2.0.gite3c6ee2b-cp312-cp312-linux_x86_64.whl
pip install \
torch-2.9.1+rocm7.2.0.lw.git7e1940d4-cp312-cp312-linux_x86_64.whl \
torchvision-0.24.0+rocm7.2.0.gitb919bd0c-cp312-cp312-linux_x86_64.whl \
torchaudio-2.9.0+rocm7.2.0.gite3c6ee2b-cp312-cp312-linux_x86_64.whl \
triton-3.5.1+rocm7.2.0.gita272dfa8-cp312-cp312-linux_x86_64.whl
```

If old versions of `torch`, `torchvision`, `torchaudio` or `triton` are installed inside the
environment they may need to be removed.

After this installation for some GPUs – especially integrated GPUs like the Radeon 660M – an
additional step has to be taken.
In this case the following global shell variable has to be set.

```sh
export HSA_OVERRIDE_GFX_VERSION=10.3.0
```

After that try to run the following python script.
Since ROCm uses a bridge to access CUDA it should give back `True` if the setup was successful and
the GPU is available.

```python
import torch

torch.cuda.is_available()
```

### Hailo

The package for [Hailo chips](/wiki/hailo.md) has to be downloaded from the
[official website](https://hailo.ai/developer-zone/software-downloads).
Additional setup may be required as explained in the [Hailo article](/wiki/hailo.md#setup).

Hailo chips can be used to run converted [TensorFlow](#tensorflow) models.
The conversion process is explained in the
[Hailo article](/wiki/hailo.md#preparing-tensorflow-models-for-the-ai-hat).

To run the inference using Python on ARM boards like the
[Raspberry Pi AI Hat +](/wiki/linux/raspberry_pi.md#ai-hat)
[zlodeibaal's article in Medium](https://medium.com/@zlodeibaal/how-to-run-hailo-on-arm-boards-d2ad599311fa)
can be referenced.

### TensorFlow

This section addresses the [TensorFlow module](https://www.tensorflow.org/).
Tensorflos is a machine learning resource which is often used for
[neural networks](/wiki/neural_network.md).

Apart from [package managers](/wiki/linux/package_manager.md) and
[virtual environments](#using-virtual-environments) it can also be set up from source as explained
on [the official website](https://www.tensorflow.org/install/source).
This may especially be useful if specific configurations are needed such as vendor specific GPU
support.

#### Basic Usage of TensorFlow

The basic usage of TensorFlow is described in
[the official guide](https://www.tensorflow.org/guide/keras/serialization_and_saving).
Additionally, it is noted that the dataset may have to be shuffled manually as described in a
[comment by Y. Luo on StackOverflow](https://stackoverflow.com/questions/50184144/shuffle-in-the-model-fit-of-keras).
Finally,
[a Medium blog post](https://medium.com/@danielonugha0/how-to-change-the-learning-rate-of-tensorflow-b5d854819050)
describes how to easily change the learning rate.

#### Combining Models

Models that are normally run in sequence but trained and saved separately can be easily be combined
into a single model.
This can have some advantages, for example when using inteference for deep learning on Edge TPUs
like the [Hailo chips](/wiki/hailo.md) or the [EPS32S3](/wiki/microcontroller.md#esp32).

A simple example for the combination of two models (`model1` and `model2`) into a new model
(`combined_model`) is the following code.

```sh
output = model2(model1.output)
combined_model = tf.keras.models.Model(inputs=model1.input, outputs=output)
```

### matplotlib

The [matplotlib](https://matplotlib.org/) module is a plotting library for Python.

#### Qt Backend Not Found

When using [Wayland](/wiki/linux/wayland.md), matplotlib may fail with an error similar to the
following.

```txt
ImportError: Failed to import any of the following Qt binding modules:
PyQt6, PySide6, PyQt5, PySide2
```

This happens because matplotlib uses the Qt backend (`QtAgg`) for displaying interactive windows,
but no Qt Python bindings are installed.

To fix the issue simply install `PyQt6` inside the current
[Python environment](#using-virtual-environments).
This may look like the following example where [uv](/wiki/programming_language/python/uv.md) is
used but other environments such as [pip](#pip) can also be used.

```sh
uv add PyQt6
```

After installing `PyQt6`, interactive matplotlib windows should work correctly under Wayland.