LLM Laboratory

28.08.2025 · llm, software

Compilation PyTorch BitsAndBytes for CUDA 11.4

Date: 28.08.2025

Compilation PyTorch BitsAndBytes for CUDA 11.4 (Kepler)

photo

Table of Contents

Overview

Requirements

  • Ubuntu 20.04
  • This GPU is outdated; most tools must be built from source
  • PyTorch 2.2.0
  • Python 3.8

Test environment

  • Workstation 40 GB RAM, 200GB SSD, 750W Power supply
  • Ubuntu 20.04
  • Install python 3.8
  • NVIDIA Tesla K80

My test environment: HP Z440 + NVIDIA Tesla K80

Instructions

Ubuntu preparation

sudo apt dist-upgrade
sudo reboot

Driver setup and tools preparation

  • Install drivers nvidia-driver-470 and tools
sudo apt install nvidia-driver-470 clinfo cmake-mozilla python3.8-venv python3.8-dev git
sudo reboot
  • Install CUDA 11.4
wget https://developer.download.nvidia.com/compute/cuda/11.4.4/local_installers/cuda_11.4.4_470.82.01_linux.run
sudo sh cuda_11.4.4_470.82.01_linux.run --toolkit --samples

echo 'export PATH=/usr/local/cuda-11.4/bin:$PATH' | sudo tee /etc/profile.d/cuda.sh
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-11.4/lib64:$LD_LIBRARY_PATH' | sudo tee -a /etc/profile.d/cuda.sh
echo 'export CUDA_HOME=/usr/local/cuda-11.4' | sudo tee -a /etc/profile.d/cuda.sh
source /etc/profile.d/cuda.sh
nvcc --version
  • Check nvidia driver installation
nvidia-smi
clinfo

Build PyTorch

  • Prepare python environment
mkdir -p ~/llm && cd ~/llm
python3 -m venv .venv_llm
source ./.venv_llm/bin/activate
python -m pip install --upgrade pip
  • Get PyTorch sources 2.2.0
git clone -b release/2.2 https://github.com/pytorch/pytorch.git
cd ./pytorch
  • Compile PyTorch and install to virtualenv
pip install -r requirements.txt
USE_CUDA=1 python setup.py install
  • Check PyTorch installation
cd ~/llm
python3 -c "import torch; print(torch.__version__); print(torch.cuda.is_available());print(torch.cuda.get_device_name(0));"

Build BitsAndBytes

  • Get BitsAndBytes sources 0.44.1
git clone -b 0.44.1 https://github.com/bitsandbytes-foundation/bitsandbytes.git
cd ./bitsandbytes
  • Create custom requirements-cus.txt
cat <<EOF >> requirements-cus.txt
# Requirements used for local development
setuptools>=63
pytest~=8.3.3
einops~=0.8.0
wheel~=0.44.0
lion-pytorch~=0.2.2
scipy~=1.10.1
pandas~=2.0.2
matplotlib~=3.7.5
EOF
  • Compile BitsAndBytes and install to virtualenv
pip install -r requirements-cus.txt
cmake -DNO_CUBLASLT=true -DCOMPUTE_BACKEND=cuda -S .
make
pip install .
  • Check PyTorch installation
cd ~/llm
python -m bitsandbytes

It works!