TLDR: +1 for pi with llama.cpp on debian

Agentic coding with local LLMs

Seize the means of computation
— someone selling you expensive computers

I finally bought some hardware that can run local models.

BEHOLD, 64GB of integrated memory:

$ fastfetch -l none | rg 'OS|Host|CPU|GPU|Memory'
OS: Debian GNU/Linux 13 (trixie) x86_64
Host: Desktop (AMD Ryzen AI Max 300 Series) (A4)
CPU: AMD RYZEN AI MAX+ 395 (32) @ 5.18 GHz
GPU: AMD Radeon Graphics / Radeon 8050S Graphics / Radeon 8060S Graphics [Integrated]
Memory: 55.25 GiB / 62.09 GiB (89%)

So, how to use it?

Install Debian
Install llama.cpp
Install pi
Further Reading

1. Install Debian

We need to install a new OS.

I recommend Debian, since it’s open-source, stable, uses apt, and is widely supported.

You can setup a bootable USB like so:

wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-13.6.0-amd64-netinst.iso

Then find your USB disk:

sudo fdisk -l

And, after double checking to appease the dd gods, create the USB like so:

sudo dd if=debian-13.6.0-amd64-netinst.iso of=/dev/sda bs=16M status=progress oflag=sync

Plug that in, restart your box, furiously slam F12 or F10 during boot, then finish the install.

2. Install llama.cpp

llama.cpp is an open-source project we’ll use to run our models locally.

Install

llama.cpp isn’t available in stable Debian, so we build from source:

# personal preference: ~/bin/vendor is in my $PATH, ~/code/forks holds forks
mkdir -p ~/bin/vendor ~/code/forks
export PATH="$PATH:$HOME/bin:$HOME/bin/vendor"

# install pre-requisites
sudo apt update -y
sudo apt install -y cmake ccache build-essential git libopenblas-dev pkg-config libcurl4-openssl-dev libvulkan-dev glslc spirv-headers

# clone
git clone https://github.com/ggml-org/llama.cpp ~/code/forks/llama.cpp
cd ~/code/forks/llama.cpp

# build
cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DGGML_VULKAN=ON
cmake --build build --config Release -j $(nproc)

# symlink the binaries to somewhere in your $PATH
ln -sfv $(realpath build/bin/llama-server) ~/bin/vendor/llama-server
ln -sfv $(realpath build/bin/llama-bench) ~/bin/vendor/llama-bench

Afterwards, you should have llama-server available:

$ llama-server --version
version: 0.4.0-dev (build 10821, commit 51476e0be)
built with GNU 14.2.0 for Linux x86_64

Configure

We’ll set it up to automatically run on boot by using a systemd service.

First, create the service file:

# ~/.config/systemd/user/llama-server.service

[Unit]
Description=llama.cpp local LLM server
After=network.target

[Service]
Type=simple
ExecStart=%h/bin/vendor/llama-server \
  --offline \
  --no-models-autoload \
  --jinja \
  --host 127.0.0.1 \
  --port 8080 \
  --models-preset %h/.config/llama.cpp/models.ini
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target

Then configure it to start automatically:

systemctl --user enable --now llama-server.service

Check status:

systemctl --user status llama-server.service

View logs:

journalctl --user -u llama-server.service --all --lines=40 -f

Restart the server (e.g, after changing model options):

systemctl --user restart llama-server.service

And here’s a minimal models.ini:

# ~/.config/llama.cpp/models.ini

version = 1

[*]
ctx-size = 0
flash-attn = on
threads = 8
threads-batch = 16
batch-size = 1024
ubatch-size = 128
reasoning = auto
reasoning-preserve = true
image-min-tokens = 1024
no-cache-idle-slots = true

3. Install pi

pi is an open-source coding harness.

Install like so:

npm install -g --ignore-scripts @earendil-works/pi-coding-agent

Open pi:

$ pi

 pi v0.85.1
 escape interrupt · ctrl+c/ctrl+d clear/exit · / commands · ! bash · ctrl+o more
 Press ctrl+o to show full startup help and loaded resources.

 Pi can explain its own features and look up its docs. Ask it how to use or extend Pi.

Then configure it to use your local llama.cpp:

/login llama.cpp

And install and load some models:

/llama

Select Download model... and enter this one:

ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M

After it downloads, you can finally select the model

/model