Run a Private AI Chatbot on a Raspberry Pi 5 with Ollama

Last updated September 2026. Some links on this page may be affiliate links, which means RPi4U may earn a small commission at no extra cost to you.

By the end of this guide you will have a small AI chatbot running entirely on a Raspberry Pi 5. You can chat with it in the terminal, send it questions from other computers on your network, and optionally give it a web chat page. Everything you type stays on the Pi, so nothing leaves your house and there is no subscription.

We use Ollama, a free tool that downloads open language models and runs them with one command. Set your expectations first: the Pi 5 runs these models on its CPU, so answers appear a few words per second. It is great for learning, drafting short text, and private questions, not for replacing a big cloud assistant.

At a glance

Difficulty	Intermediate
Time	About 45 minutes, plus model downloads
Works on	Raspberry Pi 5 (8GB or more recommended)
Parts cost (approx, US)	About $203 (Pi 5 8GB $175, 27W PSU $12, Active Cooler $6, microSD ~$10); NVMe is optional

What you need

• Raspberry Pi 5, 8GB. The 16GB model lets you try bigger models. A 4GB board only handles the tiniest ones.
• Official 27W USB-C power supply, so the Pi gets full power under heavy load.
• Raspberry Pi Active Cooler. Language models keep all four CPU cores busy for minutes at a time, and without a fan the Pi will slow itself down.
• A good microSD card (32GB or larger, A2 rated), or for faster model loading an M.2 HAT+ with an NVMe SSD (optional).
• A case that fits the cooler (optional).
• Another computer to flash the card and to connect over SSH.
• A home network (Ethernet or Wi-Fi) and an internet connection for downloads.

What to expect from a Pi 5

Ollama supports NVIDIA, AMD and Apple GPUs, plus Vulkan, but the Pi 5 has none of those it can use, so every word is computed on the CPU. In Jeff Geerling’s public benchmarks, a Pi 5 generated about 4.6 tokens per second with Llama 3.2 3B. A token is roughly part of a word, so think “a slow but steady typist.” Smaller 1B models feel noticeably quicker.

What about AI HATs? The original AI HAT+ (Hailo-8 and Hailo-8L) is built for vision tasks like object detection and cannot run language models. Raspberry Pi has since released the AI HAT+ 2, which uses a Hailo-10H chip with 8GB of its own memory and is designed for small generative models. It uses Hailo’s own software and model list rather than the standard Ollama install in this guide, so we don’t cover it here.

Step 1: Flash Raspberry Pi OS Lite and update

Use the Lite (no desktop) version so more RAM is left for the model.

1. Install and open Raspberry Pi Imager on your computer.
2. Choose Raspberry Pi 5 as the device and Raspberry Pi OS Lite (64-bit) as the operating system. The current release is based on Debian Trixie.
3. Pick your microSD card or NVMe drive as the storage.
4. In the customization options, set a hostname (we use pi-ai), a username and password, your Wi-Fi details if needed, and turn on SSH.
5. Write the image, put it in the Pi, and power it on.

After a minute or two, connect from your computer and update everything:

ssh yourname@pi-ai.local
sudo apt update && sudo apt full-upgrade -y
sudo reboot

Step 2: Install Ollama

Ollama’s official Linux script detects the Pi’s 64-bit ARM processor, installs the program, creates an ollama system user, and sets up a background service that starts at boot.

curl -fsSL https://ollama.com/install.sh | sh

When it finishes, it reports that the API is available at 127.0.0.1:11434. Check that the service is running:

systemctl status ollama

Press q to leave the status screen.

Step 3: Pick a model that fits your RAM

A simple rule of thumb: the model needs about its download size in free RAM, plus some room for the conversation and the system. Sizes below are the default versions from the Ollama library.

Model tag	Download	RAM fit	Good for
gemma3:1b	815MB	4GB and up	Fastest replies, simple questions
qwen3.5:0.8b	1.0GB	4GB and up	Tiny but recent all-rounder
llama3.2:1b	1.3GB	4GB and up	Quick chat, summaries
qwen3:1.7b	1.4GB	4GB (tight), 8GB comfortable	Step-by-step reasoning
llama3.2:3b	2.0GB	8GB	Best everyday starting point
phi4-mini	2.5GB	8GB	Math and logic
qwen3:4b	2.5GB	8GB	Better answers, slower
gemma3:4b	3.3GB	8GB	Writing help, can describe images
qwen3:8b	5.2GB	16GB	Highest quality here, very slow

Start with llama3.2:3b on an 8GB Pi, or llama3.2:1b on a 4GB Pi. Qwen3 models “think out loud” before answering, which is interesting to watch but makes replies take longer.

Step 4: Chat with your model

The first run downloads the model, then opens a chat prompt:

ollama run llama3.2:3b

Type a question and press Enter. A few useful commands inside the chat:

• /set verbose shows speed stats after each answer, so you can compare models.
• /? lists all chat commands.
• /bye exits the chat.

Back at the normal prompt, manage your models with these:

ollama list              # models you have downloaded
ollama ps                # models loaded in memory right now
ollama pull gemma3:1b    # download without starting a chat
ollama rm qwen3:8b       # delete a model to free space

Models live in /usr/share/ollama/.ollama/models, so keep an eye on free space with df -h if you collect several.

Step 5: Watch the temperature

Open a second SSH window while the model is answering and run:

watch -n 2 vcgencmd measure_temp

This prints the chip temperature every two seconds (press Ctrl+C to stop). With the Active Cooler fitted, you should hear the fan spin up and the reading should stay well clear of 85°C, the point where the Pi starts slowing itself down to protect the chip. If it climbs near that, check the fan cable is plugged in and give the case some airflow.

Step 6: Use it from other devices

Ollama has a REST API on port 11434. On the Pi itself you can test it right away. Setting "stream": false returns the whole answer at once instead of word by word:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Give me three names for a garden robot.",
  "stream": false
}'

By default Ollama only listens to the Pi itself. To let other computers on your network reach it, add a setting to the service:

sudo systemctl edit ollama

An editor opens. In the space near the top marked for your edits, add these two lines, then save and exit (Ctrl+O, Enter, Ctrl+X in nano):

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Apply the change:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Now, from a laptop on the same network, run the same curl command but replace localhost with pi-ai.local (or the Pi’s IP address). Apps that support Ollama can point at http://pi-ai.local:11434 too.

Security note: the API has no password. Only do this on a home network you trust, and never forward port 11434 on your router to the internet.

Step 7 (optional): Add a web chat page with Open WebUI

Open WebUI gives you a ChatGPT-style page in the browser, with chat history and a model picker. It runs in Docker and supports 64-bit Pi boards. Install Docker first using Docker’s Debian guide, finish Step 6 so the container can reach Ollama, then start Open WebUI (pick your own secret key):

sudo docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data -e WEBUI_SECRET_KEY=your-secret-key --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Open http://pi-ai.local:3000 in a browser and create an account to start chatting. Open WebUI uses RAM too, so stick to the smaller models if you run both on an 8GB Pi.

Troubleshooting

• Ollama reports a memory error or the Pi freezes: the model is too big. Remove it with ollama rm and pick a smaller one from the table.
• Answers get slower over time: the Pi is probably throttling from heat. Check vcgencmd measure_temp and make sure the Active Cooler fan is connected.
• The first reply takes a long time: the model is loading from storage into RAM. Later replies are faster. An NVMe SSD shortens this wait.
• Other devices get “connection refused”: recheck the override with systemctl cat ollama, then run the reload and restart commands again.
• The installer stops with an error: confirm you flashed the 64-bit OS with uname -m. It should print aarch64.

Where to go next

• Pair your chatbot with vision: our AI camera object detection guide uses the Raspberry Pi AI HAT+ to spot objects in real time.
• Want more RAM for bigger models? Compare boards in our Raspberry Pi buying guide or answer a few questions in the Pi picker.
• Keep your home network private in other ways with our Pi-hole ad blocker tutorial.

Sources: Ollama Linux docs, Ollama FAQ, Ollama CLI reference, Ollama API: generate, Ollama hardware support, Ollama model library, geerlingguy/ai-benchmarks, Raspberry Pi OS documentation, Raspberry Pi hardware documentation, Raspberry Pi Active Cooler, Raspberry Pi AI HAT+ 2 announcement, Open WebUI quick start.