llama.cpp quickstart (part 2)

How to quickly use llama.cpp for LLM inference (part 2). This is a follow-up of a previous post on the same topic.

Update 2026-08-14: add some info about using ROCm (for AMD GPUs) and add some benchmarks.

Update 2026-08-18: add benchmark for flash attention.

Table of content

Manual compilation

Compilation with Vulkan (generic GPU) support:

sudo apt-get install libvulkan-dev glslc spirv-headers cmake

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
git checkout b10069

cmake -B build -DGGML_VULKAN=1 -DLLAMA_OPENSSL=ON -DCMAKE_INSTALL_PREFIX=/opt/llama.cpp-b10069
cmake --build build --config Release # -j12

sudo cmake --install build --config Release

addopt /opt/llama.cpp-b10069

where addopt is a script which sets up your environment variables (PATH, LD_LIBRARY_PATH, etc.).

Here I am compiling with Vulkan support in order to try hardware acceleration on integrated AMD GP. You can adapt your options to enable other hardware backends.

Run models

Download and execute with a single model:

llama-server --api-key-file keys.txt -hf ggml-org/gemma-4-E4B-it-GGUF:Q4_0

Execute in router mode which makes it possible to switch between all the models you have downloaded:

llama-server --api-key-file keys.txt --no-models-autoload

Security of llama-server

By default, you might see this message when starting llama-server:

CORS is set to allow all origins ('*') and no API key is set this can be a security risk (cross-origin attacks) more info: https://github.com/ggml-org/llama.cpp/pull/25655

By default,

• requests are allowed without authentication,
• CORS is allowed;
• DNS rebinding attacks is possible,
• tool execution is not enabled.

You might want to use the following flags:

• --api-key or --api-key-file to enable authentication based on API keys;
• --cors-origins localhost to disable CORS from any origin.

Useful parameters

• -dev none, don't offload to GPU

Some small models

Gemma 4:

• ggml-org/gemma-4-E2B-it-GGUF:Q4_0
• ggml-org/gemma-4-E2B-it-GGUF:Q8_0
• unsloth/gemma-4-E2B-it-GGUF:Q4_K_M
• unsloth/gemma-4-E2B-it-GGUF:Q5_K_M
• ggml-org/gemma-4-E4B-it-GGUF:Q4_0
• ggml-org/gemma-4-E4B-it-GGUF:Q8_0
• unsloth/gemma-4-E4B-it-GGUF:Q4_K_M
• unsloth/gemma-4-E4B-it-GGUF:Q5_K_M
• HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive:Q5_K_M
• unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q5_K_M
• TrevorJS/gemma-4-26B-A4B-it-uncensored-GGUF:Q4_K_M

Qwen 3.6:

• ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M
• ggml-org/Qwen3.6-35B-A3B-GGUF:Q8_0

Bonsai:

• prism-ml/Bonsai-1.7B-gguf:Q1_0
• prism-ml/Bonsai-4B-gguf:Q1_0
• prism-ml/Bonsai-8B-gguf:Q1_0

Useful commands

• Check disk usage in HuggingFace cache: ncdu ~/.cache/huggingface/hub/
• Check disk usage in HuggingFace cache: hf cache ls
• see list of models in cache: llama-server -cl
• List available devices: llama-server --list-devices

References

• Build instructions
• ggml.org model collection
  • gpt-oss-20b-GGUF
  • Qwen3.6-35B-A3B-GGUF
  • gemma-4-E2B-it-GGUF
  • gemma-4-12B-it-GGUF
• Prism ML on HuggingFace
• How to Run MTP Models: Multi-Token Prediction Guide

Appendix, Compilation with ROCm (AMD)

For my iGPU (AMD Ryzen 5 5600G, targeting gfx900 architecture):

sudo apt-get install librocm-smi-dev rocm-device-libs-21 hip-utils hipcc libhipblas-dev librocblas-dev

export HIPCXX="$(hipconfig -l)/clang"
export HIP_PATH="$(hipconfig -R)"

cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx900 \
  -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=1 \
  -DLLAMA_OPENSSL=ON \
  -DCMAKE_INSTALL_PREFIX=/opt/llama.cpp-b10069
cmake --build build --config Release # -j12
sudo cmake --install build --config Release

sudo setfacl -m u:$USER:rw /dev/kfd

# Needed at runtime for this GPU:
export HSA_OVERRIDE_GFX_VERSION=9.0.0 # etc.

I do not recommend using ROCm for this (integrated) GPU:

• the performance is not really better than using CPU;
• ROCm is (still) quite unstable on this device (and tends to crash the system/GPU).

Appendix, benchmark

Benchmark on a AMD Ryzen 5 5600G with Radeon Graphics with 40 GiB of RAM (shared with the iGPU), on comparatively small models.

This is a integrated GPU so the result (memory shared with the CPU, somewhat weak GPU) so you should not expect to have similar results with other types of GPUs (such as discrete AMD GPUs). You should reproduce the benchmarks on your system.

Core benchmark

llama-bench -dev none,Vulkan0,Rocm0 -hf "$models" -ngl 5000

Take away:

• using the GPU through ROCm usually did not give me significantly better results than using the CPU;
• using the GPU through the Vulkan backend gives better performance for text ingestion;
• the text generation, the results depend on the model.

MTP benchmark

Here I am checking the impact of using Multi-Token Prediction (MTP) on ggml-org/gemma-4-E4B-it-GGUF:Q8_0.

llama-cli -n 250 --perf -st --warmup --simple-io \
        -hf "$model" -sysf "$prompt_file" -p "$user_prompt" \
        --spec-type none

llama-cli -n 250 --perf -st --warmup --simple-io \
        -hf "$model" -sysf "$prompt_file" -p "$user_prompt" \
        --spec-type draft-mtp --spec-draft-n-max "$draft_n_max"

MTP is only used for text generation to there is no impact on text ingestion.

Take away:

• nice improvement (for text generation) when using MTP.

Given the relatively small output length, the stochasticity of speculation hits, and the noise of these results, this benchmark should be taken with some pinch of salt.

Flash attention

Here I am checking the impact of using flash attention.

llama-bench -hf "$models" -dev none,Vulkan0 -fa off,on -ngl 5000

Take away:

• when using the Vulkan backend, I get a very minor improvement for some models;
• no significant difference in other cases.