/dev/posts/

llama.cpp quickstart (part 2)

Published:

Updated:

How to quickly use llama.cpp for LLM inference (part 2). This is a follow-up of a previous post on the same topic.

Manual compilation

Compilation with Vulkan support:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
git checkout b10034

sudo apt-get install libvulkan-dev glslc spirv-headers cmake

cmake -B build -DGGML_VULKAN=1 -DLLAMA_OPENSSL=ON -DCMAKE_INSTALL_PREFIX=/opt/llama.cpp-b10034
cmake --build build --config Release

sudo cmake --install build --config Release

addopt /opt/llama.cpp-2026-06-16/

where addopt is a script which sets up your environment variables (PATH, LD_LIBRARY_PATH, etc.).

Here I am compiling with Vulkan support in order to try hardware acceleration on integrated AMD GP. You can adapt your options to enable other hardware backends.

Run models

Download and execute with a single model:

llama-server --api-key-file keys.txt -hf ggml-org/gpt-oss-20b-GGUF
llama-server --api-key-file keys.txt -hf ggml-org/Qwen3.6-35B-A3B-GGUF:Q8_0
llama-server --api-key-file keys.txt -hf ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M
llama-server --api-key-file keys.txt -hf ggml-org/gemma-4-E2B-it-GGUF:Q8_0
llama-server --api-key-file keys.txt -hf ggml-org/gemma-4-E2B-it-GGUF:Q4_0
llama-server --api-key-file keys.txt -hf ggml-org/gemma-4-12B-it-GGUF:Q4_0

Execute in router mode which makes it possible to switch between all the models have downloaded:

llama-server --api-key-file keys.txt --no-models-autoload

Security of llama-server

By default, you might see this message when starting llama-server:

CORS is set to allow all origins ('*') and no API key is set this can be a security risk (cross-origin attacks) more info: https://github.com/ggml-org/llama.cpp/pull/25655

By default,

You might want to use the following flags:

Check disk usage

ncdu ~/.cache/huggingface/hub/

References