llama.cpp quickstart (part 2)

Published: Jul 16 2026

Updated: Jul 20 2026

How to quickly use llama.cpp for LLM inference (part 2). This is a follow-up of a previous post on the same topic.

Manual compilation

Compilation with Vulkan support:

sudo apt-get install libvulkan-dev glslc spirv-headers cmake

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
git checkout b10069

cmake -B build -DGGML_VULKAN=1 -DLLAMA_OPENSSL=ON -DCMAKE_INSTALL_PREFIX=/opt/llama.cpp-b10069
cmake --build build --config Release

sudo cmake --install build --config Release

addopt /opt/llama.cpp-b10069

where addopt is a script which sets up your environment variables (PATH, LD_LIBRARY_PATH, etc.).

Here I am compiling with Vulkan support in order to try hardware acceleration on integrated AMD GP. You can adapt your options to enable other hardware backends.

Run models

Download and execute with a single model:

llama-server --api-key-file keys.txt -hf ggml-org/gpt-oss-20b-GGUF

llama-server --api-key-file keys.txt -hf ggml-org/Qwen3.6-35B-A3B-GGUF:Q8_0
llama-server --api-key-file keys.txt -hf ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M

llama-server --api-key-file keys.txt -hf ggml-org/gemma-4-E2B-it-GGUF:Q8_0
llama-server --api-key-file keys.txt -hf ggml-org/gemma-4-E2B-it-GGUF:Q4_0

llama-server --api-key-file keys.txt -hf ggml-org/gemma-4-E4B-it-GGUF:Q4_0
llama-server --api-key-file keys.txt -hf ggml-org/gemma-4-E4B-it-GGUF:Q8_0

llama-server --api-key-file keys.txt -hf ggml-org/gemma-4-12B-it-GGUF:Q4_0

llama-server --api-key-file keys.txt -hf prism-ml/Bonsai-8B-gguf:Q1_0
llama-server --api-key-file keys.txt -hf prism-ml/Bonsai-4B-gguf:Q1_0
llama-server --api-key-file keys.txt -hf prism-ml/Bonsai-1.7B-gguf:Q1_0

llama-server --api-key-file keys.txt -hf a-m-team/AM-Thinking-v1-gguf:Q2_K

Execute in router mode which makes it possible to switch between all the models you have downloaded:

llama-server --api-key-file keys.txt --no-models-autoload

Security of llama-server

By default, you might see this message when starting llama-server:

CORS is set to allow all origins ('*') and no API key is set this can be a security risk (cross-origin attacks) more info: https://github.com/ggml-org/llama.cpp/pull/25655

By default,

requests are allowed without authentication,
CORS is allowed;
DNS rebinding attacks is possible,
tool execution is not enabled.

You might want to use the following flags:

--api-key or --api-key-file to enable authentication based on API keys;
--cors-origins localhost to disable CORS from any origin.

Useful parameters

-dev none, don't offload to GPU

Useful commands

Check disk usage: ncdu ~/.cache/huggingface/hub/
see list of models in cache: llama-server -cl
List available devices: llama-server --list-devices