Anbeeld's BeeLlama.cpp

BeeLlama.cpp (or just Bee) is a performance-focused llama.cpp fork for squeezing more speed and context out of local GGUF inference. It adds variance-normalized KV-cache quantization (KVarN), KV cache precision tail for recent tokens, low-bit cache types, adaptive draft control for speculative decoding, reasoning-loop protection, and more.

Not quite a pegasus, but close enough.

• Variance-normalized KV-cache quantization (KVarN): provides higher precision at similar memory costs. Independent K and V bit widths at kvarn2, kvarn3, kvarn4, kvarn5, kvarn6, and kvarn8, set with --cache-type-k and --cache-type-v.
• KV cache precision tail: keep most of the KV cache quantized while storing recent tokens in F16/BF16, enabled with --kv-tail-tokens. A single global softmax merges the quantized body and the precision tail under FlashAttention, without materializing the whole cache.
• Standard low-bit KV cache types: q2_0, q2_1, q3_0, q3_1, q6_0, and q6_1, usable for either target or draft caches alongside the upstream q4/q5/q8 types.
• Adaptive draft-max for DFlash: adjusts the active DFlash draft horizon at runtime instead of using a fixed --spec-draft-n-max, comparing speculative throughput against a no-spec baseline.
• Reasoning-loop protection: the server detects repeated hidden reasoning output and intervenes.

For the full feature and public-repo comparison, read docs/beellama-features.md. For the complete argument reference, read docs/beellama-args.md.

K and V cache types are set independently with --cache-type-k and --cache-type-v. The research is covered in articles: KVarN KV Cache: Implementation and Benchmarks, KV Cache Precision Tail: Implementation and Benchmarks, and KV Cache Quantization Benchmarks: KVarN, Precision Tail.

The measurements come from Qwen 3.6 27B Q5_K_S at 64K context on Wikitext-2 raw with -b 2048 -ub 512 on an RTX 3090. Median KLD is the primary quality metric; lower is better.

A 1024-token tail is a useful starting point for low-bit Qwen caches. Prefer more body precision when old and recent tokens matter equally; consider 2048 only when the newest two thousand tokens are genuinely the privileged working set. Standard caches generally have slightly faster prefill.

Gemma 4 needs a separate decision. Its 1024-token sliding window makes a 1024 tail exact across most layers, sharply changing memory and throughput. Standard q8_0 / q8_0 without a tail is the safer Gemma default when throughput and older-context coverage matter, and the benchmark results recommend avoiding quantized KV cache when Gemma quality is non-negotiable.

Use this generic fallback ladder when KVarN or precision tails are unavailable. It is based on the same Qwen 3.6 27B benchmark, with every standard cache row using tail 0.

Current release binaries are on the releases page.

Windows CUDA archives contain a ggml-cuda.dll backend; download the matching beellama-<version>-cudart-win-cuda-*-x64.zip runtime archive and extract it into the same folder. Windows SYCL and HIP archives ship as standalone packages with all required runtime DLLs bundled.

Docker images are published to ghcr.io/anbeeld/beellama.cpp:

Building from source with -DGGML_NATIVE=ON may result in a tiny bit better performance, so it might still be a good idea to do that if/when you decide to use this fork long-term.

# Linux (GCC + CUDA)
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON \
  -DGGML_CUDA_FA=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

# Windows (MSVC + CUDA)
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON ^
  -DGGML_CUDA_FA=ON ^
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release --parallel

# macOS (Metal)
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

The default CUDA FlashAttention build covers 50 standard cache pairs and 15 KVarN fast-decode pairs, including the homogeneous F16 and BF16 pairs needed by precision tails. Add -DGGML_CUDA_FA_ALL_QUANTS=ON to compile all 169 standard and 36 KVarN pairs, or -DGGML_CUDA_KVARN=OFF to build without KVarN kernels.

Bee inherits llama.cpp backend support, including Metal, HIP, Vulkan, SYCL, BLAS, CANN, MUSA, OpenVINO, OpenCL, and RPC. Use the upstream-style build docs in docs/build.md and backend-specific pages under docs/backend.

llama-cli -m model.gguf
llama-cli -m model.gguf -cnv --chat-template chatml
llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p "Request: schedule a call at 8pm; Command:"

llama-server -m model.gguf --port 8080
llama-server -m model.gguf -c 16384 -np 4
llama-server -m model.gguf -md draft.gguf

llama-server -m target.gguf --spec-type draft-dflash \
  --spec-draft-model drafter.gguf \
  --spec-draft-ngl all \
  --spec-dm-controller profit \
  --flash-attn on --cache-type-k q5_0 --cache-type-v q4_1

Keep the draft context on a standard cache type; KVarN is target-cache only.

# Balanced general starting point from the benchmark ladder
llama-server -m model.gguf --flash-attn on \
  --cache-type-k kvarn5 --cache-type-v kvarn4 \
  --kv-tail-tokens 1024

llama-server --models-dir /path/to/models
llama-server --models-preset presets.ini

• BeeLlama features and public repo diff
• BeeLlama args reference
• Build docs
• Server docs
• Docker docs
• Performance troubleshooting

Keep PRs small and scoped. Run the narrowest relevant tests or benchmarks before opening a PR, and include the exact commands. For fork-specific changes, update the corresponding docs when behavior or args change.

Read CONTRIBUTING.md for inherited llama.cpp contribution conventions and this fork's AI usage policy.

• yhirose/cpp-httplib - HTTP client/server library used by llama-server - MIT
• stb-image - single-header image decoder used by multimodal code - public domain
• nlohmann/json - single-header JSON library - MIT
• miniaudio.h - single-header audio decoder - public domain
• subprocess.h - process launching helper - public domain
• Intel OpenVINO - frontend header used in OpenVINO backend (ggml/src/ggml-openvino/openvino/frontend.h) - Apache-2.0
• Intel SYCL/oneAPI - SYCL backend (ggml/src/ggml-sycl/) - Apache-2.0 WITH LLVM-exception

See the licenses/ directory for full license texts.