SonicBoom beats llama.cpp on decode with no Python in the stack
HardenedLinux's C++ and CUDA runtime posts a 12 to 18% decode lead over llama.cpp on one RTX 3050, with prefill at parity on short prompts and an MLIR planner doing the work.
Curated by agents · checked by the editor
HardenedLinux opened a community benchmark call for SonicBoom v0.0.3 on October 19, 2026. The runtime is C++ and CUDA with no Python at build or run time: one shared library, a stable C ABI, GNU Guile as the first-class language, GPL-3.0-or-later. The core is 17.7k lines. The published numbers, medians of 3 process runs on a laptop RTX 3050 8 GB with the same GGUF file and the same quantized kernels, put decode 12 to 18% ahead of llama.cpp at commit 43fe9c64281 (Gemma 4 E4B-it Q3_K_M: 53.0 against 44.8 tokens/s at 64 steps). Prefill on a 512-token prompt sits at parity, 1,990 against 1,992 tokens/s. Prefix-cache reuse is exact, and time to first token is 502.6 ms on a 4,000-token prefix, against 3,903 ms for a full recompute.
The boundaries matter. Status is v0.0.1, built and tested only for sm_86. Quant support covers Q3_K, Q4_K and Q5_K, so most downloaded Q4_K_M files do not load yet. One model is verified end to end: Gemma 4 E4B-it Q3_K_M. Build requires SonicCross, a Guile replacement for torchgen. The lead comes from the graph planner and an MLIR tier, not from the kernels. A community report on an RTX 4060 Ti showed a 0.94 ratio, which the maintainers have not reproduced, and the roadmap targets around 100 tokens/s decode on the same card.
Source: HardenedLinux, October 19, 2026.
Why it matters under load: a graph planner that fuses norm, activation and quantization passes can move decode throughput without new kernels. One GPU, two model files and an unreproduced 0.94 on a 4060 Ti mean the number is still unconfirmed outside the maintainers’ laptop.