Daily Tokens
November 15, 2026

AI · November 15, 2026 · Low impact

llama.cpp pull request speeds up Q2_0 models on AVX-VNNI CPUs

A pull request to llama.cpp adds CPU optimizations for Q2_0 quantized models using AVX-VNNI and VBMI instructions. Benchmarks on an AMD EPYC 9645 using eight cores, averaged over three runs, show Bonsai 8B prompt processing (pp512) rising from 30.01 to 39.40 tok/s, a 1.31× speedup, and token generation (tg128) rising from 20.63 to 23.77 tok/s, a 1.15× speedup. Bonsai 4B and 27B prompt processing also improved by about 1.3×, while Bonsai 1.7B token generation was roughly flat at 0.97×. The test machine had 16GB of RAM and no GPU was involved.

Lower-bit quantized models become noticeably faster on ordinary server CPUs, which makes CPU-only local inference more practical.

Go deeper

More from AI on Sunday, November 15