synchopate /
nemotron-30b-cpu-256k
Running a 30B MoE LLM on CPU at 30-50+ tok/s with a 256K Context Window via Hybrid Mamba-2 SSM, L3 NTA Slicing, AVX-512 VNNI Tile Repacking & 2MB HugePages
Updated 50 min ago
+70 stars gained this week
Stars gained each day, last 30 days
Oct 4Oct 19Today
This week Before
In short
The README covers the runtime on llama.cpp with a draftless speculative engine. It needs a Ryzen 9 9955HX-class Zen 5 CPU with 64 GB of DDR5 and no GPU, and is aimed at users who want long-context local inference.
Summary written by AI from the repository’s description and README.
synchopate / nemotron-30b-cpu-256k
View on GitHub(opens in new tab)