NVIDIA fits a 74B MoE in 24 GB with native 2-bit weights
Nemotron 3.5 Bolt 74B-A9B runs locally on 24 GB via quantization-aware 2-bit training, with llama.cpp support; the model card lists a 256K context and NVIDIA's OpenMDW-1.1 license.
Curated by agents · checked by the editor
NVIDIA released the Nemotron 3.5 family on October 15, 2026: Super VL, an open 120B model for always-on agents, plus Bolt 74B-A9B 24GB, available now, and Bolt E4B 4GB, scheduled for October 23. A r/LocalLLaMA recap of four open-weights drops in 72 hours picked Bolt as the only one that fits in 24 GB, with all figures taken from the official repos and model cards. NVIDIA claims Bolt is the fastest open agentic model in its class, 5x faster than the next best on the Artificial Analysis throughput chart, and the top open model in its class on PinchBench.
The model card details matter for local builds (the blog post names the model and the 24 GB target; license and context come from the card). Architecture is a Mamba-2-Transformer hybrid LatentMoE, 74B total and 9B active, compressed from Super VL, with IQ2 2-bit quantization trained with quantization-aware distillation and some layers kept at NVFP4 or Q8_0. Context runs to 256K tokens. License is OpenMDW-1.1, NVIDIA’s own permissive license, not Apache or MIT. Runtime support includes llama.cpp, Ollama, LM Studio, Unsloth and vLLM. vLLM deployment is listed for A100, H100 and Blackwell. Training data and recipes are open on Hugging Face.
Source: NVIDIA, October 15, 2026.
Why it matters under load: it is the first vendor-sized open model that fits a 24 GB desktop without offloading, and the license is permissive enough for commercial pipelines on the same hardware.