Meta unveils "Llama 4" — first MoE architecture, Scout and Maverick released open
On April 5, 2025, Meta announced its next-generation model series, “Llama 4.” Marking the Llama family’s first use of a Mixture-of-Experts (MoE) architecture, the natively multimodal “Llama 4 Scout” and “Llama 4 Maverick” were released with open weights. A larger model, “Behemoth,” was also previewed.
Details
- Llama 4 Scout: 17 billion active parameters across 16 experts. Fits on a single NVIDIA H100 GPU while offering, at the time, one of the largest context windows among openly available models at 10 million tokens
- Llama 4 Maverick: 17 billion active parameters across 128 experts. Meta claimed it outperforms GPT-4o and Gemini 2.0 Flash across a wide range of benchmarks
- Behemoth (previewed): An MoE model with 288 billion active parameters and roughly 2 trillion total parameters, positioned as a “teacher model” still in training and not yet released at this point
- License: Released under the Meta Llama license, allowing commercial use for many companies, and made available via Hugging Face and other platforms
- First MoE adoption: Marked a shift away from the dense Transformer designs used previously, becoming the first model group in the Llama series to adopt an MoE architecture
What happened next
Training on Behemoth continued afterward, with its release repeatedly delayed throughout 2025. Llama 4 was also reported to have received a somewhat muted market response, and heading into 2026 Meta began shifting its focus to a new model line separate from Llama, the “Muse” series.