Newcomer Mistral AI unveils "Mistral 7B" — enters the scene by outperforming much larger models

On September 27, 2023, French startup Mistral AI announced its first in-house model, “Mistral 7B.” Billed as “the most powerful language model for its size to date,” it made waves by outperforming Llama 2 13B on every benchmark and beating the far larger Llama 1 34B on many metrics as well — all released under the unrestricted Apache 2.0 license.

Details

• Performance: Despite having just 7.3 billion parameters, it outperformed Llama 2 13B on every benchmark, approached Code Llama 7B on code generation, and maintained strong performance on English-language tasks
• Technical features: Adopted Grouped-Query Attention (GQA) for faster inference and Sliding Window Attention (SWA) to handle longer sequences with less compute
• License: Released under the unrestricted Apache 2.0 license, allowing free use, modification, and commercial use — a bold choice to make the company’s effective debut work fully open
• Distribution: Model weights were published on GitHub and Hugging Face, letting anyone download and try it immediately

What happened next

The strength of this debut model earned Mistral AI rapid success in raising large funding rounds. That December, the company unveiled the mixture-of-experts model “Mixtral 8x7B,” cementing its position at the forefront of the open-model movement.