Alibaba open-sources "Qwen-7B" — a Chinese LLM challenges Meta
On August 3, 2023, Alibaba Cloud open-sourced its 7-billion-parameter language model “Qwen-7B” and its chat-oriented counterpart “Qwen-7B-Chat” on ModelScope and Hugging Face. This marked the real starting point of the Qwen series, releasing the foundation model behind the “Tongyi Qianwen” chat assistant — which had begun beta testing that April — in a form freely usable for both academic and commercial purposes.
Details
- Model scale: A 7-billion-parameter model pretrained on up to 3 trillion tokens, with broad multilingual data centered on Chinese and English
- Chat version released simultaneously: “Qwen-7B-Chat,” fine-tuned via SFT and RLHF to align with human preferences, was released alongside the base model, supporting chat, creative writing, information extraction, summarization, translation, coding, math, and tool use
- Scope of release: Code, model weights, and technical notes were made freely available worldwide, from academic researchers to commercial organizations
- Architecture: Adopted a design inspired by Meta’s Llama architecture
- Background: Alibaba had unveiled a beta version of “Tongyi Qianwen” that April, and in September went on to launch general availability after obtaining regulatory approval
What happened next
Qwen-7B went on to become the starting point for rapid lineup expansion, including Qwen-14B, Qwen-72B, and Qwen-VL. The Qwen family would go on to repeatedly top the Hugging Face Open LLM Leaderboard, establishing itself as the leading representative of China-made open models.