Cognition Releases FrontierCode 1.1, an Updated Coding Evaluation Benchmark
Cognition, maker of the autonomous AI software engineer Devin, has released FrontierCode 1.1, an update to its internal benchmark for evaluating coding model quality. The revision improves how the benchmark distinguishes legitimate internet use from unfair shortcuts and relaxes grading criteria that had been producing false negatives.
Details
- Better internet-use judging: Refines the methodology for telling legitimate use of the internet, such as documentation lookups, apart from unfair shortcuts like directly retrieving solution code
- Relaxed grading criteria: Loosens 75 overly strict grading rules that were causing false negatives
- Diamond subset retired: The previous Diamond subset has been deprecated as the benchmark’s methodology matures
- New scores published: Adds a new score for Sonnet 5 and an updated score for Fable 5
- Context: As frontier models get increasingly capable at retrieving information, the update aims to keep the benchmark fair while staying realistic about how models are actually used
How to try it
- FrontierCode 1.1 is primarily an internal benchmark Cognition uses to evaluate coding models, not a new user-facing feature of Devin itself
- Individual scores and methodology details are published on Cognition’s official blog