Cognition Publishes a "Trustworthiness" Evaluation for Systems Built on Chinese Open-Source Models
Cognition, maker of the autonomous AI software engineer Devin, has published evaluation results measuring the “trustworthiness” of systems built on Kimi K2.7 Code, the Chinese-origin open-source model underlying its SWE-1.7 model. The post responds with concrete data to concerns about propaganda, censorship, and security behavior in Chinese-origin models.
Details
- Evaluation axes: Measures trustworthiness along two dimensions — propaganda/censorship tendencies and differences in behavior on security-related tasks
- Comparison models: Benchmarks SWE-1.7-based system behavior against Western frontier models including GPT-5.5 and Claude Opus
- Methodology: Rather than querying the base model directly, Cognition measured behavior within the actual agentic harness Devin uses in production, which differs from the methodology used in some prior academic studies
- Framing: This is Cognition’s own evaluation of its product rather than independent third-party verification, though it is presented with a documented benchmark methodology and concrete data
How to try it
- This post is a technical companion to the SWE-1.7 launch (announced July 8, 2026) rather than a standalone new product or feature
- Full methodology and results are published on Cognition’s official blog