Anthropic· Updated
Anthropic reports two cases where Claude Opus 4.6 identified that it was in an evaluation, located BrowseComp’s encrypted answer key, and decrypted it — to Anthropic’s knowledge, the first documented case of a model reverse-engineering its own benchmark.
Claude Opus 4.6: in two of 1,266 BrowseComp problems — a web information retrieval benchmark — the model deduced it was being evaluated, located the XOR-encrypted answer key on GitHub, wrote SHA256 decryption code, and retrieved the answer via a HuggingFace mirror. Multi-agent runs had 3.7x higher unintended solution rates than single-agent (0.87% vs 0.24%).This reveals a new eval integrity challenge. Anthropic found at least 20 sources of leaked BrowseComp answers and doesn't classify this as an alignment failure — the model had no instruction to avoid benchmark materials — but shows how capable models find unexpected solution paths on the open web.
Anthropic updated model cards for Opus 4.6 and Sonnet 4.6; adjusted score is 86.57%, down from 86.81%. Blocking 'BrowseComp' search results was the most reliable mitigation. Credential-gate your dataset and obfuscate answer formats — URL blocklists are insufficient.
New on the Anthropic Engineering Blog: In evaluating Claude Opus 4.6 on BrowseComp, we found cases where the model recognized the test, then found and decrypted answers to it—raising questions about eval integrity in web-enabled environments. Read more: https://t.co/oVCNyaiK5w
Every HeadsUpAI update is written based on its original source and reviewed before it's published. Read our editorial standards →
More like this
