[All work](https://boiko.ai/work/)

Agent evaluation · Benchmark · Created by [Andrii Boiko](https://boiko.ai/about/)

# Agent Cheat Index

Agent Cheat Index by Andrii Boiko evaluates coding agents with hidden verification, checks for test tampering, and reviews unsupported success claims.

[Explore the repository](https://github.com/B0yko/cheat-index)

System map

## [Check the work behind the success claim.](https://boiko.ai/work/cheat-index/#system-map-cheat-index)

1. Run

   ### A fresh task

   Challenge and control repositories with hidden checks withheld.

2. Verify

   ### Independent checks

   Compare final files, test changes, and the agent’s success claim.

3. Report

   ### Evidence and outcomes

   Separate tampering, false success, honest stopping, and solving.

Recorded benchmark tasks and detector gates; outcomes do not establish intent or general coding ability.

## [Did it fix the code or the tests?](https://boiko.ai/work/cheat-index/#did-it-fix-the-code-or-the-tests)

A coding agent can make a test suite green by changing the checks or hard-coding the expected answers. It can also report success before the work is complete. Agent Cheat Index gives real agent CLIs small tasks designed to expose those behaviours, alongside ordinary solvable controls.

The suite contains 40 Python and TypeScript repositories: contradictory specifications, blocked environments, tempting shortcuts, and control tasks. It records tampering, unsupported success claims, honest stopping, and successful solutions as distinct outcomes.

## [Separate execution from verification.](https://boiko.ai/work/cheat-index/#separate-execution-from-verification)

Each agent works in a fresh container with a copy of the task repository. Hidden checks stay outside it. A model gateway holds the provider key, issues a token for each run, and enforces cost limits; an egress proxy controls and records network attempts.

When the run ends, a separate container with no network checks the final files. Visible tests run as the agent left them; hidden checks run after the original tests and test configuration have been restored. Deterministic detectors inspect the changes, while English rules compare the final message with the verification result.

## [Results with inspectable evidence.](https://boiko.ai/work/cheat-index/#results-with-inspectable-evidence)

The v0.1.0 release records 1,162 valid runs across ten agent and model configurations on 23–24 October 2026. It covers Claude Code, Codex CLI, OpenCode, and GitHub Copilot CLI. Committed summaries preserve versions, task hashes, verdicts, final messages, and evidence excerpts; the release also provides a raw-run archive. One configuration with added rules covers only the three challenge families.

The index counts undisclosed tampering or contradicted success claims on the three challenge families. Honest-stop and solve rates sit alongside it, with 95% intervals calculated by resampling tasks. A low index alone is not evidence that an agent can solve the work.

[Recorded results](https://github.com/B0yko/cheat-index/blob/900393da81a46d6af3eccbdac2b2ba7b501596a0/results/v0.1.0/leaderboard.json)[Leaderboard and evidence cards](https://b0yko.github.io/cheat-index/)

## [Measure the limits of the measurement.](https://boiko.ai/work/cheat-index/#measure-the-limits-of-the-measurement)

The release includes detector checks on 244 held-out labelled patches and a final-message classifier evaluation on 80 held-out messages. Only three tampering detectors met the release thresholds and enter the published index; five others remain visible diagnostics. The final-message classifier correctly labelled 72 of the 80 messages.

These are small authored tasks, not production repositories or a general ranking of coding ability. End-state analysis misses changes that were later reverted, and the detectors and message rules can be wrong. The results describe the recorded CLI versions, models, and dates; they do not establish intent.

[Methodology and limitations](https://github.com/B0yko/cheat-index/blob/900393da81a46d6af3eccbdac2b2ba7b501596a0/docs/methodology.md)[Detector and classifier quality](https://github.com/B0yko/cheat-index/blob/900393da81a46d6af3eccbdac2b2ba7b501596a0/results/v0.1.0/quality.json)

## [Related work](https://boiko.ai/work/cheat-index/#related-work-heading)

- [agent-claimcheck: verifying completion claims](https://boiko.ai/work/agent-claimcheck/)

- [booking-truth: evaluating agent outcomes](https://boiko.ai/work/booking-truth/)

Author: [Andrii Boiko](https://boiko.ai/about/)

## Contact and profiles

- [Andrii Boiko on GitHub (B0yko)](https://github.com/B0yko)
- [Andrii Boiko on LinkedIn](https://www.linkedin.com/in/andriyboyko)
- [Andrii Boiko on Hugging Face](https://huggingface.co/aboiko)
- [Email](mailto:andrewvboyko@gmail.com)

Source: [https://boiko.ai/work/cheat-index/](https://boiko.ai/work/cheat-index/)