The agent may not be lying. By its own check, everything really does add up. It is just that the agent wrote the code, the agent thought up the tests, and the agent is the one who said the final “done.”
It is far too easy for an agent to believe itself
There is a language model inside the agent. Underneath, it continues the text one token at a time. The instructions, the work so far, and the command output make up the context for each new step. That is how an agent can write code, run tests, and decide the job is done. But a coherent sequence of actions is no guarantee of a correct result.
Research has found language models agreeing with users even when the users are wrong, a behavior known as sycophancy. One possible cause is fine-tuning on human ratings: if answers that support a user’s position receive higher ratings, the model can learn to reproduce that behavior.1
We developers add a habit of our own. I have more than once written tests to confirm what I already believed: feed in the expected data, get the expected answer, tick the box. Far less often does a test start with the question, “What should break this?”
When an agent does the same, one mistake travels through the whole job. First it lands in the code. Then in the test, because the test is built on the same assumption. The tests pass, the agent takes that as confirmation and writes “Done.” What actually happened is that it agreed with itself three times.
Telling an agent to check itself again sometimes helps, sometimes changes nothing, and sometimes degrades an answer that was right. Research on intrinsic self-correction in reasoning has found the same limitation: without an external signal, a second look by the same author is not independent evidence.2
A test should hunt for a bug, not for confirmation
Edsger Dijkstra explained in 1969 what tests cannot prove:
Program testing can be used to show the presence of bugs, but never to show their absence.
Edsger Dijkstra, Notes on Structured Programming, EWD249
A red result gives you a specific failure: here is the input, here is the actual behavior, here is what was required. A green result tells you only about the conditions you picked. On this input and in this build, the check found no failure. It says nothing about any other input.
Ten years later, Glenford Myers turned that asymmetry into a working rule. Testing is running a program with the intent of finding errors, and a good test is one with a high probability of finding an as-yet-undiscovered defect.3
What changes from here is not one check but the way the work runs. Written code does not go straight to “done.” First someone makes an honest attempt to break it in a way that can be repeated. If that works, the author gets a counterexample instead of a reviewer’s opinion.
The principle matters especially in agent-driven development. The same agent writes the implementation, picks the checks, reads their output, and draws the conclusion. A mistake in the original assumption can survive all four steps. So checking has to move into a role of its own, where the agent stops defending what it built and starts looking for grounds not to call the change ready.
How Gopnik works
That role belongs to Gopnik, a set of three skills.
gopnik-setup
Gets Gopnik oriented in the project: finds and runs the project's own checks, works out where the finished result gets used, and records reproducible routes for the runs that follow.
gopnik-critic
Tries to refute a claim before it is published: checks it against primary sources, then puts it through an independent check whose task is to prove the opposite.
gopnik
Looks for what can break, attacks the code and the delivered result, then returns a verdict with its evidence and the limits of what was checked.
gopnik-setup runs when you first add it to a project and is not repeated for every task. After that, the skills can be used separately. In our working cycle, the roles break down like this:
One of our working cycles
- Issue from the trackerthe team
- Critique of the framing and the approachgopnik-critic
- Implementationthe main agent
- Attempt to break the changegopnik
- Verdict and evidencegopnik
The last step is the verdict. Gopnik does not stop at a list of the commands it ran: it answers whether the change is ready and attaches the evidence.
NOT READY
Broken: re-running the install leaves a stale file behind.
Revision: 6eafa7d - the re-check produced the same result.
Abridged: the command, file path, and full output are omitted here.
A counterexample means NOT READY. READY appears when the mandatory checks pass, there are no blockers, and the last round of checking covers the exact revision. Gopnik names separately the limits of what it did not check.
What it looked like in practice
On a large internal project where agents were heavily involved in development, one of the first runs with Gopnik turned up four classes of defect. We fixed the bugs and pinned the cases down with tests.
Then we looked at the trend in the Git history.
After that, we revisited the criterion for readiness itself.
What “done” means now
Gopnik does not prove there are no bugs left in the code. What it changes is who gets to say “done.” One green run is no longer enough: the agent has to show how it tried to break the change, which version it checked, and what it did not check.
Next time an agent tells you everything works, do not ask it to check again. Ask how it tried to call its own bluff.
Gopnik is MIT-licensed. The repository has installation instructions for Claude Code and Codex, a description of the three skills, and an example project configuration.
Sources
- Towards Understanding Sycophancy in Language Models, Sharma et al., 2023.
- Large Language Models Cannot Self-Correct Reasoning Yet, Huang et al., 2023.
- The Art of Software Testing, Glenford J. Myers, 1979.
- Notes on Structured Programming, EWD249, Edsger Dijkstra, 1969.