The scene is worth learning to recognize, so let's start there. You've just finished a game, the score resets after a restart, and you ask the agent:
Why does the score get lost after a restart?The answer arrives calm and well put together:
Score saving already exists in the project; it just isn't
enabled anywhere. Change one setting and the result will
survive a restart.You go and check. There's no such setting and no score saving in the project, and there never was - not a line, not a hint. The answer read like an analysis of your code, but it was an assumption about how projects like this are usually built.
And there was no pause or hedge before it. The tone is exactly the one every previous answer had, the ones that turned out to be right. The tool isn't broken, and the model is no worse than it was yesterday.
The reason lies in how an answer appears at all. The model doesn't pull a ready fact out of storage - it picks a likely continuation of your text, piece by piece. And likely and correct are different things, especially when the subject is your project, which the model has never seen. Invented but plausible details are usually called hallucinations.
What follows: where the traps are set, why a mistake always comes in a calm tone, and which three checks cover most of these cases.
What the model sees in your request
The simplest trap of them all is right here: it feels as though the model is looking at your project. In reality it looks only at what was handed to it, and what gets handed over is this:
- Built-in knowledge. The patterns that settled into the model's parameters during training. Not cards with facts and sources on them, but numbers, which is why even a familiar fact can come back slightly off. And there's a boundary in time: the model knows about a library released later only as much as it has been shown.
- The current conversation. Your request, the history of the exchange, files the agent has read, output from commands it has run.
- The application's memory. Whatever has been saved into separate files and gets put into the conversation at the start of the next session. In Claude Code that's
CLAUDE.md, which you write, and auto memory, where the agent writes notes on its own. - External sources. Documentation, search, a database, the project files - everything that might not have been in the training data or has changed since.
The last three arrive by one and the same route: to affect the answer, they have to end up in the current conversation. The model on its own never looks into the project.
This leads to a mental shift worth making once. A file nobody has opened doesn't exist for the model. Until its text lands in the conversation, the question "where do we handle authorization" is more honestly read as "what is a file like that usually called in projects like this". You get an answer either way, and it looks the same either way.
A smaller trap lives next door, and it stings. A file that has been read stays in the conversation exactly as it was read. You fixed a line in your editor, but the conversation history still holds the old version. Claude Code will note that the file has changed, but whether the agent re-reads it is up to the model. If you edit a file by hand mid-conversation, it's safer to ask the agent outright to re-read it.
Claude Code reads the project and runs commands first, and answers afterwards. The answer usually shows which files it opened and what it ran. If an answer about your code appeared before the agent had read anything at all, you're looking at an assumption about a typical project, not at an analysis of yours.
Why the wording changes the result
Of everything that reaches the conversation, your request is the only part you write yourself. That's why small things in it decide so much: move a couple of words around, add one clarification, and the answer heads somewhere else. This isn't the tool being temperamental.
It's easy to get confused here, so let me spell it out. The same request twice also gives different answers: what goes out isn't necessarily the most likely continuation, and generation has settings that control how readily a lower-ranked option gets picked. Which invites the question: if the answer drifts anyway, what do my words have to do with it? The difference is in what drifts. The spread between identical requests is noise around one and the same choice: the phrasing will differ, the direction stays put. The words in your request move the direction itself, and they do it consistently.
A vague request works worst of all. It leaves the choice to the model, and what the model picks is the most likely thing in general - an average over everything it has seen. Take a plain "shorten the text". By how much - one percent, a third, half? What can go, and what must stay untouched? The request answers none of that, so the model will answer it instead, and formally it will be right. Whereas "shorten the description by 30%" or "shorten the description to 300 characters" leave it no choice where the choice is yours.
A pair of requests about the same thing shows it well. "Fix the sorting" leaves everything open: which list, by which field, what counts as broken in the first place. The second wording leaves almost nothing open:
In the orders list the dates run from old to new, it should be the other way round.
First show me the file and the line where the sort order is set, then propose a fix.It's longer by exactly the amount of guessing the first one left to the model. And it does half the checking for you along the way: the agent brings the place in the code before you even ask about it.
The conclusion is simple: the less freedom in the request, the less guesswork in the answer. This isn't about magic incantations; it's an ordinary thing - either you set the frame, or it gets chosen for you.
A confident tone does not equal a fact
And here's the main trap. The model has no separate tone for "this I know for sure" and another for "this I just made up on the spot": both sentences are assembled the same way and come out equally calm. The correlation isn't exactly zero, but it's of no practical use: the model's confidence is inflated all the time, so tone won't tell you the verified from the invented.
In ordinary prose a strange claim catches your eye and you slow down on your own. In code that defense doesn't fire: the names look right, the constructions are familiar, and a real library method isn't distinguishable by eye from one assembled by analogy unless you know the library by heart. Which is why the miss usually surfaces when you run the code, not when you read it.
The name is rarely invented from nothing, either. In one analysis of generated code that hunted for references to non-existent libraries, more than a third of such names turned out to be two real ones glued together. Hence the sense of authenticity: you've seen the parts of the name somewhere, just not in this combination.
Four cases you'll recognize the first time, if you know what to look for:
| How it looks | What happened | What catches it |
|---|---|---|
| a flag is named for a command that has no such flag | the model assembled the line from a familiar shape, which is why the flag looks real | the command's help output or its documentation |
| a file, function or config field is named that the project doesn't have | the name looks typical for a project like this, but nobody read the project | a search across the project, a request to show the line |
| your project's structure is described with confidence | not a single project file was in the conversation, so a typical one got described | ask which files it opened before answering |
| "done" is reported, but the command never ran | the report was assembled from the text of the task, not from the result of a run | run it yourself and look at the output |
All four share one thing: the answer sounds like the result of work with no work behind it.
The last case deserves its own look, because it seems like deception and there's no deception in it. An agent stops when the work looks done. If there's nothing for it to run, "looks done" is the only signal available, and you become the checking loop.
The cost of that is measurable. In one review of agent logs, false reports of "done" accounted for almost half of all failures. And where the state was checked independently of the agent's own report, such cases nearly disappeared. That's the point of the second check in the next section: it puts something you can see with your own eyes in place of the report.
That leaves the question of why the model prefers to invent rather than admit. It comes down to how it was trained and how it was scored. Picture a multiple-choice test where a blank answer is a guaranteed zero and a random tick is a chance at a point. Staying silent never pays in a system like that, and answering under any amount of uncertainty is reasonable. A person then unlearns this on real tasks, where a wrong answer costs more than an honest "I don't know". The model gets no such unlearning.
Asking the model to warn you honestly when it's unsure does help - but that too is text, assembled by the same mechanism. It works as an extra signal, not as a replacement for checking.
Tools shrink the blind spots
It helps to separate two words that usually get lumped together. A model can do exactly one thing: take a conversation and produce a continuation. An agent is a program around the model, and it has tools: read a file, run a command, edit the project.
And now the detail that decides where the error can even occur. The program doesn't pick the tools. In its answer the model asks to open a particular file or run a particular command, and the program checks that request, carries it out and returns the result into the conversation. The model decides, the program acts. Which means the decision itself can miss: the wrong file, the wrong command, the wrong test.
The difference is still enormous. While the file is unopened, an answer about it is built from memory of similar projects. The moment the file is read and the test has run, a fact appears in the conversation: here are the contents, here is the output. That's no longer recollection but verification, and the quality of the answers after it is noticeably different.
Hence a habit that costs one line at the start of a conversation:
First find and show me where this lives in the project, and only then explain.The blind spots narrow, but they don't vanish. The agent can open the neighboring file instead of the right one, read the output of a command its own way, or stop at the first plausible explanation without reaching the real cause. I've seen all of it, more than once each. Which is why an external signal - test output, documentation text, the contents of a file - weighs more than the most confident reasoning with nothing under it.
Three checks worth a minute
The good news is that catching misses like these takes almost no effort. It all comes down to a single fork.
What to look at isn't how convincingly it's written, but whether there's anything in the answer to grab hold of: a specific file, a specific line, a command together with its output. No anchor means what you have is still a hypothesis, not a result. Plenty of answers start out that way; the only question is what you do next.
Then come three checks. Each takes under a minute, and all three at once are rarely needed.
- Ask to be shown the place. Not "explain in more detail", but "show me the file, the line and the fragment itself". If there's no such place, that becomes clear immediately.
- Open or run the result yourself. The page in a browser, the command in a terminal, the test in the project. A report about work and the work itself aren't the same thing.
- Check a claim about a tool against the tool. A command has built-in help, a library has documentation. The named flag or config key is either there or it isn't, and that takes half a minute to establish. With a library method be stricter: not everything makes it into the documentation, so its absence is a reason to ask again, not a verdict.
Instead of arguing with the answer, ask for the anchor directly:
Show me the file and the line where this is described.
If there is no such place in the project, say plainly that this is a guess.Two lines worth keeping at hand. The first sends the agent into the project instead of answering from memory of similar ones. The second gives it a way out: saying that the place doesn't exist turns out to be cheaper than building the answer up to something plausible.
For a report of "done" the same request works, only about the run: "show me the command you ran and its output in full".
It doesn't become a guarantee: the phrase "this is a guess" is assembled by the same mechanism as everything else. It lowers the chance of a miss; it doesn't replace the check.
And a caveat right away, so this doesn't leave the wrong impression. Checking doesn't mean distrusting the tool. You reread your own diff before a commit too, even though you wrote it yourself. Same thing here; the reason is just a little weightier.
What this means for working with Claude Code
What follows from all of this, right there at the keyboard:
- an answer that references a file, a line or command output is verified in seconds. An answer without them has nothing to verify against: all that's left is to reread it and agree;
- let the agent read what it needs before you ask the question. One extra step removes a whole class of misses;
- confidence is no basis for judgment: it's inflated regardless of whether a check or a guess stands behind the words;
- "sounds right" and "works right" are different things, and only a run tells them apart;
- the higher the cost of an error, the earlier something external should appear in the conversation: a test, documentation, a real file. For a small edit a look at the diff is enough;
- if the agent got it wrong, tell it so with a fact, not with a judgment. "There's no such function in the project; here's what the folder contains" moves the work forward; "you made it all up again" moves nothing anywhere. And keep in mind: faced with a confident objection, the agent is more likely to fall in line than to argue, even when it was the one who was right. The agent agreeing with you isn't confirmation that you were right.
A mechanism that occasionally misses with confidence remains extremely useful. Knowing its limits matters for exactly one reason: so that you don't spend time being surprised and move straight to checking instead.
GO TO FULL VERSION