How to prove a fix works

The scoring bug is fixed. The reply contains terminal output, a screenshot of the scoreboard and a short "done". Plenty of files, and a simple question: which of them actually answers the task's requirement?

Evidence isn't just any file that happens to sit next to a fix. It's the result of a specific check, one that shows what was checked, on which state of the code, and what came out. Different materials answer different questions, and no single one answers them all.

We'll look at how to turn a requirement into a testable claim, what to record about a command run, and what a log, a diff and a screen capture each confirm. Then we'll put it all into a short EVIDENCE_LOG with six fields and check whether a ready-made set of materials is enough.

The criterion defines the check

One example runs through the whole article: a small bonus game with a score on a scoreboard. The score starts at 0, three matches should add 3 points, and before the fix they added 30. The scoring rule was made up for this game. In another project it could be anything at all, so don't carry it over.

You can't check the phrase "the score is fixed". It's unclear what to run and what to look at. A testable claim has three parts: the starting state, the action and the expected observation. That gives you "with the score at 0, three matches add 3 points, and the scoreboard shows 3". Now the claim has a check.

Next, that claim breaks down into separate observations. The calculation and what the player sees are different things, and different evidence confirms each of them. The third row of the table is about scoring on the leaderboard: the task may not have that requirement at all, but if it does, it needs an observation of its own.

The third column is the most useful one here. Evidence almost always confirms less than you'd like, and the boundary lies wherever the action you actually performed ends. In our example the calculation test doesn't check the scoreboard, and the scoreboard capture doesn't show the changed line of code.

And keep two things apart: the requirement is what has to be true at the end, and the result is what you actually got. When they get mixed up in one paragraph, "should add 3" quietly turns into "adds 3", even though nobody ran anything.

The command and its result

Five things describe a check run. None of them is obvious to whoever repeats the check a week later:

• the folder the command was run from;
• the command itself, word for word;
• the state of the code: before the edit, after the edit, and with what uncommitted changes;
• how it ended: the exit code and the summary counters;
• which checks actually ran and which didn't.

It's also worth distinguishing between "you need to run npm test" and "I ran npm test, here's the output". The first is an instruction, a plan for later. The second is a fact. This is where it's easy to pass off an intention as a result, especially when the reply neatly lays out what ought to be checked.

npm test doesn't run "all the tests there are". It runs whatever is in the test field of the scripts section in a specific package.json, the one belonging to the project npm picked. To avoid guessing which project that is, run the command from its root and write that folder down in the evidence log. In another project the same name may hide a different tool, a different set of steps or nothing at all. That happens too.

A command ends with an exit code: 0 usually means "the tool considers the run successful". In bash you see it with echo $? right after the command; in PowerShell, for an external program like npm, it's $LASTEXITCODE. The catch is that zero doesn't tell you the test you care about actually ran: it could have been skipped, filtered out by name or never written. So next to the exit code you always need the counters and the name of the test you're interested in.

What to keep from the log

Run output can be long, and nobody copies all of it into the evidence log. A short excerpt gets four things: the name of the failed test, the expected and actual values, the reason for the failure and the summary counters. The rest lives in a file you link to. Below is a shortened example output made up for this article, not a record of an actual run.

✖ three matches give 3 points (2.4108ms)
  AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
  30 !== 3
      at TestContext.<anonymous> (src/bonus-game/score.test.js:8:10)
ℹ tests 1
ℹ pass 0
ℹ fail 1
ℹ skipped 0

This is an excerpt; the full output is saved to a file, say logs/test-fail.txt. The excerpt already shows the main points: it's the three-matches test that failed, 3 was expected, 30 came back, and no checks were skipped.

Raw output goes into the evidence log as is. A summary in your own words ("the test failed because of a wrong multiplier") reads more clearly, but it's already a conclusion rather than an observation, and nobody can verify it from the log. Keep the two side by side: first the tool's lines, then your explanation.

The location in the stack trace line is where the assertion fired, i.e. the test file. You look for the cause in the game code, and the multiplier you find there is a result of reading the code, not a line from the log. In the evidence log, label these two sources differently; otherwise later on nobody can tell where the number 10 came from.

What a diff confirms

A diff shows the difference between two states of a file. It doesn't tell you the code was run even once after the edit, let alone that the behavior changed the way you expected.

--- a/src/bonus-game/score.js
+++ b/src/bonus-game/score.js
@@ -1,5 +1,5 @@
-const POINTS_PER_MATCH = 10;
+const POINTS_PER_MATCH = 1;

 function calcScore(matches) {
   return matches * POINTS_PER_MATCH;
 }

An excerpt like this needs a one-line caption underneath: which states were compared. For example, "working tree against commit a1b2c3d before the edit, command git diff HEAD"; the identifier here is made up, and your log needs your real one. This command compares the current tracked files with HEAD, staged changes included. Plain git diff without HEAD shows only the difference from the staged state and may already be empty after git add (a classic way to confuse yourself). Without the caption a diff reads as "something compared with something", and nobody can repeat the comparison.

Also check that the test's expectation stayed the same - 3. If the expected number changed along with the code, that's a different story: the check adjusted to the behavior instead of the behavior adjusting to the requirement.

Next comes a small technical detail that's easy to get wrong. While the edit is uncommitted, you see it when you compare the working tree with the last commit. After the commit the same comparison shows nothing, and that doesn't mean nothing changed in the task. The change just moved into history, and now you need to compare the commit with the previous state. An empty diff proves nothing on its own; what matters is which two states you compare.

What a screenshot shows

State and action

A capture with 3 on the scoreboard confirms exactly one thing: at the moment it was taken, the scoreboard showed 3. That's not the same as "three matches added 3 points". Between those two claims lies everything the capture doesn't contain: what score you started from and what was actually done.

To turn a capture into evidence, you write down the scenario next to it. Nothing fancy, four points:

1. what was opened: the page address or how the game was launched, and which version it is - with the edit or without;
2. the starting state: the score starts at 0, no old data got loaded;
3. the action: three matches were collected;
4. the observation: the scoreboard shows 3, captured right after the action.

Without this record, any capture could have been produced a dozen different ways. A tab left open since yesterday with the old version, a different run, a score that was already above zero - any of these gives you a picture that looks convincing and means nothing.

This article has no scoreboard screenshot, and that's on purpose. A screenshot for a check is taken after reproducing the scenario on the right version, by hand or with automation tools; a scoreboard drawn or put together for an illustration doesn't become evidence, even if it looks identical. When you take your own capture, grab the whole scoreboard, put the page address in the scenario record and save the capture next to that record - otherwise a week later you won't remember which run it was.

What the model can read

You can pass an image into the session and ask Claude to read it. The wording here makes all the difference: ask it to name the visible number and list anything it's unsure about ("is the starting state visible", "is it clear which version this is"), not to confirm a conclusion you've already decided on. Ask "confirm the bug is fixed" and you can easily get polite agreement that checks nothing.

Whether you can pass an image at all depends on the model and the environment you work in. In Claude Code you can give the path to a saved image and ask it to read the file. If the image didn't load, check the path and whether the file is accessible first; switching the model alone doesn't guarantee the problem goes away.

One scenario, before and after

A useful pair looks like this: the same check fails on the broken implementation and passes on the fixed one. The failure shows that the check notices this defect at all, and the pass shows that after the edit the defect is gone from the checked scenario.

This works under one condition: you compare the same input data and the same expectations. The same starting score, the same number of matches, the same expected value of 3. Here's the green summary of the same test after the edit:

✔ three matches give 3 points (1.3983ms)
ℹ tests 1
ℹ pass 1
ℹ fail 0
ℹ skipped 0

Between the runs the implementation changes, while the test, the input data, the expectation and the run conditions stay the same. If you had to change the requirement or fix the expectation in the test, get a new pair of runs with that check: explaining it in words doesn't replace running it on both states. A silent "it was red, now it's green" is misleading in that situation.

And two files named before.txt and after.txt don't guarantee anything by themselves. A file name isn't evidence. You have to look inside. Same command? Same test? Same expectation? Deliberately bringing back the broken state (that same git stash trick) is a separate check with its own rules. Here it's enough to know that the red before the edit comes from a real run, not from memory.

A short EVIDENCE_LOG

In this article EVIDENCE_LOG is a plain text log with six fields in this order: SYMPTOM, LOG CLUE, FAILURE ON THE BROKEN VERSION, FIX, HONEST GREEN, SCREENSHOT. That makes it easier to line up the symptom, the fix and the result. The order of the fields doesn't make the entries trustworthy by itself.

Everything we collected above fits into these fields. The command and the folder go into the failure field, the states being compared go into the fix, and links to full files go wherever they came from. There's no need to keep a separate set of documents alongside it.

Below is a sample of how to fill it in: it shows what to write in each field rather than reporting on checks that were actually done. In your own log, replace the text with your actual results.

SYMPTOM: in the bonus game, with a starting score of 0, three matches
give 30 points instead of 3. Happens every time: open the game,
collect three matches, look at the scoreboard.

LOG CLUE: the run output shows "30 !== 3" in the test
"three matches give 3 points", full output in logs/test-fail.txt.
Separately, from reading the code: in src/bonus-game/score.js
the multiplier POINTS_PER_MATCH is 10. This comes from reading the code,
the number 10 doesn't appear in the test output.

FAILURE ON THE BROKEN VERSION: HEAD a1b2c3d (made-up identifier),
clean working tree; the commit has the broken calculation and the test.
Folder - /project/bonus-game, command - npm test.
In package.json scripts.test = node --test. "three matches give 3 points" fails,
reason "30 !== 3", tests 1, pass 0, fail 1, skipped 0, exit code 1.
Output excerpt above, full output - logs/test-fail.txt.

FIX: POINTS_PER_MATCH changed from 10 to 1 in
src/bonus-game/score.js; no other changes. HEAD is still a1b2c3d.
Diff: git diff HEAD, working tree against that commit;
saved to diffs/score-fix.diff.
The expected value in the test stayed the same - 3.

HONEST GREEN: the same npm test command from the same folder, on the code
after the edit described above. The same test passes: tests 1, pass 1,
fail 0, skipped 0, exit code 0.
Full output - logs/test-pass.txt.
The five-matches scenario was not run.

SCREENSHOT: screenshots/bonus-score-3.png. The game was reopened
at http://localhost:3000 from the same working tree
after the edit. Score at start 0, three matches collected,
the scoreboard shows 3. Saving the result to the leaderboard was not checked.

The paths in the example are just an example. Full logs, diffs and images go wherever the project already keeps something similar; you don't need a special mandatory folder structure for them.

Note the last lines of fields 5 and 6. Mark a gap directly, in one phrase: "was not run", "was not checked". that's not an admission of laziness, it marks where the conclusion stops. A log without such lines looks more complete, but in fact it just stays silent about what nobody did.

Check yourself

The task has two criteria: the calculation gives 3 points for three matches starting from zero, and the interface shows 3 after those three matches. You have six materials in hand:

1. the agent's message "done, bug fixed";
2. an instruction to run npm test;
3. actual runs of one test: red before the edit and green after;
4. a diff with the multiplier change;
5. a single capture showing 3, with no description of the run;
6. a record of the reproduced scenario with a capture attached to it.

Which of them are enough for the two criteria, and what remains unchecked? Work out your answer before reading on, no peeking.

The breakdown. The first criterion is covered by item 3: a real pair of runs on one test. The diff from item 4 explains what changed in the code, but it doesn't replace a run - a change and an execution are different things. The second criterion is covered by item 6: a capture together with a scenario record that shows the starting state and the action. Items 1 and 2 confirm nothing: a "done" message contains nothing you can verify, and an instruction describes a future action. Item 5 shows the number 3 and stops there; you can't establish the scoring from it. And saving the result to the leaderboard stays unchecked even with all six materials on the table: none of them is a scenario in which the saved value was read back.

You can also put together a log like this in a session. Here's a ready-made prompt:

Put together an EVIDENCE_LOG with the six fields. For each conclusion, give
the actual check, the state of the code and the source of the result.
Mark separately what was not run. For the screenshot, name only
what is visible, and don't infer from it that the internal calculation is correct.

How to gather evidence with Claude Code

Roughly, the workflow looks like this:

1. the agent ties each criterion to a specific check: what to run, on which state of the code, what counts as the result;
2. it runs the checks that are available in this environment and shows the command with its raw output;
3. it attaches the observations: output excerpts, links to full files, a description of the reproduced scenario;
4. it states the limit explicitly: what was left unrun, and what can't be claimed from the materials at hand.

Checking the evidence itself is still on you: open the attached files and look at what's inside, compare the scenario with the one described in the criterion, and ask one awkward question about the item where no check is visible. That's usually enough to tell gathered evidence from a neatly formatted summary.

Technical reference: git diff, git stash, Node.js test runner, npm test, images in Claude Code.