Session 4
Verify Before Merge
A green CI check on a suspicious diff, decide first, then build a verification habit that matches risk.
- Judge whether a green CI diff is safe to merge
- Separate throwaway experiments from production work
- Use verification levels instead of rubber-stamp review
- Write verification notes others can trust
A situation you may recognize: decide first
Session 2 implemented the payment-helper refactor. A declined-card test failed. The AI “fixed” it. CI is green. Here is the change:
- expect(result).toEqual({ ok: false, code: 'card_declined' })
+ expect(result.ok).toBeDefined()Many reviewers would merge here, CI is green; it looks like a test fix. What actually happened: the assertion was weakened. The test no longer checks the error shape. Production can still be wrong while CI stays green. CI only runs what someone wired into the pipeline.
Sit with that for a moment. Nothing “failed” in the pipeline. The diff looks like responsible test maintenance. That is exactly why Session 1 warned about steps that feel easy after a success, review fatigue makes us accept plausible fixes. Session 4 builds a habit for that moment: name the phase, pick the verification level, look for red flags.
Two kinds of work, two standards
Before any diff, especially an AI-assisted one, ask a question many teams skip: Is this a throwaway experiment, or production work?
The answer changes what “done” means. An experiment can be wrong and still valuable; it taught you something. Production work can be green and still dangerous; it ships to users.
| Kind | Purpose | Standard |
|---|---|---|
| Exploration | Learn quickly; rule out bad ideas | OK to discard, like a rough sketch |
| Production | Ship durable change to users | Same care as always |
Many say yes if the code “looks fine.”
The missing question is which phase this is. Spikes are valuable, and should often stay off main. Production merges need a clear human check, not just a successful experiment.
Agents make exploration cheap. That is a genuine gift; you can spike an approach in an afternoon that once took a week of scaffolding. The trap is merging experimental output because it compiles, or because CI is green on a branch that was never meant to ship.
Thinking in Code calls separating these phases phase separation: move fast while exploring; move carefully before shipping. You may already do this informally with draft PRs or spike branches. The new part is that AI output looks finished earlier: so the phase question needs to be explicit, not implied.
Verification levels: climb to the rung that fits
“Looks fine” is not a method. It is what we say when we are tired, when we trust the author, or when the diff is too large to hold in working memory. Real checking has levels: and the level should match risk, not enthusiasm.
The ladder below is a vocabulary for pull requests, standups, and your own notes. You do not need every rung on every change. You do need to know which rung you skipped, and why.
Verification ladder
Climb to the rung that matches risk. AI speed does not lower the bar — it raises the cost of skipping it.
| Rung | What you do | Enough when… | Typical change |
|---|---|---|---|
| 1 Glance | Size, touched paths, scope creep | Docs-only edit in a safe folder | Low risk |
| 2 Read | Logic, edge cases, naming, deletions | Small change in a module you know well | Low risk |
| 3 Run | Tests, lint, typecheck, build | Any behavior or config change | Behavior change |
| 4 Reproduce | Manual or scripted repro of the flow | User-facing paths, regressions, perf | User-facing |
| 5 Adversarial | Second session: "find what's wrong" | Auth, payments, migrations, public APIs | High risk |
Rubber-stamp review stops at glance. Production merges should name which rung you reached.
Rule of thumb: if you would not have skipped this check before AI tools existed, do not skip it now because the draft arrived quickly.
Session 2’s three sessions map naturally:
- Plan → read carefully
- Implement → run tests before handoff
- Review → at least run; go higher if the change is risky
For payment code, you usually need run, reproduce, and at least one adversarial look, someone trying to break what was built. If you only read the diff, you are trusting prose. If you only run CI, you are trusting wiring. Reproduce means your hands on the flow that users touch.
| If the change is… | A reasonable minimum |
|---|---|
| Docs or comments in a safe area | Glance or read |
| Small logic in a module you know well | Read + run tests |
| User-facing behavior | Run + reproduce the flow |
| Auth, payments, migrations, personal data | Run + reproduce + adversarial review |
Use the table as a floor, not a ceiling. When in doubt, climb one rung. The cost of an extra reproduction is usually smaller than the cost of a quiet production bug.
Red flags on the jagged frontier
Session 1’s jagged frontier shows up in review as a polished diff on a wrong assumption. The model can be confident and articulate while wrong, that combination is new for many reviewers who previously associated “well explained” with “probably correct.”
Read the list below as patterns to notice, not accusations. Two or more together mean: slow down, climb the ladder, and ask what behavior the tests actually prove.
- Neighbor step failure: refactor fine; import one folder away wrong.
- Happy-path tests only: tests mirror the mistake instead of catching it.
- Silent behavior change: “refactor” also changes defaults or errors.
- Plausible fiction: comments describe behavior the code lacks.
- Success, wrong target: tests pass because the wrong suite ran.
The payment-helper “fix” is #2. When two or more flags appear, climb the ladder.
What good process would have looked like
- Session 1 plan named declined-card behavior as in-scope.
- Session 2 ran payment package tests (agent rules from Session 3).
- Session 3 review asked: what behavior does this test prove that the diff alone does not?
- A skeptical second pass: how could payment errors break without failing these tests?
The lesson is not “find a smarter model.” It is treat plausible work as innocent until reproduced: the same standard you would apply to a eager junior’s first pull request, applied to a fast draft.
Skills like review-changes and review-pr encode questions that catch assertion weakening. They are optional shortcuts for the adversarial rung; the habit matters more than the tool.
A verification note you can leave in a pull request
Verification notes are how you show your work to reviewers, especially on AI-assisted diffs where line count is high and attention is scarce. You are not performing virtue. You are making it easy for the next person to see what was checked and what was not.
Verification:
- Phase: production merge (not a spike)
- Ran: `pnpm --filter payments test`
- Reproduced: successful checkout and declined-card path by hand
- Adversarial: second session reviewed error handling
- Not verified yet: provider sandbox outage behaviorThat last line is a kindness; it says what is still unknown.
The merge checklist at the end of Session 6 is there to bookmark.
When to keep extra human control
Some areas were always “senior hands” territory. AI does not remove that; it accelerates drafts in those areas too, which makes skipping review more dangerous, not less.
Auth, crypto, personal data, novel architecture, performance without benchmarks, irreversible migrations. The AI can still propose. You hold the bar. The top rung of the ladder is not optional there. If your team has no written policy yet, Session 5’s lead notes include a starting point for what may enter which tools.
Before Session 5
Session 5 opens with a standup many teams recognize: the same correction in three people’s chats, and what changes when the rule is written down once.
Next: Session 5: Team Patterns