goalline How it worksClaude Code hookGitHub ActionBenchmarkCommandsGitHub

goalline / How it works

How to tell if an AI coding agent edited a test to make it pass

Run the old version of each test the change edited against the new code. If an old test fails, behaviour changed and the test was rewritten to match. goalline does this for every pull request, with Jest, Vitest or pytest, and shows the old and new values.

Why a green check can't tell you

CI runs the new tests on the new code. When a change edits the code and its test in the same diff, both sides move together and the suite passes either way. A reviewer then has to spot, in the test diff, that an expectation changed, and decide whether it should have.

Coding agents make this common. ImpossibleBench (arXiv 2510.20270) gives agents tasks whose tests contradict the spec, so any pass is a shortcut, and reports that "an LLM agent with access to unit tests may delete failing tests rather than fix the underlying bug".

The method: old tests on new code

This is real goalline output for a change that made add(2, 3) return 6 and edited the test's 5 to 6:

goalline goalpost · example · base main (merge-base 62fac20) → head 94ac494

Old versions of 1 edited or deleted test file(s), run on the new code.

  MOVED   test/add.test.mjs
          1 of 1 old tests fail on the new code
            ✗ adds two numbers: Expected 5, received 6

FLAGGED (1 moved). Old tests no longer pass on the new code: behaviour changed and the tests were edited to match. A person should confirm that was intended.

The verdicts

VerdictMeaningFlagged
HELDEvery old test still passes on the new code.no
MOVEDAn old test fails on the new code: behaviour changed and the test was edited to match. Shown with the old and new values, such as Expected 5, received 6.yes
BROKEThe old test file no longer loads against the new code (a missing module or export).yes
CRASHEDThe old test crashed inside the new code (a TypeError, say) before reaching its expectations: its fixtures or mocks lack something the new code needs. Not judged; for Jest and Vitest, the assertion transplant then runs the old expectations with the new file's setup.yes, not judged
EMPTYNo old tests ran in that file.no
UNKNOWNThe test runner did not report the file, or its failing old tests fail on the old code too (the environment or a flaky test), so they say nothing about the change.yes, not judged

Flagged files are sorted by how much a reviewer is needed. Needs a look is an old test whose lines the change removed or rewrote. Low risk is an old test the change only added lines to (a list that grew, a mock that was added), so every old line is still there. An added early return, dead branch, skip or focus never counts as low risk.

With ts-jest, an old file that only fails to compile against the new types is run again with the type check off, so it is judged on behaviour. A crash in the new code is not called a behaviour change: it is CRASHED, and for Jest and Vitest the old expectations are transplanted into the new test file, with its setup, and run again.

Try it

# One change: old tests on the new code (exits 2 when something is flagged)
npx github:intikhab49/goalline goalpost --base origin/main -- npx jest {files}

# Your history: what it would have flagged on the last 200 merged PRs
npx github:intikhab49/goalline history --from-git -- npx jest {files}

Vitest: -- npx vitest run {files}. pytest: -- python -m pytest {files}, with the project's virtualenv active. goalpost also writes a Markdown summary ready to paste as a pull request comment.

What it can't catch

Related