A Mutation Round That Reads the Exit Code Scores a Suite That Never Started as a Kill
A mutation harness that trusts the test runner's exit status will count a suite that never started as a killed mutant. Ephesus did that twice. Its checked-in round now reads vitest's JSON report, scores only tests the baseline proved can pass, and refuses any verdict it cannot earn.
A suite that refuses to start also exits 1
Ephesus is a pre-alpha Electron harness that runs AI coding agents behind gates its owner sets. Since its eighth milestone, a mutation round has been part of its verification bar: plant one hand-aimed change in the code a package defends, run that package's tests, and put the file back.
The suite has a free-memory gate. test/global-setup.ts refuses to start below 1 GB free, because under that the project watched vitest's forks die mid-run, and it exits 1 without running a single test. On 9 September 2026 a round reported a score of 11/12 with free memory at 0.90 GB, and every verdict in it was unearned (decisions log).
On 2 October a fresh scratch harness made the same mistake: 14 of 28 runs executed zero tests and were reported as kills, three controls included (implementation record). No round had ever run from a checked-in tool. One milestone alone ran 11, each from a script written into a session scratchpad and deleted with the session (GYM-008 proposal), so every fix had to be remembered and the bugs came back.
The planted control is the round's certificate
A planted control is a change no test can see, such as a reworded doc comment, and it must survive. In the 9 September round it came back killed, so something other than the tests had done the killing; on 2 October, three controls were among the false kills. mutate.cjs now refuses a spec that declares no control, and a killed control makes the whole round INVALID, exit status 2.
The same logic guards survivals. An anchor that matches nothing turns its mutant into a no-op that "survives", and one that matches twice is an edit nobody chose. The tool counts byte occurrences, overlapping ones included, and refuses any anchor whose count is other than one before the baseline runs. The proposal states the principle in a line: "an unearned kill and an unearned survival are the same error".
mutate.cjs keeps mutants hand-aimed. GYM-008 rejected generators such as Stryker, which it says produce mostly equivalent mutants, and a CI job, because a round rewrites source files.
Exit status 1 covers kills and runs that prove nothing
Before writing the rule that reads each outcome, the author produced it by running vitest 4.1.11 on a fixture and recorded the report shape (record, section 3.2). Five of those shapes show why neither signal works alone:
| What happened | What vitest leaves | Verdict |
|---|---|---|
| a test that passed at the baseline fails | that test failed, exit 1 | kill |
| a module the test imports stops parsing | the file failed with no tests, exit 1 | kill, reported as a file kill |
| global setup throws, as the memory gate does | a report holding no file, exit 1 | INVALID, retried |
| the code under test kills the worker | success: true, its test pending, exit 1 | INVALID |
| an unhandled rejection | success: true, every test passed, exit 1 | INVALID |
The exit status says 1 for the memory gate, and the report says success for a dead worker. So classify() reads the report first, then the exit status, then the set of tests that passed at the baseline. Only a refusal to start is retried, by default twice with a 30-second wait; a run that hits the default 900-second timeout is INVALID and is not retried.
A kill needs a test the baseline proved can pass
The kill rule changed last, after a second adversarial review. Until commit b20f83b, any failing test counted, including one the mutant switched on that had never passed. Now the baseline records which tests passed, keyed by file, full name and position among same-named tests. A mutant run is a kill only if one of those fails or a listed file cannot load at all; any other failure is INVALID.
The same baseline set closes two earlier holes. A file whose tests are all skipped would pass vacuously, so each listed file must hold a test that passed at the baseline. A dead worker leaves tests pending, so every baseline test must pass or fail under each mutant.
Refusing every skipped, todo or .only test outright was the stricter option, and it was rejected: the suite's platform-gated skips would bar those files from any round. The cost of keying by name is written down as a known limit in TEST-STRATEGY section 10: a same-named test can stand in for the one that passed.
The tool passed its own round, then a hostile review broke it
mutate.cjs ships with a spec aimed at itself. Its first round found a survivor: with the tracked-file check deleted, nothing refused a round file that .gitignore hides, so a test was missing and was written. Round 2 was clean.
An independent pass then tried 22 attacks across 18 fixture rounds against that version and still drew verdicts it had not earned. A describe.skip file gave a green baseline and then a survivor. An assume-unchanged flag let the restore destroy uncommitted work. Each finding was reproduced against the committed tool before a fix was written.
Every round file is also re-hashed after each run and each restore, because OneDrive twice put a restored file back with the mutant in it. That is a late write, like the one git's detached maintenance produced in Ephesus CI.
A second pass found eleven more ways in, and those were recorded as known limits in issue #70 rather than closed. A vitest retry setting, for one, turns a kill into a pass. The fourth round, on the final code, killed 43 of 43 real mutants with the control surviving, in 57 min 55 s on Windows (PR #71).
Three rules worth copying into any mutation harness
- Plant a no-op control in every round and void the round when it dies.
- Score from the structured report, over the tests that passed at the baseline. Treat zero tests, a timeout, or a non-zero exit with no failure as no verdict.
- Make a survivor stop the round with a question: is this state reachable from outside the module, and can any listed test read the changed line? Ephesus prints no percentage, by design.
The only other checked-in spec aims a mutant at the exit status itself. Its C1 drops the process.exit call from the README checker, so the script prints its failure, exits 0 and passes CI. All 8 real mutants in that round died, but C1 needed a new test first: every earlier test called check() directly.
GYM-008's metrics were due for measurement on 17 October, and as of 19 October the ledger records no result.
Update, 29 October: the Rust rewrite generates its mutants
On 19 October, the day this post was written, Ephesus accepted ADR-0041: every component is rewritten in Rust. The TypeScript tree that holds mutate.cjs is frozen as the behavioural reference and is not developed further. New rounds read their spec from .cargo/mutants.toml.
The Rust round, cargo xtask mutants, runs cargo-mutants, which generates its mutants itself, while mutate.cjs reads hand-aimed ones from a spec. The control was kept in a new form: the round runs again over a quarter of the mutants with a test filter that matches no test, and every mutant that builds must survive. If the control catches one, the round is INVALID.