builders · review · first seen Nov 6, 10:33 UTC

arXiv paper tests silent failures in LLM code repairs

An announced preregistered study found that a release-note edge input reduced the share of wrong LLM repairs passing a typical-input release check. Builders reviewing repairs after API migrations have a concrete check to try, but not a guarantee that it will catch every wrong repair.

L0 · primary1 source4 claims checked4 min read

What shipped

This is an announced study, not a shipped repair tool. It covered 73 tasks drawn from 60 documented changes in CPython, Node.js and SQLite. The researchers executed every candidate on the real runtimes and checked it against withheld contracts. That matters for reading the results: the study looked beyond whether a repaired procedure ran cleanly or passed the inputs visible to a release check. [1]

Over the complete repair space, an all-assertions gate using typical inputs released 0.410 of wrong repairs. Adding one edge input written from the release note reduced that figure to 0.065. The comparison is about wrong repairs that passed a check, not the share of all repair attempts that succeeded. It supports testing a documented boundary in addition to typical behavior; it does not make the edge input a complete correctness test. [1][2]

The excerpt reports 0.061 on 15 unseen clusters for the release-note edge input. A blind GPT-5.6-luna writer, which never saw the program, reached 0.070 using three inputs. These are distinct ways of constructing checks, with different input counts. The result gives a team a reason to compare a release-note-derived input with its existing gate before adding a separate test-writing stage. [1]

The study also enlarged its menu to 1491 fragments, then added 233 hard distractors that matched the legacy output on typical inputs. In that setting, the excerpt reports that the edge input and the frontier-model writer passed their preregistered tests, while two local-model writers failed theirs. It also reports that 0.168 and 0.153 of the hard wrong programs still passed. The harder cases leave a material residual failure rate. [1][2]

Why it matters if you build

For a team checking an LLM repair after a documented API change, the workflow implication is narrow and usable: keep the typical-input assertions, then derive a separate edge input from the release note before accepting the repair. A clean run and passing typical assertions were not enough in this study. The extra input targets the changed behavior rather than asking for another typical example. [1][2]

This adds work at review time. Someone must read the relevant release note, identify an edge input from it, and run the repaired procedure on the applicable runtime. The claims do not price that work. They do show why it may be worth testing: the typical-input gate and the edge-input gate released different shares of wrong repairs over the study’s complete repair space. That comparison is a reason to measure the check in your own workflow, not to assume the reported rate will transfer to it. [1][2][3]

Do not treat a test-writing model as an automatic substitute for that review. The excerpt reports a result for a blind frontier-model writer, but two local-model writers failed their preregistered tests. It also says the edge input reduced wrong releases per attempt across three open-weight families on all three menus without lowering correct completion. Those findings favor evaluating the specific check and writer you would use; they do not establish that any local writer will produce an adequate edge case. [1][2]

How to try it today

1. Choose one LLM repair prompted by a documented API change in CPython, Node.js or SQLite.
2. Keep your current assertions on typical inputs as a baseline release check.
3. Read the change’s release note and write one input that exercises an edge of the documented change.
4. Run the repaired procedure on the applicable runtime with both the typical inputs and that edge input; inspect the result before release.
5. Record whether the edge input rejects a repair the typical-input gate would have accepted, without treating a pass as proof of correctness.

What we checked

Every claim below was walked down the source ladder. The rung is how close the source is to the thing itself; the tense is the source's own.

1. announced L0 · primary

   The preregistered study covered 73 tasks and 60 documented changes in CPython, Node.js and SQLite.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
2. announced L0 · primary

   An all-assertions gate on typical inputs released 0.410 of wrong repairs over the complete repair space.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
3. announced L0 · primary

   One edge input written from the release note cut that figure to 0.065.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
4. announced L0 · primary

   Two local-model writers failed their preregistered tests.

   per arXiv cs.CL/cs.AI/cs.LG (L0)

What is not confirmed

The supplied claims do not establish that one release-note edge input catches every wrong repair. The hard wrong programs that still passed make that limit explicit. They also do not establish the same rates for a team’s own code, release notes or release gate. [1][2]

The claims do not establish the cost of authoring and maintaining these inputs, or that the two local-model writers are useful as drop-in replacements for a human-written release-note edge input. Their reported preregistered tests failed. [1][2]

Who should act, who should wait

Act now if your team accepts LLM repairs after documented API migrations using typical-input checks: add a release-note-derived edge input to the next relevant review and compare what each gate accepts. Wait before making this a universal release policy or replacing review with an automated local-model writer. The study reports a substantial reduction in wrong repairs passing its gate, but also residual hard failures and two failed local-writer tests. [1][2][3][4]

Sources

• Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations — arXiv cs.CL/cs.AI/cs.LG L0 · primary