agents · review · first seen Nov 6, 10:33 UTC

Terminal-Bench releases 2.1 repairs after benchmark audit

An audit of Terminal-Bench 2.0 identified issues in roughly 30% of the benchmark, and the authors released repairs as Terminal-Bench 2.1. If you use its scores to choose or study coding agents, check which version produced each result before comparing them.

L0 · primary1 source3 claims checked4 min read

What shipped

The authors released the audit repairs as Terminal-Bench 2.1. Their paper describes a maintenance problem: a benchmark can pass review at release and still develop defects as agents find valid solutions that existing verifiers reject, or as evaluation infrastructure changes assumptions built into tasks and tests. The audit identified issues in roughly 30% of Terminal-Bench 2.0. That is a reason to inspect the benchmark version behind a score, not evidence that every affected task had the same kind of defect. [1]

The described validation workflow combines community reports, task artifacts, agent traces, automated oracle checks, and human-gated repairs. This matters because a disputed result need not end at a score: the task and trace can be examined, a proposed correction can be checked, and a person can gate the repair. The excerpt describes that workflow and the release of repairs; it does not provide a task-by-task account of what was changed in 2.1. [1]

The authors report that the repairs increased accuracy by over 10 percentage points and moved leaderboard positions by up to three ranks. Those are reported changes to evaluation results, not a claim that an agent gained capability. A builder reading a score should therefore treat the benchmark version as part of the result. A builder reading a leaderboard should also check whether the positions being compared came from the same repaired evaluation. [1][2]

Why it matters if you build

For a model-selection workflow, the immediate risk is a comparison made across incompatible evaluations. If an older result came from Terminal-Bench 2.0 and a newer one came from 2.1, a difference may reflect repairs rather than a difference between agents. The reported accuracy change is large enough to warrant revisiting a decision that relied on a narrow score gap. The reported movement of up to three leaderboard ranks likewise makes a position alone a poor record of why one agent was preferred. [1][2][3]

For an evaluation team, the practical cost is review work. Keep the task artifacts and agent traces alongside results so that a rejected solution can be examined rather than dismissed from its score alone. Route a suspected defect through an oracle check and human review before treating a repair as settled. This follows the workflow the authors describe; the supplied excerpt does not state how much time that review takes, which tasks were repaired, or whether your own evaluation will see a similar score change. [1][2]

How to try it today

1. Identify which of your saved Terminal-Bench results used 2.0 and which used 2.1; label the version next to each score.
2. Flag decisions that compared scores from different versions, especially decisions based on a small score difference.
3. For a result you dispute, gather its task artifact and agent trace before judging whether the verifier rejected a valid solution.
4. If you propose a task repair, use an automated oracle check and human review rather than changing a score by hand.
5. Revisit any decision based mainly on a leaderboard position, noting the authors’ reported movement of up to three ranks.

What we checked

Every claim below was walked down the source ladder. The rung is how close the source is to the thing itself; the tense is the source's own.

1. announced L0 · primary

   An audit identified issues in roughly 30% of Terminal-Bench 2.0.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
2. announced L0 · primary

   The repairs increased accuracy by over 10 percentage points.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
3. announced L0 · primary

   Leaderboard positions moved by up to three ranks after the repairs.

   per arXiv cs.CL/cs.AI/cs.LG (L0)

What is not confirmed

The supplied material does not identify the affected tasks, give per-agent score changes, or establish that every leaderboard position moved. It reports an accuracy increase of over 10 percentage points and movement of up to three ranks, but it does not show whether those changes apply to the particular agents you use. It also does not establish that a 2.1 score predicts performance in your own workflow. [1][2][3]

Who should act, who should wait

Act now if a Terminal-Bench score or rank will influence a coding-agent choice today: record the version, avoid direct cross-version comparisons, and reopen any close decision that depended on an older result. Wait before claiming that a specific agent improved or declined unless you have comparable results for it. If you do not use Terminal-Bench in a decision, there is no evidence here that you need to change your agent; the useful action is to preserve evaluation context when you next assess one. [1][2][3]

Sources

• Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks — arXiv cs.CL/cs.AI/cs.LG L0 · primary