agents · review · first seen Oct 28, 09:33 UTC

FeatureArc introduces benchmark for evolving coding features

FeatureArc was announced to evaluate coding agents as they extend their own earlier work, rather than answer one request on a fixed codebase (293). That matters if you need an agent to keep a feature correct through later changes, not just complete its first stage (293, 296).

L0 · primary1 source5 claims checked4 min read

What shipped

FeatureArc was announced as a benchmark for successive extensions of the same feature. The distinction is whose code the next request builds on: the agent must continue from its own earlier implementation. The source contrasts this with feature benchmarks that evaluate a single request on a fixed codebase. For a builder, that makes the unit of evaluation closer to an evolving feature than an isolated coding prompt. It does not, by itself, show that any agent can sustain that work reliably. [1]

The announced design combines dependencies between stages, explicit requirements and interface contracts, and executable checks across four programming languages. Its task-construction pipeline uses repository histories. On reference code, the pipeline validates fail-to-pass tests for new behavior and pass-to-pass tests for existing behavior before and after each stage. That pairing matters: adding the requested behavior is not enough if an extension breaks behavior the feature already had. [1][2]

The reported evaluation covered 10 coding agents and 73 manually reviewed tasks, each with 2 to 10 stages. Each agent worked in one persistent session and workspace, without external test feedback or rollback. Those conditions are part of what was evaluated, not incidental setup. An agent could not use reported test results to revise a stage or roll back a bad change within this evaluation. [1][2]

Pooled first-stage success was 62.7%. I would read that as a starting measure, not a measure of completed features: the benchmark is explicitly about what happens after the first implementation. The supplied excerpt begins to describe a gap between first-stage and whole-feature success but stops before giving the later figure. There is no supported whole-feature rate to report here. [1][2]

Why it matters if you build

If I were choosing an agent for feature work today, I would ask it to extend the same feature more than once and check both the new requirement and the old contract after every change. FeatureArc gives a clear reason for that workflow: each stage depends on earlier code, while its checks distinguish new behavior from behavior that should remain intact. A good first answer would be insufficient evidence for the job I actually need done. [1][2][3]

The evaluation conditions also limit how I would use its results. A team that gives agents external test feedback or routinely rolls back changes is running a different process from the reported persistent-session run. Conversely, a team that lets an agent keep extending one workspace without those aids has a more direct reason to examine this setup. Neither comparison supplies a performance estimate for that team's own workflow. [1]

There is a practical cost to taking this question seriously: a local trial needs staged requests and checks for both newly requested and previously working behavior, not just a verdict on the final patch. The announced pipeline validates those kinds of checks on reference code. The supplied material does not say how much effort a builder would spend preparing an equivalent trial, so I would treat that effort as a planning consideration, not a measured benchmark result. [1][2]

How to try it today

1. Choose one feature you already expect to extend, rather than an isolated coding request (293).
2. Write the first requirement and its interface contract before asking an agent to implement it (293).
3. Write a later extension that depends on the agent's first implementation (293).
4. Check the new behavior and the behavior that should still pass after each stage (294).
5. Record whether you supplied test feedback or allowed rollback; those choices differ from the reported evaluation if you did (296).

What we checked

Every claim below was walked down the source ladder. The rung is how close the source is to the thing itself; the tense is the source's own.

1. announced L0 · primary

   FeatureArc evaluates coding agents on successive extensions of the same feature.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
2. announced L0 · primary

   FeatureArc includes executable checks across four programming languages.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
3. announced L0 · primary

   The evaluation tested 10 coding agents on 73 manually reviewed tasks of 2 to 10 stages.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
4. announced L0 · primary

   Each agent worked in one persistent session and workspace without external test feedback or rollback.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
5. announced L0 · primary

   Pooled first-stage success was 62.7%.

   per arXiv cs.CL/cs.AI/cs.LG (L0)

What is not confirmed

The supplied claims do not establish a whole-feature success rate, a ranking among the 10 agents, or results broken out by language or task length. The excerpt ends before its first-stage-to-whole-feature comparison is complete. The 62.7% figure should not be used as a completion rate for multi-stage features. [1][2][3]

These claims also do not establish how an agent performs when it receives external test feedback or can roll back a change. Those options were absent from the reported runs. Nor do the claims establish that the benchmark materials are available for a builder to run today. [1][2]

Who should act, who should wait

Act now if you are selecting an agent to maintain a feature through successive requests: add staged extensions and checks for retained behavior to your own trial. Wait before changing a purchasing or deployment decision on the strength of the first-stage number alone. The reported setup is specific, and the supplied material gives no complete-feature result to compare with your acceptance threshold. [1][2][3][4]

Sources

• FeatureArc: Benchmarking Coding Agents on Evolving Features — arXiv cs.CL/cs.AI/cs.LG L0 · primary