agents · review · first seen Nov 6, 10:33 UTC

Embodied agents score higher with exposed criteria

A controlled study announced that exposing success criteria raised an embodied coding agent’s score under the given verifier, but the advantage vanished when the criteria changed slightly. Builders should test whether a score gain survives a valid change in evaluation.

L0 · primary1 source5 claims checked4 min read

What shipped

This is a reported study finding, not a shipped agent or benchmark update. The paper examines Criteria Exposure, in which a learning agent can see the benchmark success criteria. The authors say several recent works use this practice. Those criteria were originally written to verify policies trained by imitation learning, rather than to guide an agent writing robot-control programs. That difference matters when reading a score: the criteria can demand more than the corresponding language instruction. [1][2]

The authors report controlled experiments with a self-improving agent both with and without criteria exposure. With exposure, the agent quickly produced code with a higher success rate under the given criteria. That is a result about the verifier used in the experiment. On its own, it does not show that the code better performed the task described in language. [1]

The comparison that changes my reading of the result is the altered-criteria test. When the authors changed the success criteria slightly, without departing from the original instructions, the exposed agent’s success-rate advantage disappeared. The finding separates doing well against one stated verifier from retaining that advantage under another verifier that still fits the task. It does not say that the agent ceased to produce useful code. [1][2]

The paper also reports that skills developed under criteria exposure transfer poorly. I would keep that finding separate from the altered-criteria result: one concerns performance under a changed check, and the other concerns what carries over from the agent’s learning. Both narrow what I would infer from the higher initial success rate. Neither supports treating criteria exposure as a general improvement in embodied task execution. [1][2][3]

Why it matters if you build

If I were evaluating an embodied coding agent today, I would record whether it saw the success criteria before interpreting its score. Exposure is part of the evaluation setup, not a small detail to leave out. The reported gain is real under the given criteria, but the changed-criteria result means I would not use that gain alone to decide that the agent learned the instructed task. [1][2][3]

A practical check is to keep the language instruction fixed and make a slight, instruction-consistent change to the success criteria. Then compare the exposed and unexposed conditions under both checks. That adds evaluation work, but it tests the precise limitation reported here. If the advantage disappears, I would report the original score as tied to its verifier, rather than present it as a durable task gain. [1][2]

I would also avoid using the exposed agent’s acquired skills as the sole reason to adopt that learning setup for later work. The reported transfer is poor. The source does not give a basis here for pricing the extra evaluation work or predicting how a particular agent will perform on a particular next task. For a decision today, the useful distinction is narrower: success under the visible criteria and transfer beyond them need separate checks. [1][2]

How to try it today

1. Write down the language instruction and the success criteria you plan to use. Mark whether the learning agent can see those criteria.
2. Run an evaluation with criteria exposed and one without exposure; compare success under the given criteria.
3. Change the criteria slightly while keeping them consistent with the original instruction. Run the comparison again.
4. Report the given-criteria result and the changed-criteria result separately. Do not call the first a task gain if its advantage disappears in the second.
5. If you intend to reuse learned skills, test that transfer separately rather than infer it from the original success rate.

What we checked

Every claim below was walked down the source ladder. The rung is how close the source is to the thing itself; the tense is the source's own.

1. announced L0 · primary

   Several recent works expose benchmark success criteria to the learning agent.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
2. announced L0 · primary

   The success criteria were originally written to verify policies trained by imitation learning.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
3. announced L0 · primary

   With criteria exposure, the agent quickly produces code with a higher success rate under the given criteria.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
4. announced L0 · primary

   That success-rate advantage disappears when the criteria change slightly without departing from the original instructions.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
5. announced L0 · primary

   Skills developed under criteria exposure transfer poorly.

   per arXiv cs.CL/cs.AI/cs.LG (L0)

What is not confirmed

The supplied findings do not establish the size of the score advantage, how much transfer fell, or how broadly either result applies beyond the controlled experiments. They also do not establish that every criteria-exposed agent fails the instructed task. The supported conclusion is that this agent’s advantage under the given criteria disappeared after a slight, instruction-consistent change, and that its developed skills transferred poorly. [1][2][3]

Who should act, who should wait

Move now if you evaluate or build embodied coding agents using visible success criteria: add an instruction-consistent altered-criteria check before relying on a higher verifier score. Wait before treating criteria exposure as evidence of better task completion or transferable skill without those separate results. The study supports a change in how to test and describe the gain, not a blanket decision to stop using criteria exposure. [1][2][3][4]

Sources

• Learning the Task or Learning the Verifier? Rethinking Criteria Exposure in Embodied Self-Improvement — arXiv cs.CL/cs.AI/cs.LG L0 · primary