agents · review · first seen Oct 28, 09:33 UTC

M4Bench evaluates clinical EHR derivation agents

M4Bench was announced as a test of whether AI coding agents can reproduce clinical EHR derivations. I would pay attention to its controls: they suggest that a complete operational specification matters more than packaging instructions as a skill.

L0 · primary1 source4 claims checked4 min read

What shipped

The announced benchmark contains 28 derivation tasks across 15 task families on MIMIC-IV and eICU. These tasks concern definitions that do not sit neatly in an EHR schema: item codes, time windows, missingness rules, and the grain of the output. Each choice can change the resulting cohort. I read M4Bench as a test of whether an agent can implement those choices, not merely produce code that runs. [1]

The reported campaign ran Codex in Docker isolation with GPT 5.4 Mini and GPT 5.5. Across the campaign, clinician-reviewed skills raised reward by 0.136 over the no-skill baseline; 26 of 28 tasks improved. That is evidence for the skills in this campaign, not a general result for every agent or clinical derivation. The reported measure is reward, and the excerpt does not define a clinical threshold for acting on a derived result. [1]

The controls make the aggregate gain more useful to a builder. On an eight-task subset, an operational specification alone raised reward by 0.162. Adding skills on top raised it by 0.024, a difference the source describes as indistinguishable from zero. On a separate ten-task subset, decoy skills scored like no skill, while raw reference SQL nearly matched targeted skills. Those comparisons point toward task-specific procedural detail rather than the skill format itself. [1][2]

Why it matters if you build

If I were assigning an agent an EHR derivation today, I would first write down the choices the schema cannot supply: which item codes count, where each time window begins and ends, how missing values are handled, and what one output row represents. That is the workflow change this evidence supports. A prompt that names a variable but leaves those choices implicit asks the agent to settle parts of the research definition on its own. [1][2]

I would not treat a skill file as the first purchase or the first engineering task. The eight-task control suggests a complete operational spec can capture most of the measured benefit on that subset, while the added skill effect there is not distinguishable from zero. Raw reference SQL nearly matching targeted skills on the ten-task subset gives a second reason to inspect the information being delivered before investing in its wrapper. Neither subset establishes that skills have no value elsewhere. [1][2]

The practical cost is specification work and review. Someone still has to make the derivation choices explicit and check that an agent's output follows them. M4Bench reports a reward comparison in a Docker-isolated Codex campaign, not a measured saving in review effort or a guarantee about downstream conclusions. I would use the result to decide what to test in my workflow, not to waive checks on a clinical derived variable. [1][2][3]

How to try it today

1. Pick one EHR derivation and record its item codes, time windows, missingness rules, and output grain before asking an agent to implement it.
2. Run the same derivation once without those procedural details and once with a complete operational spec; compare the outputs against the intended definition.
3. If you already use a clinician-reviewed skill, repeat the comparison with that skill added to the operational spec. Keep the spec and the task fixed.
4. Review disagreements at the level of codes, windows, missingness, and output grain before deciding whether the skill wrapper contributed anything useful.

What we checked

Every claim below was walked down the source ladder. The rung is how close the source is to the thing itself; the tense is the source's own.

1. announced L0 · primary

   M4Bench includes 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
2. announced L0 · primary

   In a Docker-isolated Codex campaign using GPT 5.4 Mini and GPT 5.5, skills raised reward by 0.136 over the no-skill baseline.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
3. announced L0 · primary

   On an eight-task control subset, an operational spec alone raised reward by 0.162.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
4. announced L0 · primary

   On that subset, skills added 0.024 on top, indistinguishable from zero.

   per arXiv cs.CL/cs.AI/cs.LG (L0)

What is not confirmed

The supplied results do not establish that skills add a measurable benefit beyond an operational spec across all 28 tasks. The reported 0.024 increment, indistinguishable from zero, comes from the eight-task control subset. Nor does the campaign establish that its reward gain translates into a particular clinical outcome. Those are different questions from whether agents scored better on these derivations. [1][2][3][4]

Who should act, who should wait

Move now if you build agents for EHR derivation and can review a task's intended definition. Make the operational spec explicit, then test whether a skill adds value against that baseline. Wait before making a skill format the default on the strength of the aggregate gain alone: the controls leave its additional contribution uncertain on the tested subset. Do not treat an improved benchmark reward as permission to skip derivation review. [1][2][3][4]

Sources

• M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents — arXiv cs.CL/cs.AI/cs.LG L0 · primary