agents · review · first seen Nov 3, 10:32 UTC

arXiv study tests agents on enterprise text-to-SQL delivery

An arXiv study announced results from a simulated enterprise text-to-SQL engagement in which agents had to discover requirements and deliver a post-training recipe. A builder should care because strong discovery results did not translate into consistently passing deliveries.

L0 · primary1 source4 claims checked4 min read

What shipped

This is a study, not a shipped agent or deployment. The researchers grounded a simulated customer in a real enterprise text-to-SQL engagement. The customer began with an ambiguous request. In the discovery phase, an agent had to uncover acceptance criteria, data permissions, and hardware constraints; in the delivery phase, it had to plan and execute a post-training recipe. That sequence matters: the task tested whether an agent could turn a conversation into decisions and work that met the customer’s requirements, rather than answer a fully specified coding prompt. [1][2]

The study announced a requirement-discovery evaluation across 15 models. Its strongest models found over 80% of withheld requirements, while open-weight models found 40–50%. Those are discovery results, not delivery pass rates. The excerpt also identifies incomplete discovery as a bottleneck and says 40% of withheld requirements were missing or misrepresented in the engagement. A builder should keep those measures separate rather than treating one discovery percentage as a prediction of what a particular agent will deliver. [1][2][3]

The delivery result is the harder check. Only 25% of runs across eight coding-agent configurations passed all acceptance criteria. The study identifies unreliable execution, underused compute budgets, and a lack of systematic comparisons across base models and post-training methods as further bottlenecks. It also warns that benchmark accuracy does not guarantee production readiness. The supplied excerpt does not break the pass rate down by configuration, so it cannot identify a configuration to adopt. [1]

Why it matters if you build

If I were deciding whether to put an agent into this kind of workflow today, I would make the discovered requirements a reviewable handoff before delivery begins. The simulated request was ambiguous, and the agent needed to elicit acceptance criteria, permissions, and hardware constraints. A missed permission or constraint can leave the delivery plan aimed at the wrong target even when the agent has written a plausible recipe. Human review of what the agent discovered is a narrower decision than trusting it to own the entire engagement. [1][2][3]

I would also separate a delivery attempt from acceptance. The reported 25% all-criteria pass rate makes an unreviewed handoff difficult to justify on this evidence alone. Check the work against the criteria the customer actually confirmed, and inspect whether the agent compared base models and post-training methods rather than merely spending less than the available compute budget. The study names these gaps but supplies no cost figures here; it does not establish the price of adding review or running more comparisons. [1]

The limit is specific. These results come from a simulated customer grounded in a real text-to-SQL use case, not from a reported production deployment of the evaluated agents. They support testing discovery and delivery as separate gates in a similar engagement. They do not establish that the same percentages will hold for another customer, another task, or a particular coding-agent configuration. [1][2][3][4]

How to try it today

1. Start with an ambiguous text-to-SQL request and record the requirements your agent discovers.
2. Check that its discovery covers acceptance criteria, data permissions, and hardware constraints before approving a delivery plan.
3. Have a person review the discovered requirements for omissions or misrepresentations.
4. Ask the agent to state its post-training recipe and any comparisons across base models and post-training methods.
5. Judge the completed run against every confirmed acceptance criterion, not benchmark accuracy alone.

What we checked

Every claim below was walked down the source ladder. The rung is how close the source is to the thing itself; the tense is the source's own.

1. announced L0 · primary

   The study evaluates requirement discovery across 15 models.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
2. announced L0 · primary

   The strongest models discover over 80% of withheld requirements.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
3. announced L0 · primary

   Open-weight models discover 40-50% of withheld requirements.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
4. announced L0 · primary

   Only 25% of delivery runs across eight coding-agent configurations pass all acceptance criteria.

   per arXiv cs.CL/cs.AI/cs.LG (L0)

What is not confirmed

The supplied results do not show which of the eight configurations passed, how performance varied among them, or whether any configuration is ready to run without review. They also do not establish production performance, costs, or that a high requirement-discovery score causes a delivery to pass all criteria. [1][2][3][4]

Who should act, who should wait

Move now if you can use an agent inside a reviewed text-to-SQL engagement: test its requirement discovery, confirm permissions and constraints, and gate delivery on all acceptance criteria. Wait on handing it an ambiguous customer request and treating the resulting recipe as accepted without review. The study announced promising discovery results, but its reported delivery runs passed every criterion only 25% of the time. [1][2][3][4]

Sources

• Can LLM Agents Be Forward-Deployed Engineers? Requirement Discovery and Delivery for Enterprise Text-to-SQL — arXiv cs.CL/cs.AI/cs.LG L0 · primary