agents · review · first seen Nov 3, 10:32 UTC

CLEO fits a minimal agent harness to a self-hostable LLM

Researchers announced CLEO, a method for fitting a minimal agent harness to a fixed, self-hostable LLM for investigative analysis. Builders should care because its reported held-out result is close to a larger out-of-the-box baseline, but the evidence is limited to the authors’ evaluation.

L0 · primary1 source4 claims checked4 min read

What shipped

CLEO, short for Constrained Loop for Environment Optimization, is an announced method for fitting a minimal agent harness to a fixed, self-hostable LLM. The authors frame it around government intelligence analysis of large, messy evidence collections, where security and cost constraints can rule out hosted frontier models and broad, hard-to-audit tool surfaces. I would evaluate this as a harness-selection approach for that setting, not as a claim that a new model shipped. [1]

CLEO uses an expert-validated dataset to decide which tools, guardrails, memory policies, and loop components to admit. A component must improve accuracy while remaining general and auditable. That is the central engineering decision: test the parts around the fixed model rather than assume a broader agent harness is better. The supplied excerpt does not name the components CLEO selected, so it gives a selection rule rather than a configuration to copy. [1]

The authors report an 83.8% pass rate on their held-out dataset. They report that this is within 1.2 points of the strongest out-of-the-box Claude Code SDK baseline. Both figures belong to the authors’ evaluation. They are a reason to test whether a fitted harness works on your cases, not an estimate of the pass rate you will get on a different evidence collection. [1][2]

The authors also report that the CLEO harness is two orders of magnitude smaller than that baseline. The supplied excerpt does not define the size measure or give a component count, operating cost, or running time. Read the size and pass-rate claims together: the reported result makes a compact harness worth investigating, but it does not tell you which capabilities the smaller surface omits. [1][2]

Why it matters if you build

For a team using a fixed, self-hostable LLM, this changes the harness review from a choice of agent package to a decision about each proposed component. An expert-validated dataset can test whether a tool, guardrail, memory policy, or loop component improves accuracy; generality and auditability are further admission conditions. That gives builders a way to explain why a component is present. The source does not determine which components your own investigations require. [1]

The reported baseline comparison poses a useful local question: how much pass-rate difference accompanies a smaller harness on the work you actually perform? On the authors’ held-out dataset, CLEO reached 83.8%, within 1.2 points of their strongest stated out-of-the-box baseline, while the harness was reported as two orders of magnitude smaller. Repeating the comparison with your expert-validated cases is more defensible than treating those figures as a forecast. [1][2][3][4]

Security, cost, and audit needs motivate the work, but the excerpt reports no deployment budget, measured cost reduction, or security assessment. A narrow tool surface may be easier for your team to review; its reported size is not proof that it is cheaper to operate or safe for a particular evidence collection. Keep those operational checks separate from the authors’ held-out pass-rate result. [1][2][3]

How to try it today

1. Choose a fixed, self-hostable LLM and a bounded investigative-analysis task.
2. Assemble expert-validated examples and reserve cases for held-out evaluation.
3. List candidate tools, guardrails, memory policies, and loop components, including how each can be audited.
4. Test whether each candidate improves accuracy while remaining general and auditable; admit only those that do.
5. Measure the resulting harness on held-out cases. Compare its pass rate and size with an out-of-the-box baseline instead of assuming the reported figures transfer.

What we checked

Every claim below was walked down the source ladder. The rung is how close the source is to the thing itself; the tense is the source's own.

1. announced L0 · primary

   CLEO admits tools, guardrails, memory policies, and loop components that improve accuracy on an expert-validated dataset while remaining general and auditable.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
2. announced L0 · primary

   The CLEO harness reaches an 83.8% pass rate on the authors’ held-out dataset.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
3. announced L0 · primary

   The CLEO harness is within 1.2 points of the strongest out-of-the-box Claude Code SDK baseline.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
4. announced L0 · primary

   The CLEO harness is two orders of magnitude smaller than that baseline.

   per arXiv cs.CL/cs.AI/cs.LG (L0)

What is not confirmed

The supplied excerpt does not establish performance beyond the authors’ held-out dataset, identify the selected components, or provide a configuration to reproduce the harness. It does not define the measure behind the size comparison or establish cost and security outcomes. These are questions for local verification, not findings that CLEO failed those tests. [1][2][3][4]

Who should act, who should wait

Move now on a bounded evaluation if you have a fixed, self-hostable LLM, expert-validated investigative examples, and a need for a small, auditable agent surface. Record why each component passes the admission test. Use the reported pass rate, baseline gap, and relative size as questions to investigate locally, not as acceptance thresholds for your task. [1][2][3][4]

Wait before replacing an existing workflow if that decision depends on a specified implementation, verified deployment costs, a demonstrated security property, or evidence that the reported pass rate transfers to your cases. The supplied result does not establish those conditions. A local test is justified; a production switch based on the size claim alone is not. [1][2][3][4]

Sources

• Optimize the Harness, Not the Model: Fitting a Minimal Agent to a Self-Hostable LLM for Investigative Analysis — arXiv cs.CL/cs.AI/cs.LG L0 · primary