agents · review · first seen Nov 6, 10:33 UTC

XStack benchmarks PyTorch-to-JAX repository migration

XStack was announced as a benchmark for agents migrating PyTorch repositories to JAX on a clean TPU machine. Builders can use its reported pass rates to screen agents, but not to predict the outcome for their own repository.

L0 · primary1 source4 claims checked4 min read

What shipped

The XStack authors announced a benchmark built around repository migration rather than an isolated code translation. An agent receives a PyTorch repository and a clean TPU machine, then must deliver a JAX repository. The artifact says the authors release XStack to make progress on this task measurable. The supplied material does not give a download path or an execution command. [1]

A migration passes when its output meets the authors' grading criteria. Those criteria are based on the source repository's published result, authenticated and calibrated by the authors. That gives a builder a more concrete target than simply receiving JAX files. It does not, by itself, establish that the result meets every requirement of a different repository. The excerpt does not spell out the individual grading checks. [1]

The announced results separate the tested agents: Claude Opus 5 passes 75% of XStack migrations, GPT-5.6-sol passes 45%, and every other tested agent passes at most 40%. These are benchmark pass rates, not measured odds for a particular repository. The supplied material does not give the number or composition of migration tasks, so the percentages cannot tell a builder how often an agent will succeed on their own codebase. [1][2][3]

Why it matters if you build

For a researcher considering a move from PyTorch to JAX on TPUs, XStack frames the work as repository delivery on a clean destination machine. I would use that framing to set an acceptance condition before running an agent: the delivered repository must meet a result-based check, not merely contain plausible-looking JAX code. The benchmark uses the source repository's published result for its grading criteria. A builder still needs to decide whether that target matches the result their own work requires. [1]

The reported spread makes agent choice worth testing, but it does not remove the need to inspect each delivery. Claude Opus 5 has the highest reported pass rate at 75%, which still leaves benchmark migrations that do not pass. GPT-5.6-sol passes 45%, while every other tested agent passes at most 40%. The claims provide no migration duration, compute use, price, or review effort. I would not turn these pass rates into a cost or staffing estimate. [1][2][3]

The clean TPU starting point matters to the workflow: the task asks for a repository that works in the destination environment, not just rewritten source files. If the decision today is whether to automate a real migration, XStack offers a test shape and comparative results. It does not establish that a particular repository will pass. Keep a separate check against the result the source repository publishes, and do not assume the excerpt covers every condition your own project needs. [1][2][3][4]

How to try it today

1. Read the XStack primary artifact for its repository-to-clean-TPU task and pass criterion.
2. Identify the published result for the PyTorch repository you want migrated.
3. Compare your required outcome with XStack's published-result-based grading target.
4. Use the reported pass rates to select agents for evaluation, not to forecast your own result.
5. If you can obtain XStack, test a candidate and inspect its delivered JAX repository against the stated criterion. The supplied excerpt gives no run command.

What we checked

Every claim below was walked down the source ladder. The rung is how close the source is to the thing itself; the tense is the source's own.

1. announced L0 · primary

   XStack evaluates agents that migrate PyTorch repositories to JAX on a clean TPU machine.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
2. announced L0 · primary

   Claude Opus 5 passes 75% of XStack migrations.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
3. announced L0 · primary

   GPT-5.6-sol passes 45% of XStack migrations.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
4. announced L0 · primary

   Every other agent tested passes at most 40% of XStack migrations.

   per arXiv cs.CL/cs.AI/cs.LG (L0)

What is not confirmed

The supplied source does not specify the task count, repository mix, individual grading checks, access instructions, migration duration, compute consumption, or price. It does not establish that a benchmark pass will satisfy a builder's own tests or operational needs. Those remain questions for a local evaluation, not conclusions to draw from the reported percentages. [1][2][3][4]

Who should act, who should wait

Move now if you are selecting an agent for a PyTorch-to-JAX migration on TPUs and can judge a delivered repository against a published source result. Use XStack's task and reported outcomes to structure a trial, then verify the output against your own acceptance criteria. Wait before committing a migration budget or removing manual review on these numbers alone. The supplied material does not establish your expected cost, your repository's likelihood of passing, or the exact checks behind the aggregate rates. [1][2][3][4]

Sources

• XStack: A Benchmark for ML Repo Migration — arXiv cs.CL/cs.AI/cs.LG L0 · primary