research · review · first seen Nov 3, 10:32 UTC

NitiBench+ tests Thai legal QA against dated statutes

NitiBench+ was introduced with 1,670 Thai legal QA items across 17 domains and decision-date text for cited sections. For a builder evaluating answers about past decisions, the reported score drops from using the wrong version of a law make temporal retrieval worth testing.

L0 · primary1 source4 claims checked4 min read

What shipped

The paper introduces NitiBench+, built from decisions of three Thai courts. Its 1,670 items cover 17 domains. Each item is grounded in a court decision, so the applicable law has a time boundary. The authors also checked 156,650 decisions of four courts and report that up to 67% cite law since amended or repealed. That broader check explains the problem the benchmark targets; it is not the benchmark’s item count or court count. [1][2]

For each cited section, the benchmark holds the text in force on the decision date and the current text. Its retrieval corpus holds every dated version of every Thai statute section, according to the paper excerpt. This design lets an evaluator distinguish an answer informed by the law applicable to a decision from one informed by current law. The paper calls the in-force text the fair setting. The supplied material describes these components but does not establish how a builder can obtain them today. [1]

The report grades answers on the Thai Bar Council rubric, called the Bar-standard score, and measures overlap between an answer’s cited sections and the court’s. Part I tests the cost of supplying the wrong text to 13 open-weight models. Part II benchmarks models on the in-force-text setting, but the supplied excerpt ends before any usable leaderboard details. I would not infer model rankings from it. [1][2][3]

Why it matters if you build

If I were evaluating Thai legal QA against historical decisions, I would key retrieval to the decision date before interpreting answer quality. In the report’s test of 13 open-weight models, reading current text for amended law lowers the Bar-standard score by 0.25; missing repealed law lowers it by 0.79. Those are reported differences in that test, not promised gains from changing a production retriever. They show why an apparent answer failure needs to be examined alongside the version of the law the model received. [1][2][3]

I would inspect rule changes separately from wording edits. The excerpt reports a 0.45 penalty on 428 rule-changed items and none on 331 wording-only items. It also says Citation F1 does not see that penalty. A pipeline that checks only whether cited sections match the court’s can therefore miss the answer-quality effect of using the wrong version. That is a limit of citation overlap as an evaluation measure, not evidence that citation checks have no use. [1][2]

The immediate workflow change is to preserve the decision date, select the dated section text, and compare the same question with current text before judging the answer. The supplied material does not price that work or say how much engineering it takes. For a system answering only questions about current law, results on historical decisions do not by themselves establish the benefit of a temporal corpus. [1][2][3]

How to try it today

1. Choose a benchmark item and record its decision date and cited section.
2. Read that section’s decision-date text alongside its current text.
3. Run the same question with decision-date text, then with current text, while holding your evaluation procedure fixed.
4. Check the answer on the Bar-standard rubric and inspect its cited-section overlap with the court’s; do not substitute one measure for the other.
5. Repeat separately for amended-law and repealed-law cases, marking rule changes and wording-only changes where applicable.

What we checked

Every claim below was walked down the source ladder. The rung is how close the source is to the thing itself; the tense is the source's own.

1. announced L0 · primary

   NitiBench+ has 1,670 items across 17 domains built from decisions of three Thai courts.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
2. announced L0 · primary

   NitiBench+ holds both decision-date and current text for each cited section.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
3. announced L0 · primary

   Reading current text of amended law drops the Bar-standard score by 0.25 in the report's test of 13 open-weight models.

   per arXiv cs.CL/cs.AI/cs.LG (L0)
4. announced L0 · primary

   Missing repealed law drops the Bar-standard score by 0.79 in the report's test of 13 open-weight models.

   per arXiv cs.CL/cs.AI/cs.LG (L0)

What is not confirmed

The supplied claims and excerpt do not establish a download link, license, installation steps, hosted access, or a working public leaderboard. The excerpt ends mid-sentence at the leaderboard mention. They also do not establish that the score drops generalize beyond the paper’s test of 13 open-weight models, or that a production deployment would improve by either amount. [1][2][3][4]

Who should act, who should wait

Move now on your evaluation design if you build Thai legal QA for past court decisions: make decision-date text an explicit condition and compare it with current text. Wait before committing to a benchmark integration that assumes a download or leaderboard, because access details are not established here. Teams whose questions concern only current law should wait for evidence specific to that workflow rather than applying these historical-decision penalties to it. [1][2][3][4]

Sources

• NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains — arXiv cs.CL/cs.AI/cs.LG L0 · primary