LLM benchmark on procurement work

Method

A stack is the set of models assigned to the three agent roles: main, mid and fast. Different steps of each agent call different roles. A stack may use one model for all three roles or a different model per role.

The corpus is one municipal school renovation tender with two lots, in parallel English and Latvian versions: regulations, a technical specification, a bill of quantities, a draft contract, a drawing and two bids. One bid is compliant. The other contains 22 planted defects; some appear only in the spreadsheet, some only on scanned certificate pages.

For each stack, language and trial, the benchmark runs requirements extraction on the RFP once, bid analysis once for each of the two bids, and market research on the tender's items once. It uses the same jobs, sandboxes and tools as production, with one exception: every catalogue item is approved automatically so market research can start without a human step.

Bid analysis is scored on planted-defect recall (50%), precision against false alarms on either bid (30%) and verdict correctness (20%); the trial score is the lower of the two bids. Requirements extraction is scored on reference recall (60%), precision (20%) and the lots and criteria structure (20%). Market research is 40% structural (run completed, share of items with at least one quote, share of quotes with a validated source) and 60% a rubric of facts the report must contain. With more than one trial, each activity score is the median across trials. The language composite is the mean of the three activity scores; the overall composite is the mean of the English and Latvian composites.

A separate model reads each output against the answer key in a worksheet; the scorer then applies the fixed weights above. The judge is never one of the models under test, but it may come from the same vendor. The judge for each stack is listed in the stacks table.

Cost is LLM usage only. First-party calls are priced at the provider list price on the run date, with cache reads, cache writes and thinking tokens weighted as the provider bills them. OpenRouter calls use the cost OpenRouter reported. Wall-clock is the time from driver start to finish for both languages, including setup and queue wait.

Reliability is completed activity records divided by the number of attempts; a retry that later succeeds counts as two attempts and one completion. A stack with one trial is a single sample.