The model behind an AI procurement analysis is, for us, a configuration flag. The same bid-vs-tender analysis can run on Anthropic's Claude, OpenAI's GPT, Google's Gemini — or, now, an open-weight model like Zhipu's GLM. So we asked the obvious question again, this time with four engines instead of three: if we change the model, what actually changes — in cost, in speed, and in the quality of the verdict?
To find out, we ran our complete production analysis pipeline — the same one behind the "Run Analysis" button — against one realistic tender with two bids, repeated five times per model to separate signal from noise. Every stage runs on the chosen model: domain detection, requirement extraction, per-requirement coverage scoring, the main reasoning agent, and the verification pass. Here is what we found.
The setup
We built one evidence-dense tender with a pre-registered answer key, so quality could be scored objectively rather than by impression:
- A EUR 12M national health-data platform, four lots, with twenty mandatory requirements carrying hard numeric thresholds — turnover, insurance, uptime, data-residency, certifications, disaster-recovery, interoperability, audit retention, and price.
- A compliant bid that meets every requirement and physically includes all of its referenced attachments (insurance certificate, ISO certificates, the bid bond, reference letters).
- A non-compliant bid with 17 deliberately planted defects, from the obvious (turnover below threshold, price over budget) to the subtle: a disaster-recovery region quietly placed outside the EU, a "modern REST API" offered in place of the required HL7 FHIR standard, and a price total that contradicts its own line items.
The four models: Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro, and the open-weight GLM-5.2. Five runs per model per bid; no cross-model fallback — an "OpenAI run" is 100% OpenAI.
Cost: the cheap-per-token model is the expensive one
This was the biggest surprise, and it reverses the intuition. Gemini is the cheapest model per token by a wide margin — yet it was the most expensive to actually run. Median cost of a single analysis (the compliant bid):
Why? Gemini is token-hungry. Across a run it pushed several times more tokens through the pipeline than Claude or GPT — re-reading the documents again and again as it scored each requirement — and that volume swamped its low per-token price. Over both bids the gap widens: a full tender (both bids) cost about $3.02 on Gemini, $2.76 on Claude, and $1.62 on GPT. The open-weight GLM-5.2 — at roughly $0.94 in / $3.00 out per million tokens via OpenRouter — was the cheapest engine of all.
Catching the bad bid: thoroughness splits the field
On the non-compliant bid every model reached the right verdict — "do not submit," every time. What varied, sharply, was how much of the evidence each one actually surfaced. Recall of the 17 planted defects:
Claude and GPT were flawless detectors — all 17 defects, in all five runs, including the subtle ones. Gemini, the token-hungriest and priciest engine, was also the least thorough and the least consistent: a typical run caught about 13 of 17, and across runs it swung from 10 to 17. It most reliably missed the quiet omissions — a missing source-code-escrow clause (caught in only one run of five) and an over-long implementation timeline. A procurement officer relying on Gemini still gets the right "reject," but a thinner, less defensible evidence pack behind it.
Approving the good bid: no false rejections
On the compliant bid the right answer is to shortlist it, and here the models are reassuringly aligned: every one approved the good bid on every run, and none wrongly rejected a fully qualified vendor. The bid meets all twenty requirements and includes every supporting document, and the models read it that way.
The difference is temperament, not correctness. Gemini and GLM gave the good bid unqualified approval. Claude and GPT more often returned "submit, with minor improvements" — flagging a couple of genuine soft spots (a reference project at the low end of the range, a timeline with no slack) as things to tighten, not as reasons to reject. Both behaviours are defensible; the cautious one arguably gives the procurement officer more to act on.
How much each model even saw
One layer the headline verdict hides: before scoring anything, each model reads the tender and extracts a structured requirement ledger. From the same RFP, they built ledgers of very different granularity — which sets the ceiling for what can be checked downstream.
GPT split the document most finely, into 54 line items; Claude and Gemini landed near 36; GLM at 32. More items is not automatically better — it can mean genuine granularity or needless fragmentation — but it explains a lot about downstream behaviour, and it is the kind of thing a single end-to-end run never shows you.
The open-weight contender: cheap and capable, until it isn't
GLM-5.2 is the interesting newcomer. On the straightforward compliant bid it was a star: cheapest of the four, correct every time, and it extracted its own requirement ledger without trouble. Open-weight models have clearly closed much of the gap.
But it hit a hard ceiling on the complex job. On the defect-heavy bid — where the agent has to record many findings in structured form — GLM-5.2 completed only one run in six. The other five died mid-analysis: the model produced malformed JSON (a dropped comma, an unterminated string) while emitting its large list of findings, and the run could not recover. When it did finish, it caught all 17 defects — so the reasoning is there; the reliability of its structured output under load is not. For an agentic pipeline that depends on well-formed tool calls, that is the difference between a usable model and a frustrating one.
Across every valid run, all four models rejected the bad bid and approved the good one. The verdict is the dependable part. Thoroughness, cost, and — for open-weight models — reliability are where they diverge, and where the choice actually matters.
There is no single winner — there's a trade-off
Claude Opus 4.8 — the most thorough and consistent: 17/17 defects every run, and the most measured read on the good bid. Mid-pack on cost, mid-pack on speed.
GPT-5.5 — the best all-rounder: 17/17 detection, the finest-grained extraction, the cheapest of the three frontier models, and fast on the lighter job. A clean reliable default.
Gemini 3.1 Pro — fast on the heavy bid, but the catch is real: the most expensive to run despite the lowest per-token price, and the least thorough and least consistent detector. A capable engine, not the one to trust as a sole reviewer of a high-value tender.
GLM-5.2 (open-weight) — cheapest, and genuinely good on simple cases, but it could not reliably complete the complex, structured analysis. Promising for high-volume first-pass triage; not yet dependable for the hard cases.
The reasoning models have largely converged on the verdict. Where they still differ — sometimes by 2× on cost, or four missed defects out of seventeen — is in the work behind it.
What this means if you use AI for tenders
Two lessons. First: the dependable layer is now the verdict at both ends — reject the bad bid, approve the good one — but the evidence behind it still varies enough that a human stays on the recommendation. Second: model choice is a deliberate trade-off, not a search for one universal best. The thorough ones are worth their price when a missed defect is expensive; the cheap-per-token option can quietly become the costly one; and an open-weight model can be excellent value right up to the point where its output reliability gives out.
None of which removes the person signing the recommendation. It sharpens what they do: the AI reads every page of every document and never tires on page 200, and the specialist checks the evidence and keeps the decision.
How we ran it
- One synthetic EUR 12M tender, four lots, twenty mandatory requirements, with a pre-registered key of 17 planted defects in the non-compliant bid; the compliant bid includes all referenced attachments.
- The full production pipeline per run: domain detection → requirement-ledger extraction → per-requirement coverage scoring → main reasoning agent → verification of every critical finding.
- Every stage (extraction, coverage, main, verification) ran on the selected model — no cross-model fallback.
- Five runs per model per bid, reporting medians and ranges. Recall and false positives were scored by an independent LLM judge against the pre-registered key.
- Limitations: a single synthetic English tender in one domain; GLM-5.2 completed only one non-compliant run of six (malformed-JSON failures), so its detection figure there is a single data point. One operational note: OpenRouter "presets" are scoped to the account whose key issues them — the same preset name can route to a different endpoint under a different key.
Try your first AI analysis for free.