China's open-weight models caught up on tender analysis

In May we benchmarked four models on a synthetic tender and the open-weight entrant, Zhipu's GLM-5.2, came last on reliability: it completed one run in six before dying on malformed JSON. Three months later we ran the same analysis twice on a real bid — once on Claude, once on GLM-5.3 — and the two runs reached the same recommendation, named the same three problems, and finished within 17% of each other on cost. The list prices are 3.6× apart.

Both halves of that matter. The open-weight model is genuinely competitive now, and the money you save by switching to it is far less than its price sheet implies.

What we compared

The document set was a live telecommunications procurement — a mid-size software platform, 94 checkable requirements, 40 of them mandatory — against a real vendor's submitted bid. We have anonymised both parties because the tender is still open. Unlike our May benchmark there was no answer key, so neither run could be scored against a list of planted defects. What two very different engines can be compared on is whether, reading the same documents, they reach the same professional judgement.

We narrowed this comparison to the two stages where the verdict is actually formed: the main reasoning agent, which reads the tender and the bid and writes the findings, and the verification pass, which re-checks each finding against the source documents before it reaches the user. The Claude run used Opus 4.8 for reasoning and Sonnet 4.6 for verification; the GLM run used GLM-5.3 for both, reached through OpenRouter.

The price sheet says one thing

Per million tokens, this is not a close contest:

Opus 4.8 costs $5 in and $25 out; Sonnet 4.6 is $3 and $15; GLM-5.3 is $1.40 and $4.40. On output tokens that is a 5.7× gap against Opus and 3.4× against Sonnet.

The invoice says another

Across the two stages, the Claude run cost $6.64 and the GLM run $5.52 — a saving of 17%, not 72%.

The gap closes because GLM-5.3 needed far more tokens to get there. Claude billed 5.49 million tokens across 147 model calls; GLM billed 9.42 million across 223. Per token GLM was 2.1× cheaper, but it used 1.7× as many, and the two effects very nearly cancel.

Two things drive that. GLM-5.3 reasons much more: 295,000 thinking tokens against Claude's 17,000, a factor of seventeen, most of it in the verification stage. And its prompt caching behaved differently — Claude re-read a cached prefix at a tenth of the input price and paid a premium to write it, while the GLM run recorded 7.8 million cache reads and no cache writes at all. Cache economics are unglamorous and they move the bill more than the headline rate does.

The verdicts agreed

Both runs returned the same recommendation — submit, with improvements — and both independently identified the same three most serious problems in the bid: an incomplete price table that pushed a variable cost outside the fixed sum the tender required, a certificate the bidder declared it held but did not attach, and a large share of mandatory requirements the bidder admitted would not be ready by the submission date. Claude produced 12 findings, GLM 11.

On the first of those, the two disagreed about severity rather than substance: Claude called the pricing gap critical, GLM called it major. Both explained it the same way, and both cited the tender clause it contradicted.

Where they parted

Claude was more complete. It surfaced two soft spots GLM did not mention at all — weak service-level commitments that only began after the first delivery milestone, and delivery dates that were quietly conditional on the buyer responding within five working days. Neither would sink the bid; both are things a bid manager would want to know. Claude also treated a qualification-gate risk as a verified major finding where GLM logged it as an unverified minor one.

GLM was more precise. Where Claude wrote that roughly 17 of about 48 requirements were unavailable, GLM counted the mandatory subset exactly: 27 of 40, split into 12 partial and 15 not yet built, and put the percentage in the summary. It quoted the exact contract sums and the submission deadline. For a document a procurement officer has to defend, that specificity is worth something.

GLM also still broke once. On a single requirement it emitted a 98-kilobyte unterminated string and that check failed outright. That is the same failure mode that killed five of six GLM-5.2 runs in May — but it went from fatal to a single dropped item out of 94. The reliability problem has shrunk to a rough edge.

The part everyone actually asks about

Nobody in European public procurement wants to hear that their tender documents were sent to a Chinese API. The concern is legitimate and it has little to do with model quality. It is about where the documents end up, and whose legal demands reach the company holding them. For a buyer handling commercially sensitive bids, that objection ends the conversation.

Open weights change the shape of that objection, because the model stops being a service and becomes a file. Weights you can download run on hardware you control, or at a European provider you have already vetted. The sovereignty question moves from "do I trust this vendor" to "where do I want to run this," and procurement teams already know how to answer the second one.

Two honest caveats. First, GLM-5.3's weights are not out yet. Z.ai announced the model on 14 August and committed, in the same post, to "release the weights in two weeks after launch, once safety evaluation and hardening are complete." It is equally direct about why the hold exists: "as we scaled post-training, cyber capability developed faster than we expected." As of writing there is no public GLM-5.3 weights repository, and so no licence to read. The predecessor, GLM-5.2, is downloadable today: 753 billion parameters under an MIT licence with no regional restrictions. A repeat for 5.3 is a reasonable expectation, not a fact. Second, our own run did not exercise any of this. We reached GLM-5.3 through OpenRouter, which is a routing choice, not a data-residency solution. Self-hosting is what open weights make possible; it is not what we did.

What this changes

For us the model is a configuration flag — the same analysis pipeline runs on Claude, GPT, Gemini or GLM, and every stage of a run uses the engine you picked, with no silent fallback to another provider. That is deliberate, and this comparison is why it matters: an organisation that cannot send documents outside its own infrastructure should not be stuck with a worse analysis because of it. On the evidence here, it no longer is.

Z.ai's own scorecard claims GLM-5.3 matches Claude Mythos 5 on white-box code review and vulnerability discovery, which is a vendor's own number and should be read as one. Independent reporting puts it at 31.4% on Code Bench against Opus 4.8's 29.5%, and 84.5% on the CyberGym vulnerability task against Claude Mythos 5's 83.8%. The frontier is still ahead — Claude Fable 5 takes that same Code Bench test at 39.5% — but "far behind" no longer describes the gap.

What has not changed is who decides. Both runs read every page of a long tender and a long bid, both found the pricing problem that could get the bid rejected on a formality, and neither of them signs anything. A specialist checks the evidence and keeps the decision. Whether an open-weight model is good enough to prepare that evidence is no longer the interesting question. Where you want it running while it does is.

How we counted

Both runs are single production analyses of the same bid against the same tender, one day apart, with the same prompts and the same tools. The GLM figure is the cost OpenRouter reported for each call. The Claude figure is computed from Anthropic's published rates, including the cache-read discount and the cache-write premium, because first-party responses do not return a price. Both cover the main agent and the verification pass only: the full pipeline also runs document indexing and a per-requirement coverage stage, which cost more than these two stages combined and which the GLM run served with a different model, so whole-run totals are not comparable and are left out here. One run each is a data point, not a benchmark. Our May comparison, with five runs per model against a pre-registered answer key, is the more rigorous measurement.