What we learned
1. The cascade can beat the cloud, because the local model is sometimes right where the cloud is wrong
A cascade can only be as accurate as its two models allow. It gains accuracy when, among the
runs it keeps on device, the local model is right where the cloud model is wrong more often than
the reverse. On ticket triage, the local model (Laya, an open classification model running in
the browser) was right on 93% of messages on its own, and almost always right when it was very
sure. The 141 messages it answered at confidence ≥ 0.984 included 10 that Claude got wrong and
none that Claude got right and Laya didn't.
Held out, the triage cascade gained 3.0 points over Claude, 3.3 over Gemini 3.8 Flash and 2.7
over Jev, while cutting each one's bill by 52–61%. On content moderation it gained 0.7–2.3
points and cut the bill by 78–82%.
2. Calibrate against the cloud, not against a number
Our first calibrations used a fixed 95% accuracy target. On intent detection, Claude alone
reached 99.3%, so the threshold that "met the target" kept 87% of runs on a less accurate local
model: cheaper than the cloud, and worse. That is not what anyone wants from a cascade.
Belay's default target is now 'cloud', the cloud model's own accuracy on your data:
the cheapest threshold that loses nothing. 'max' picks the most accurate threshold,
which is what the examples here use.
3. The choice of cloud model moves the bill by two orders of magnitude
Every cloud call was priced from the tokens it actually used. Per million classifications:
- Claude Opus 5.5: $1,582–$2,301. The most accurate on ticket triage (94.3%).
- Gemini 3.8 Flash: $172–$230, about 9× cheaper than Opus at the same
accuracy on moderation and intents, 2.6 points behind on triage.
- Jev 1.13 (TypeSafe AI), a classification
model that answers typed questions with probabilities instead of generating text: $15–$20,
about 100× cheaper than Opus, and the closest to the moderation labels (83.7% against 76.0%
for both LLMs).
Because Jev reports its own confidence, it can be a cascade tier too. For intent detection, Jev
first and Claude only when Jev is under 0.8 confident scored 99.7%, as accurate as Claude, for
$71 per million runs instead of $2,301. Adding a third tier everywhere (on-device → Jev →
Claude) helped less: on triage it reached 95.3% for $99, between Laya → Jev (95.0%, $8) and
Laya → Claude (97.3%, $967). Jev's confidence doesn't pick out the runs where Claude would do
better.
A cheaper cloud model shrinks what a cascade can save in dollars, not in share: Laya kept
72% of triage runs on device in front of Gemini and saved 61% of its bill.
4. When the local model has no edge, you save money and that's it
On intent detection, Laya was right on every message it was sure about, but so was every cloud
model. The cascade kept about a third of the runs on device at the cloud's accuracy, and on
held-out folds it lost one example in 300 against each cloud. That's the honest result for a
task the cloud already gets right: a 30–37% smaller bill with each single cloud model, and data that stays on the device for a
third of your users' requests.
5. Generation: the on-device model was fine, the judge wasn't
Belay also runs generation tasks: an on-device LLM writes structured output, and a judge model
scores it. We extracted a calendar event (name, date, time, person, place) from 240 voice-assistant
requests with Gemini Nano through Chrome's Prompt API. Nano got 50.8% fully right, Claude
68.3% and Gemini 3.8 Flash 67.5% (the annotations are somewhat noisy, which caps everyone).
The hard part was knowing which of Nano's answers to trust:
- Laya as a judge ("does the answer contain exactly the details in the
request?") gave right and wrong extractions the same scores. The calibration kept about 1% of
runs on device (11% on held-out folds, 1.3 points below Claude).
- A verbatim check (every value appears in the request) passed 95% of Nano's
extractions. Nano copies faithfully; its mistakes are which words make up the event.
- Jev as a judge separated better and kept 12% of runs on device at about
Claude's accuracy.
- Self-consistency (three extra Nano runs per request) showed that Nano's
mistakes are systematic: when all four runs agreed, Nano was right 88 times out of 143, Claude
103 times. Combined with Jev, it kept 22% on device at 67.1%.
For this task, a cheaper cloud model saves more than the cascade: Gemini 3.8 Flash cost $294
per million runs against $4,031 for Claude, within a point of its accuracy. Gemma 4 in Chrome
is the next local model to try; it needs a GPU, which our test machine didn't have.
6. The labels are not the truth either
On content moderation (Civil Comments), Claude called 87 of 300 comments toxic; the labels say
150. The dataset counts a comment as toxic when half its raters did, which includes sharp but
civil criticism. Both LLMs read "rude enough to make someone leave the discussion" as a stricter
bar and scored 76.0%. Jev, a classifier, matched the labels better. Gemini also declined one
comment outright (PROHIBITED_CONTENT), which counts as a cloud error. A calibration
measures agreement with your labels, so write the task's question to match them.
7. The report tells you how to fix the task
The first ticket-triage calibration had 88.0% local accuracy. The report's "confident local
mistakes" table was mostly customer claims routed to account. Adding
"claims against the company" to the feedback option's description raised local
accuracy to 93.0%. (We chose that change by looking at the same examples, so treat the gain as
optimistic.)
8. Built-in AI under automation works, with some effort
Calibration runs the same browser code the app ships, in a real Chrome driven by Playwright.
Getting Chrome's built-in models to download and run there took branded Google Chrome (not
Chromium or Chrome for Testing), removing several of Playwright's default switches, loading
extensions through the DevTools protocol, and a flag to run Gemini Nano on the CPU. The
details are in ADR 0002.
A local model that learns from the cloud
Every escalation pays for a cloud answer, and a fixed on-device model throws it away.
@swissspidy/belay-learn
keeps it: its local model is a small embedding model in the page (bge-small-en-v1.5, 34 MB)
plus a classifier trained on the cloud's answers to earlier escalations and on users'
corrections. It starts by sending everything to the cloud and needs it less the longer it runs.
Its confidence is recalibrated on its own record as it learns, so the threshold keeps meaning
"at least this likely right". Watch it learn ticket triage in your browser.
We streamed 1,000–1,500 held-out inputs per task through it, one at a time, at threshold 0.9,
with 5% of confident runs still sent to the cloud as a check. The cloud is simulated with the
datasets' labels, so these numbers are the classifier's, not a cloud model's.
- On narrow tasks it takes over. Starting from nothing, the device answered
86% of ticket-triage messages at 99.7% accuracy, and 94% of the last 300. Intents went the
same way.
- Where it can't tell, it doesn't guess. Embeddings separate toxic from
civil comments poorly; the device answered about a quarter of them, at 96%. With a teacher as
noisy as Jev it kept nearly all of them in the cloud: it can't look more reliable than the
labels it learns from.
- It notices when the inputs change. Fed all the civil comments first and then
all the toxic ones, it lost its confidence as the toxic ones began and relearned, at 98.9%
accuracy on what it answered.