ticket-triage

Accuracy and local share by threshold

Runs with local confidence at or above the threshold keep the local answer; the rest go to the cloud. Hover for values.

Cloud cost per 1,000 runs by threshold

Savings projection

At the recommended threshold, assuming production inputs look like this dataset.

Local confidence distribution

Examples per confidence bin, split by whether the local answer was correct. A well-calibrated model has few wrong answers in the high bins.

Local confusion matrix

Rows are true labels, columns are local predictions (all examples, before escalation). Shade shows the share of the row.

Per option

At the recommended threshold.

Confident local mistakes

Wrong local answers that would still be accepted at the recommended threshold, most confident first. Good candidates for better option descriptions.