Scikit-Learn Matched or Beat Jev's Calibration on BANKING77

本文由「Scikit-Learn Matched or Beat Jev's Calibration on BANKING77」的公开内容整理，完整内容与后续更新请点击文末链接阅读原文。

Here is a message from the BANKING77 test set: "I lost my wallet today with all my credit cards. Will you please block the card and send a replacement?

"I gave it to a logistic regression classifier, calibrated with isotonic regression, that could choose from seven banking intents. None of the seven was "lost or stolen card", so the right answer wasn't on the list. The classifier picked card_payment_fee_charged with probability 0.984. A routing rule of "act above 0.90" would have treated a lost wallet as a fee question and moved on. The same logistic regression before calibration gave the same wrong label at 0.51 and would have escalated it. Calibration made the model better on average and more confident about this mistake.On September 15, 2026, TypeSafe AI launched Jev, a model built for this kind of decision. You send it a state (a support ticket, a transaction record, a proposed tool call) plus a set of typed questions, and it returns structured answers with probabilities: a choice from a list you defined, a score on a rubric you defined, or a yes/no with a number attached. TypeSafe says the model doesn't generate text, so there is no JSON to parse and no retry loop for malformed output [1]. (The API does report output tokens;

they aren't billed.) TypeSafe calls it a System One model, after Kahneman's fast-versus-slow distinction [2], and says it trains with a method called RLCD, reinforcement learning for calibrated decisions. RLHF rewards answers people prefer. RLCD is supposed to reward probabilities that match reality, so that outcomes assigned 0.2 happen about 20% of the time [1].When I ran these tests, TypeSafe had published no paper and no weights, and Jev was available only through its API, with input billed at $0.042 per million tokens and output free [1]. So I asked two questions. Can open tooling reproduce Jev's contract (typed questions, probabilities you can check, and abstention as a branch your code can take)?

And what does the real API add?

I rebuilt the contract on scikit-learn, measured how calibrated small open models are, whether calibration makes a confidence-gated cascade cheaper, and where the typed interface breaks, then ran the same harness against Jev.All numbers here are from my own runs unless they're cited. Details about Jev itself come from TypeSafe's documentation [1], and the cascade costs in Experiment 2 are my assumptions.In short: on the 7-intent slice, a temperature-scaled logistic regression and Jev's Choice answers are calibrated about equally well (ECE 0.022 vs. 0.014;

a paired bootstrap can't tell them apart). On the full 77-intent task, the open model is better calibrated (0.031 vs. 0.065). Jev gets there without training, using at most three labeled examples per intent in the prompt;

my models trained on 10,003 labeled queries. Jev's yes/no answers are underconfident on true cases. The none-of-the-above trap reproduces on open models and on Jev. So does the framing inconsistency, and on both, most of its raw size comes from comparing two differently calibrated numbers at one threshold.How Jev works, and how I rebuilt itJev's API has three question types [1]. A Choice picks from up to 255 options you define and returns the full distribution. A Score takes an ordered set of levels and returns the expectation, which can land between two levels. A Noul (TypeSafe's word for a yes/no truth question) returns P(true). Every answer also carries a confidence value. That number describes how concentrated the distribution is, and it is separate from the top probability.One state goes in. Each typed question (Choice, Score, Noul) is answered in isolation and returns probabilities. Image by the author.I reimplemented the contract over plain scikit-learn classifiers. The question and answer types are dataclasses:pythonfrom dataclasses import dataclass@dataclassclass ChoiceQuestion: name: str instructions: str criteria: dict # option -> description string@dataclassclass ChoiceAnswer: choice: str pro