Ask the same question to four frontier models at once. Watch the real dollar cost land per answer, then watch two independent AI judges, blind to which model said what, score every response on the same rubric. Nothing here is pre-recorded.
Every model rankings list is a sticker price and a benchmark score. Neither tells you what actually happens when you ask your own question. This runs four current frontier models (Claude Fable 5, GLM 5.3, Kimi K3, and Mistral's brand-new Large 4) against whatever you type, live, and shows the real per-token cost OpenRouter actually billed for each answer.
Then it gets judged. Not by us. Two independent "decision model" judges, Jev from TypeSafe and OpenAI's brand-new Decisions API, each score every answer on the same five-point rubric with a confidence number attached. Neither judge sees which model wrote which answer.
Decision model: A model built to answer one typed question (yes/no, pick-one, or a numeric score) with a calibrated confidence, not to hold a conversation.
Confidence: How sure the judge is in its own score, separate from the score itself. Low confidence on a high score is a real signal, not noise.
Real cost: Pulled from OpenRouter's generation ledger after the call completes, not estimated from list pricing.
Each judge answers two plain questions about every response, each as a percent with a one-word verdict, plus a note on how sure the judge is of its own answer.
"How good was the answer?" did it actually get it right, cover the question completely, and explain it clearly? This is the one most people mean when they say "how good was the answer."
"Was it honest about uncertainty?" separate question. Did the answer admit uncertainty or correctly say "that's not possible" where that was the right call, instead of confidently making something up? A short, correct "no" should score well here even if it's not a long answer.
The italic note under each one ("the judge is confident in this," "the judge is basically guessing here") is NOT part of the grade. It's how sure the judge is in the grade it just gave, judged on itself. A judge can hand out a decent-sounding score while basically admitting it isn't sure. Take that note seriously, a score the judge itself is guessing at is a rough signal, not a verdict.
Both judges land in the same neighborhood on "how good was it", a decent answer, not a disaster. They disagree more on "was it honest", and here's the real tell: neither judge trusts its own honesty grade. Jev admits only 12% sureness, OpenAI Decisions admits basically 0%, a coin flip. When a judge's own sureness collapses like that on one question but not the other, it usually means the answer itself was genuinely ambiguous on that point, and the judge is telling you that honestly instead of faking certainty.
Run at least one question to see this.
Click any question below to reveal all four answers side by side, with full scores.
Nothing logged yet.
TypeSafe AI's "System One" decision model, built by a team that includes an ex-OpenAI instruction-following researcher, backed by a $40M seed round. It only reads text or JSON, nothing else, and that narrowness is the whole point: no prose, no chat overhead, just a fast, cheap, calibrated answer to a question you define in advance.
Under the hood: not a chat model in a smaller costume. Jev runs a non-autoregressive, energy-based architecture, it scores an answer directly instead of generating it token by token. The tradeoff: a 64K-token window, far smaller than OpenAI's.
Why it won here: cheaper every single time, and on the one answer we could actually verify, it got the grade right.
Its real limit: show it a photo and it can't even start. Text and JSON only, nothing visual, not even a little.
OpenAI's direct answer to Jev's category, same three question types (yes/no, pick-one, numeric score), but it can read images as well as text. Show it a photo, a scanned form, a screenshot, and it can judge that directly. Jev cannot do this at all.
Under the hood: a distilled 1.8B-parameter head bolted onto the real GPT-6 Luna, inheriting its full 1.1M-token context window and able to answer up to 200 questions about the same content in a single call. Bigger and more versatile than Jev on paper.
Why it lost here: about double Jev's price per judgment, and on the one checkable question, it scored a correct, well-reasoned answer as barely passing, a real calibration miss.
What this test didn't show: we never once handed either judge a picture of an orange and asked "is this an orange." That's the actual edge this product has, and this experiment never touched it. On text-only judgment, right now, it just looks less calibrated than Jev.
1 trillion total parameters, only 49 billion active per token (mixture-of-experts), natively multimodal, and ranked top-five globally on cybersecurity reasoning benchmarks at launch. The biggest model in this whole lineup by parameter count, running at a fraction of the price.
Why it won here: dramatically lower cost per answer, over 1,400 quality-points-per-dollar during launch pricing, while its actual answer quality stayed close enough to the others not to matter.
The honest caveat: that launch pricing is 50% off and won't last. Even at full price it's still the value leader by about 2x, just not the blowout it is today.