- Ticket routing
- Moderation pre-filter
- Review sorting
- Document triage
The detail behind “Judgement is becoming cheap”
How a judgement model works, what I measured, where it fails, and a free check on a decision of your own. The short version is the executive summary.
Companies ration judgement because a person has to do it.
Most checks happen on a sample, on a complaint, or when someone opens the file. That made sense while each judgement took minutes of a skilled person's time. Judgement models now answer a question with fixed answers in about half a second. On short texts I measured about $17 per million decisions, on full contracts about $90.
Jev, released by TypeSafe in September 2026, is the clearest example I have tested. It is one tool among several, and the method on this page works with whichever model wins on your cases.
Decisions nobody makes today
Run a judgement on every case and a company can judge what it never judged, act before the customer asks, and sell what was uneconomic while every case needed a person.
See the four rungs →Four rungs, each resting on cheap judgement.
Efficiency sits below the ladder. Each rung above it changes more of the business and needs more than a model.
- 4Sell something new
A service, a guarantee or a product that was uneconomic while every case needed a person.Possible, not yet measuredExample: an energy supplier that guarantees every bill sits on the customer's own meter and reading, and pays automatically when it does not.
- 3Reorganise around continuous judgement
Teams that worked from separate queues work from the same judgements, with decision rights and a named owner for each threshold.Possible, not yet measuredExample: a business service level where every paused ticket is checked against its note as it happens, and the credit it earns is computed and paid without a dispute.
- 2Act before the request
You know what is wrong before the customer does, and the fix goes out before the complaint. The service desk keeps the hard cases.Possible, not yet measuredExample: a fault report checked for completeness the moment it arrives, so the technician is sent once.
- 1Judge every case
Every case is scored as it happens, instead of only when someone opens it. Example: every product pair a pricing tool compares, every review a firm receives, every note an assistant might quote.
In one workflow I inventoried, 32 judgements were listed and six to eight qualified. See the inventory.
Who reports results already
Four companies, in their own figures. Three of the four are the efficiency play.
Unblocked
Their assistant answers engineers' questions from internal notes. Jev decides which notes belong in each answer.
More of the right notes and fewer wrong ones, in production, for under two dollars per thousand questions.
SourceFurtherAI
An AI agent fills in insurance quotes on broker portals. Jev picks each next step.
A quote that took 18 minutes now takes about 5, and the setup is moving into production.
SourceCinchOps
A small IT firm sorts customer reviews and search queries by topic.
5,300 reviews sorted for 14 cents instead of 50 dollars, as accurately as the expensive model.
SourcePricogni
A pricing tool matches competitors' products so prices can be compared.
9,081 product pairs checked for 32 cents. The unclear ones go to a person.
SourceStart with what arrives in volume.
Pick what your teams receive most. Each type hides a few decisions that someone makes on every case. Pick one, put in your own numbers, and see what judging it costs today.
Decisions inside this type
Pick the one closest to what your team decides.
What does this judgement cost today?
What I measured, as a guide and not a promise: 82% of 300 news articles at a 0.90 threshold, 92.7% of those right. About a third of 300 full-contract checks at 5% errors or fewer. Your share comes from a test on your own cases.
Does this decision suit a judgement model?
Tick what is true. A decision that meets most of these is a good candidate. Where one fails, the line below it names the cheaper or better tool.
Runs the free check below with Jev and a chat model. To send this decision to me directly, use “Send me one decision” in Work with me.
Test this decision
Free, and the entry point before a Decision Discovery. Jev and a chat model answer the same framework questions on your description, live.
Text in, a probability per answer out, and a threshold decides who acts.
How it answers, next to a chat model
The same fault report and the same question go to a chat model and to Jev. Both clocks run in real time.
A ticket, a record, an article, a contract.
Written once, with a fixed set of answers.
One number per answer. The answers add up to 1.
Set from what an error costs you in each direction.
Above the threshold the software acts. Below it, a person decides.
Watch one decision end to end
A support ticket, three typed questions, the answers as numbers, and a threshold that routes the uncertain one to a person.
How many cases run on their own?
It depends on the threshold. In my test of 300 news articles, 246 were at 0.90 or higher and ran automatically, 92.7% of them right. The other 54 went to a person.
At 0.90 the software handles 82% of the articles, right on 92.7% of those. Your threshold depends on what each error costs.
What changes in the queue
The number sorts it: urgent and clear first, uncertain to a person, routine runs on its own.
One step in four
Where the judgement sits between the writing and the rules.
Prose and scans are handled before the judgement.
Cheap enough to ask on every case.
about half a secondExact and auditable, so they stay in code.
Below the threshold, a person decides.
How you ask moves the number more than which model answers
One customer with three recent complaints and a contract ending soon, asked 20 times each: will this customer cancel within 90 days?
The largest gap is between the two Jev rows. The question form changed the number more than swapping the model did.
That is why the design is the asset. The model is rented and can be replaced. You keep the decision design: the question, its answer set, the threshold, your labelled cases and the pipeline around them.
What TypeSafe says about Jev, and what I measured.
In bold: how TypeSafe describes Jev. Below each point: what I measured from Europe in September 2026. One limit comes from independent tests.
- 20 to 200 times faster.Measured: 0.5 s against 2.0 s for a large chat model, 4 times faster. An independent banking test found 13 times. The servers are on the US west coast, so distance counts.
- 40 to 1,000 times cheaper.Measured: about $17 against $87 per million decisions, 5 times cheaper. The banking test found 22 times.
- Calibrated and consistent: it says when it is unsure, and similar inputs give similar outputs.Measured: 0 changed answers in 200 identical requests. When it was 90% sure or more, it was right in 92.7% of 246 cases.
- Always a valid answer, several questions at once.By design the answer is one of your options, so software never has to read text. Measured: 8 questions on one fault report in a single request took 0.40 s, as long as 1 question. 40 questions took 0.44 s. The answers matched those asked alone.
- Close to frontier models on instinctive judgement.TypeSafe calls this its hardest claim to prove. Measured against a large chat model: 89% against 87% on news, 80% against 84% on full contracts.
- Not a chat model. It writes nothing and gives no reason. You define the shape of the answer, like a multiple-choice question.What to do: split a decision into several questions and keep each answer as the record. They run in one request without extra waiting.
- Weak at step-by-step reasoning such as maths or chess.Measured: also weaker on full contracts, 80% right against 84% for a large chat model. Keep sums and dates in code.
- No deep knowledge of niche domains.Put the needed context into the question. TypeSafe offers to train Jev on your own cases.
- Calibration drops on unfamiliar tasks. Not on TypeSafe’s listIndependent testers found it 2 to 4.4 times less reliable. Check about 50 of your own cases before you set a threshold.
- Served from the US west coast only.Personal data needs pseudonymising first, or an open model on your own servers.
All limits and what to do about each: Where it fails.
Measured on the same cases against the alternatives.
I ran Jev, a large chat model and a free model on your own hardware on the same public test sets, with the same instructions. Company results elsewhere on this page are their own figures.
My measurements, September 2026. Costs in US dollars as billed. Trained classifier accuracy from the ContractNLI paper; its cost and time were not measured. The free model runs locally, hardware not counted.
What others measured since the launch
Independent tests from the first two weeks after the release on 15 September 2026. No method wins everywhere, cascades save the most, and confidence has to be checked on your own data.
77 banking intents, 3,080 cases. Large chat model 84.4%, Jev 81.0%. Jev answered 13 times faster and cost about $0.11 against $2.42 per 1,000. OpenRouter, 22 Sep
A cascade on 8-way routing. Jev decided when at least 80% sure (80.6% of cases), the rest went to a large chat model: 90.0% against 89.4% for the large model alone, at 25.7% of its cost. ayautomate, 20 Sep
Yes, no or maybe questions. Statistically tied with the best of six chat models, at 1/28 of the price. Jevals, 18 Sep
Legal texts (LexGLUE). Jev 69.9 against a chat model's 71.3, at a quarter of the cost. A trained classifier beat both with 77.4. jev-test
Search result ranking. Jev alone did no better than similarity search. The two combined won clearly. rerank eval
Phishing emails. A small chat model won, 81.3% against Jev's 62.6%. phishing bench
Is the confidence honest? Well calibrated on one intent set, overconfident on another, and zero malformed answers. ASSAY-001
What it cannot do, and what it does badly.
Eight questions procurement, IT and data protection ask first. Tap a row for what it means and what to do.
Open weightsYou cannot download the model or run it yourself. The connectors are open.NO
You depend on one supplier for pricing, availability and model changes. Procurement and risk will ask about it.
Plan an exit path: the same decision run by a chat model or a small trained classifier at a known cost, behind your own interface. About 31 open reproductions appeared within a week of launch; the best trails by about 6 points on unseen tasks. An open model with the same interface, Laya, followed in September 2026. I have not tested it yet.
Self-hosted or on-premiseHosted service only. Nothing local announced.NO
Case text leaves your network.
Open judgement models with commercial licences run on your own servers. In published tests they automate somewhat fewer cases than the best hosted model, so I measure the gap on your cases.
EU data residencyProcessing in the United States only. Zero data retention needs an enterprise contract.NO
Personal or regulated data that must stay in the country cannot go to Jev as it is. That covers EU customer data and regulated sectors such as health in the Gulf.
Start with data that is not personal, such as telemetry and logs, or pseudonymise first. Otherwise use an EU-hosted open model. Ask about residency, retention and version pinning in the first conversation.
Reads PDFs and images directlyNo images, audio or PDFs. Extract the text first.NO
The price covers the judgement only. For PDFs and scans the extraction step costs more.
Price the whole pipeline. Cases that are already text or records, such as tickets, logs and CRM fields, have the best economics.
Explains its answerIt returns probabilities, with no written reason.NO
When a customer or a regulator asks why, the model gives no sentence. For decisions about private customers with legal or similar effects, the GDPR asks for meaningful information about the logic and a way to reach a person.
Split the decision into several typed questions, such as line test present, access window usable and line ID valid, and store each answer with its probability. The stored answers are the explanation. A chat model’s written reason reads well, and it need not be what drove the answer.
Generates textIt picks one of your answers. It does not write.NO
Drafts, summaries and replies need another tool.
Let a language model write and extract, and give the judgement model the question (see section 04, one step in four).
Fixed-choice decisionsPicks from 2 to 10 answers you define, in one pass.YES
It always picks one of your options. Without a way out, off-topic cases were never flagged: 0 of 30 out-of-scope messages, at 0.99.
Add "cannot tell" or "none of these" to every answer set.
Probability outputOne probability per answer, repeatable: 0 changed answers in 200 identical requests in my test.YES, CHECK IT
Reliable on public tests, about 2 to 4.4 times worse calibrated on tasks the model has not seen.
Label about 50 of your own cases per question and correct the probabilities before you set a threshold.
Production readyReleased on 15 September 2026. In production at one company in its own figures. New sign-ups paused on 23 September 2026; access through OpenRouter continued. The default version changes underneath you.EARLY
Supplier risk is higher than for an established API.
Pin a model version, set the threshold on that version, and keep a fallback that sends every case to people.
Regulatory points are general; check them per country and per client.
Where the judgement goes wrong
Prove the numbers on your own cases before anything is automated.
Four steps, the same for both plays. The same steps run in a Decision Test (see section 08).
-
1
Run a baseline
A public test set goes through your setup first. A result far from mine means the setup is wrong.
ProducesA test setup you can trust -
2
Test cases that fit no option
Feed cases whose right answer is not on your list. A confident answer there makes automation unsafe.
ProducesHow often it is confidently wrong -
3
Check it on your own data
Label 50 to 300 of your own cases and compare the prediction with what happened. Run a free similarity model and a chat model on the same cases. If the free model comes close, the decision is sorting and you may not need a judgement model for it.
ProducesAccuracy and a reliability plot for your cases, next to the baselines -
4
Set the threshold from error costs
Decide what a wrong answer costs in each direction and set the cut-off from that.
ProducesThe threshold and the share decided without a person
What a decision maker holds
See methodology
M1: Why not ask a chat model for a percentage?
A chat model is trained with RLHF to write answers people like. A percentage is text it writes on top of that, and it is often too sure of itself. TypeSafe trains Jev with a method it calls RLCD, reinforcement learning for calibrated decisions. The aim: when Jev says 70%, it is right about 70% of the time. It writes no text and weighs your fixed options in one pass. TypeSafe has not published the details. On my news test a current chat model, asked outright for probabilities, was as well calibrated as Jev, so the gap depends on the task. Open copies with the same interface exist. One running on an ordinary chat model got 86.7% right, but said 99% or more on 82% of the articles and was still wrong on 20 of them.
M2: News test, accuracy, cost and time
Models and version: GPT-5.6 Luna, GPT-5.6 Terra, jev-1.13-20260917. Simple Jev, an open copy, reached 86.7% and 86.0% on the two backing models. Accuracy on 300 cases with 95% ranges: Jev 89.0% (85% to 92%), Luna 87.3% (83% to 91%). They got 30 of the same 33 articles wrong, which points at unclear articles and label errors in the public set. Calibration error (ECE, lower is better): Jev 0.076 against Luna's 0.045. Cost per case as charged by OpenRouter: Jev $0.000017 at about 410 input tokens, Luna $0.000087. Time per call from Europe: Jev 0.49 s, Luna 1.95 s.
M3: Free similarity model and judgement pairs
qwen3-embedding 0.6B on Ollama, cosine similarity between the text and each option, softmax for confidence. News: 86.3% with an instruction prefix, 73.3% without. Judgement pairs: 57.5% with the prefix, 62.5% without, 71.5% with the best option wording tried; with the prefix it gave both texts of a pair the same answer in 85 of 100 pairs.
M4: Full contracts and chat models on contracts
300 rows of ContractNLI (full NDAs, 17 fixed statements, entailed, contradicted or not mentioned). Jev 79.7% with no training. The free model reached 47.7%, below answering each statement with its most common label (67.3%). The ContractNLI paper reports 83.8% and 87.5% for fine-tuned BERT base and large on the full test set. Jev decided 34% of the contracts at 5% errors or fewer. GPT-5.6 Luna asked for a probability per option reached 84.0% at $0.00072 and 2.8 s per contract, against Jev's $0.00009 and 0.44 s: about $90 against $720 per million contract checks.
M5: Time and cost across settings
Time from Europe: Jev 0.49 s; TypeSafe quotes about 0.1 s near its US servers; an independent US test saw 0.13 s. List price $0.042 per million input tokens, output free: about $6 per million at TypeSafe's 150-token example, about $17 at the measured 410 tokens.
Several questions at once, measured on 29 September 2026 with one fault report, medians of five rounds: 1 question 0.38 s, 8 questions in one request 0.40 s, 40 questions 0.44 s. The same 8 questions as separate requests took 3.3 s one after another and 1.0 s all sent at once. The case text is billed once per request: 8 questions together cost about $38 per million cases, as 8 separate requests about $129. The answer to the main question was the same alone and in the group of 8 (send back 0.84 to 0.88). Script and raw data: results/parallel-questions-2026-09-29.
M6: Repeatability
0 changed answers in 200 identical requests here; probabilities moved by 0.01 on clear cases and 0.03 on an uncertain one. Other testers report 1 to 2% flips. The chat model moved 7 to 10 points on the uncertain case in section 04 and none on the clear one.
M7: Classic method against labelled examples
TF-IDF plus logistic regression, same 300 articles. The classic method is free; its price is the labelling.
M8: TypeSafe's own chart: accuracy against cost
TypeSafe's published comparison, not measured by me; reference labels from two frontier models, with no human labels. The cheapest model on the frontier; the best models score 5 to 6 points higher at 200 to 375 times the cost.
M9: The confidence value
Each response also carries a third value, confidence, which measures how far the winner sits from the rest. It is computed from the probabilities and adds no information of its own; yes/no questions return only the probability. Independent audits found the confidence value is an exact function of the top probability, so threshold on the probability.
M10: Limits in detail
Access: no downloadable weights, no public checkpoint, no offline binary. TypeSafe paused new sign-ups on 2026-09-23; access through OpenRouter continued. jev-latest is an alias that changes underneath you; my runs reported jev-1.13-20260917. On an inference benchmark German lost 5.5 to 6.8 points against English and cost about 3 times the tokens. Calibration is about 2 to 4.4 times worse on tasks the model has not seen, with low probabilities overstated and high ones understated; about 50 labelled cases per question and a one-line correction bring it back. Tune the threshold first; one production team found a threshold sweep did what nine prompt variants could not.
M11: Sample sizes and baseline
My baseline reached 89.0% on 300 AG News articles; a model calibrated on its own evaluation set says nothing about your taxonomy. For a rare outcome you need about 300 cases per class, with at least 100 of the rare one. Tune the threshold before the prompt, and pin the model version the threshold was set on.
M12: Telecom requests
300 requests from bitext/Bitext-telco-llm-chatbot-training-dataset (synthetic customer requests written for chatbot training, CDLA-Sharing-1.0), shuffled with Random(7). Question: “Which team should handle this customer request?” with seven teams (billing, complaints, consumption, contact, payment, services, subscription), the same short description of each for every model. Jev 95.3% (92% to 97%), calibration error 0.007; at 0.90 it decided 90% of the requests with 98.9% right; $19.58 per million, 386 ms median. GPT-5.6 Luna asked for a probability per team: 97.7% (95% to 99%), calibration error 0.066; at 0.90 it decided 75% with 100% right; $144.28 per million, 2373 ms median. Free similarity model (qwen3-embedding 0.6B, local): 83.3%, but a mean top probability of 0.16, so its numbers cannot carry a threshold. Jev’s most confident errors sat at two category boundaries: raising a minutes allowance (labelled consumption, answered subscription) and reporting poor coverage (labelled complaints, answered services); the chat model made the first of these errors too. Script and raw data: live/telco_bench.py, live/telco_embedding.py, results/telco-routing-300.json.
Every comparison and calculation on this page was run by me. Results of other companies are their own figures. Raw responses are stored in the lab.
Decision engineering, from the first workshop to a live decision.
I find the decisions worth automating, design them, test them on your historical cases, calculate what they are worth and get them into production. I am tool-neutral: the method that wins on your cases gets the recommendation, including free ones.
-
1
Decision Discovery
90-minute workshop
With the people who own a process. We go through what arrives in volume and who decides what on each case.
- You get: 3 to 5 decisions worth testing, each with its question, answer set, volume, where the outcome is recorded, and whether it is efficiency or doing things differently.
-
2
Decision Test
1 to 2 weeks
50 to 300 of your historical cases with known outcomes. Jev and the baselines run on the same cases: rules, a free similarity model, a chat model, and a trained classifier where labels allow.
- You get: accuracy, calibration, the threshold, the share decided without a person at your error rate, the economics per year, and a go or no-go in writing.
-
3
Production Pilot
4 to 8 weeks
One live decision, integrated into your system. A shadow run first, then a share goes live. Every uncertain case falls back to a person, and a sample of automated cases is checked each month.
- You get: a running decision, its records, and a production recommendation.
I reply within 48 hours with the method I expect to win. Free, and no customer data is needed at this stage.
Data, works council and IT security
By default the test uses methods that run inside your network: rules, local similarity models, trained classifiers and open judgement models on your own servers. Hosted models such as Jev are added only after your data protection officer approves the provider, the region and the processing agreement. Every report shows which method ran in-house and which sent case text to a hosted provider. The shadow run records system decisions only; the performance of individual staff is out of scope.

Strategist and builder for AI and new ventures, with startup experience. I work inside your team, from the first question to a process that runs in production. Email: peter.milan.trapp@gmail.com, LinkedIn
- Tested independently. Every comparison and calculation on this page was run by me. Results of other companies are their own figures.
- Pilots and workshops in several industries on where cheap judgement changes decisions, processes and teams.
- A workshop kit inside the lab, so a team can run the inventory without me.
More from the lab: IndustriesWhat people took from this