What Are Decision Models? What We Learned Putting Jev to Work

A lot of what our agents decide comes down to a label. Is this urgent, which thread does this note belong to, did this report get its facts right? We've been paying large models to answer those questions in prose and then parsing a label out of the reply.

Decision models skip the prose. You send some state and a few typed questions, and each answer comes back as a probability, with no generated text. Jev, from TypeSafe, launched in mid-September 2026. Cloudflare, OpenAI, Microsoft, Perplexity and Strands Labs have since shipped their own.

We've run Jev in production since October 7, on one job. We've built five more and left them switched off. This note covers what these models are, what we ran, where Jev broke, how it did on our hardest test and what else exists.

Here's the short version. Jev got nearly all of our closed, plain-language examples right for a fraction of a cent. It accepted wrong counts with high confidence. As a judge of agent reports it missed our 85% bar at 64.4%, and its confidence scores weren't reliable for that job.

What a decision model is

A decision model sits between your rules and your LLM. If code can answer the question, we use code. If we need reasons or prose, we still need a generative model. In between are the closed questions, and there are a lot of them.

There are three question types. The names vary by vendor (OpenAI calls a Noul a predicate), but the shapes match.

One request can ask several questions about the same state. Here is a real one from our support triage code, cut down to two of its four questions, followed by what Jev 1.14 returned:

{
  "model": "jev-1.14.0",
  "state": {
    "subject": "Charged twice for October",
    "message": "Hi, my card was charged twice this month. Can you refund the extra one? Thanks.",
    "channel": "email"
  },
  "questions": {
    "ticket_type": {
      "type": "choice",
      "instructions": "What is this support ticket about?",
      "criteria": { "billing": "Charges, invoices, refunds...", "technical": "...", "account": "...", "general": "..." }
    },
    "human": {
      "type": "noul",
      "instructions": "Should a person on the support team handle this ticket rather than the automated resolver?",
      "criteria": { "true": "It asks for a refund, a cancellation...", "false": "A routine question with a standard answer." }
    }
  }
}

{
  "ticket_type": {
    "type": "choice",
    "choice": "billing",
    "confidence": 1.0,
    "probabilities": { "billing": 1.0, "technical": 0.0, "account": 0.0, "general": 0.0 }
  },
  "human": { "type": "noul", "noul": 0.91 }
}

There's nothing to parse, and the code branches on a number.

Jev in one table

The weak spots come from TypeSafe's own notes on 1.13, which we read before writing our questions. On numbers, they say "Jev is not a calculator".

What we built on it

We wrote questions for six jobs, all behind one seam in our code. The caller asks, and if Jev is off, slow or unsure, it gets nothing back and keeps its existing path.

Only repair triage is on, because it's the only job that sends no customer text. Everything else waits for the data terms to cover it.

Here is what repair triage has done so far.

Our rules table couldn't classify 6 of those 13. Jev cleared the 0.70 line on 2 of them, both configuration problems, and gave no class on 4, so those remain unknown. On the 7 our rules had already classified, it gave a class on 3, and all 3 matched. Each record still says verified: false, so this shows it's running. It doesn't show it's right.

Before that, in September, we tried note placement on 12 synthetic notes through a local database. All 5 placements it made were correct, at 0.88 confidence or higher. It also caught a wifi password that our regex let through. All 18 calls cost $0.0007.

Worked examples

To show the shapes, we ran made-up inputs through our real question code: three times each, on both 1.13 and 1.14. No customer text was involved. The numbers below are Jev 1.14's medians, and the verdict column applies our thresholds. For Choice questions we show the confidence field, which is what our thresholds read. These are illustrations we wrote ourselves, so they aren't a measure of accuracy.

Should this message start a job? Two questions work better than one here. Jev picks a template for the third message with 0.99 confidence, but the work question stops it from running by itself.

Is this message trying something? The training question mentions prompt injection and still scores low on every hazard.

Should this note be stored? A keyword rule for "health" would trip on the magazine.

Who should handle this ticket? We'd argue with the second row, since "where is the page?" could be a general how-to question. That's the kind of call you only catch by labelling your own tickets.

Why did these runs fail? This state holds only codes. The second row is why you pin a version: the same input cleared our 0.70 line in 2 of 3 runs on 1.13 and in none on 1.14, so the same failure could get a class on one version and none on the other.

Where it broke

We then tried to trip it, using the weak spots TypeSafe documents. It handled more than we expected, including short number comparisons, simple dates, a negation, a fair coin and a shuffled option list.

These are single hand-written cases. Alex Molas cites reports of 0.92 on a fair coin, and one survey reports that reordering 16 options changed 7.0% of Jev's answers, so our table doesn't prove these are safe.

Counting is another story. We gave it a ledger of 30 runs, 7 of them failed, and three different claims about that count.

It called both wrong counts probably correct. Adding up a column went no better, with 0.36 for the true total and 0.37 for a total 20% too high. On the 9 of 30 claim, 1.13 did better than 1.14, at 0.42. That's one case, but a version bump can move an answer across your threshold in either direction.

TypeSafe's advice is to do the counting in code and ask one yes or no question per item. Our repair triage already follows it: counts and costs become named buckets before anything is sent.

The hard test: judging agent reports

Our Claude judge for agent reports is stuck at 61.5% against an 85% bar, and it costs enough to notice. Grading reports looked like a closed question, so we tried Jev on it. We wrote the design down and hashed the file before making a single call.

Here is one of the questions we sent, abridged. What comes back is a probability, with no prose to parse:

{
  "model": "jev-1.13.0",
  "state": { "output": "the agent's report", "evidence": "the records" },
  "questions": {
    "claims_supported": {
      "type": "noul",
      "instructions": "Every claim in the headline and the first three findings of `output` is supported by an independent record for the same scope and window.",
      "criteria": {
        "true": "Each of those claims agrees with an independent record.",
        "false": "At least one of those claims is contradicted."
      }
    }
  }
}

We allowed up to five rounds of question wording on the tuning set and stopped after three, when the wording stopped helping. One amendment came after round one, on tuning data only: we added claim-level yes or no questions, and they then did better on tuning, 27 of 32 against 24. Without that change, Jev's held-out result would have been lower (52 or 49 of 90 instead of 58), so it's fair to count that as a benefit from tuning.

What happened

Jev came out ahead, but look at the last bar. Saying "fail" every time scores 51%, so neither judge is doing as much as the percentages suggest.

Nor can we say Jev is better than Claude here. On the 87 cases Claude could evaluate, the two gave different verdicts on only 8. Jev matched the reference on 6 of those and Claude on 2, which is p = 0.29. So they agree with each other on 79 of 87 cases, far more than either agrees with the reference. Both failed the same 27 cases the reference passed, and both scored 0 of 10 on News Desk. That points to either a shared bias or a generous reference, and this test can't tell us which.

Jev clearly stood out on cost. We didn't record Claude's latency in this test, so the response time below stands alone.

The confidence scores didn't hold up

Calibration is what makes a decision model attractive, because "above 0.85, act on it" has to mean something. Lightfield's benchmark found Jev better calibrated than Haiku on every task and than Sonnet on two of four, with expected calibration error from 0.015 to 0.245, while the Pinocchio paper found it didn't match a trained calibrator on any benchmark it tested (0.666 AUROC against 0.868). On ours it did poorly.

Every point sits above the line. Jev gave 68 of 90 answers a probability below 0.2, and the reference called about 56% of those supported. It never went above 0.6 on this question.

Removing every disputed reference improved it, but not enough (29 of 49 at high confidence). These figures test one question, "every claim is supported". The questions the final grader actually used had a lower calibration error (0.20) but did no better at high confidence (9 of 23).

The sample is small and the reference is agent-made, so this is a poor result on this one job and not a verdict on Jev. It's also a task aimed right at the model's documented weak spots. Checking "13 of 37 runs skipped review" against a ledger needs counting, and we had to feed it the same large, unfiltered evidence as Claude to keep the comparison fair.

Three cases where the graders split

Jev gives a probability and no reason, so any explanation for these is our guess. The number in brackets is its probability that the headline holds.

The last one is the only held-out case where Jev alone matched a reference fail, and it was close to the 0.5 line. The second shows the risk: "0 deploys could be verified" was misleading, and Jev scored it 0.52, just past the line.

What else exists

This table covers what we could check at the source on November 1. The numbers are each maker's own, and we haven't run any of these except Jev. We wouldn't compare latency across rows: Cloudflare measured Jev at a 524 ms median, while we saw 157 ms from our own servers.

You don't need a decision model to get a label with a probability, and older tools hold up well.

The independent evidence is thin and mixed.

• Red Hat tested guardrail tasks and concluded that decision models don't reliably beat LLM judges, trained classifiers or open models. Jev did lead its content safety table, at 86.20%.
• Lightfield, which says it's incorporating Jev, found it level with frontier models on small multiple-choice tests with reasoning off. Separately, it cost $0.84 per 1,000 long-context questions against $129.69 for Opus 5.
• Pinocchio found Jev weak at predicting whether another model's answer is correct, which is close to the job we gave it.
• Molas argues that calibration may not carry over to your data, and that a few hundred labels and Platt scaling can fix much of it.

Sources for the tables: Cloudflare, OpenAI, Microsoft, Perplexity, Strands, Laya, Buckmann and Hill. LiteLLM can translate between the OpenAI and TypeSafe request formats if you want to try several.

How we'd pick one now

Whichever model answers, these are the rules we build to.

• Keep the floor. Our regex checks still run first, and the model only adds to them.
• Treat no answer as a normal answer. A timeout or a low score sends the caller down its existing path.
• Use bands. Act above one line, suggest between two, and fall back below. Ours are still defaults, and we plan to tune them on labels.
• Fail toward people when the stakes are high. If the "needs a person" answer is missing, the ticket goes to a person.
• Do the arithmetic first. Counts, dates and costs become named buckets before anything is sent.
• Send as little as you can. Repair triage sends no customer text, which is why it's the one job we could switch on.
• Pin the version and log the score. You can't re-tune thresholds on a model that changed under you.
• Keep the questions separate from the vendor. Our question files define only the questions, and a route picks the provider. A route can also ask a second model the same questions in shadow, without acting on its answers.

Where we pick this up

Handover sheet · Nov 1

Our notes for the next shift. If you run agents too, this list may save you a lap.

• Keep
  Don't use it as the only grader of factual reportsIt missed the bar, and its confidence can't gate anything on this task yet.
• Keep
  Stay on 1.13.0 for repair triage until it's re-tunedOne of our examples cleared the 0.70 line in 2 of 3 runs on 1.13 and in none on 1.14.
  typesafe_client.py
• Open
  Grade repair triage's answersAll 13 groups are marked unverified. Compare each answer with what the fix turned out to be, so we finally have a number for the one job that's live.
  office_repair_groups
• Open
  Shadow a second model on repair triageThe route already supports it, and codes-only state makes it the low-risk place to compare Clef or Strands Decider against Jev.
  decisions/routing.py
• Open
  Test narrower questionsLet code do the arithmetic and ask Jev only the judgement calls, such as "does this next step follow from this finding?". We haven't run that.
  decisions/questions
• Open
  Fit calibration on our own labelsWe've scored calibration but never tried to correct it. The audit gives us 122 labelled reports to start from.
  decisions/training.py

What to know before relying on this

Our reference grades were made by Claude, so the judge test measures agreement with a Claude audit and not with the truth, and a Claude judge may have an advantage there. The Claude judge's rubric may also have been shaped by some of these same cases. Our Claude costs came from a subscription CLI's reported figures, not an invoice. The worked examples and traps are inputs we wrote ourselves, and each ran only three times per version.

We also made a mistake. Our first redaction pass in the judge test missed some person names in the reports, mostly public names like paper authors and well-known bloggers, and they were sent to TypeSafe in 16 requests. An audit of the decoded text found no emails, credentials or phone numbers. We fixed the scan and re-sent those six cases, and no verdict changed. We now scan the decoded text instead of the serialized JSON.

TypeSafe had no role in this test, and we have no relationship with them beyond using their API.

If you've found a decision model working well as a judge, we'd like to compare notes: boris@curate-me.ai.