Jev Confidence and Calibration Explained

“What do the probabilities actually mean?” was one of the sharper questions in the Hacker News launch thread. This page answers it with TypeSafe’s definitions and examples from our own calls.

Probability, confidence and calibration

Term	Where you see it	What it means
Probability	noul, and probabilities on Choice and Score answers	How likely each answer is
Confidence	confidence on Choice and Score answers	One number summarizing how concentrated the probabilities are
Calibration	A property of the model, not a field	Across many answers, stated probabilities match how often they are right

Confidence is derived from the probabilities. A Choice with probabilities 0.96 / 0.04 / 0 / 0 is confident (we got 0.94). A Score split 0.87 / 0.13 / 0 is less so (0.8). A spread-out distribution means Jev sees no clear winner. TypeSafe describes confidence as a sensible default and returns the full distribution so you can compute your own measure.

Noul answers have no confidence field. The yes probability is the signal: near 0 or 1 is confident, near 0.5 is not.

Calibration applies to groups. TypeSafe trains Jev with RLCD so that, across many predictions, answers given 0.8 come true about 80% of the time. It does not guarantee any single answer. A 0.95 can still be wrong; it should just be wrong rarely.

Why this matters

An LLM will usually give you an answer even when it should not, and if you ask how sure it is, the number it writes is not reliable. Jev’s probabilities let your code say “I’m not sure” on the model’s behalf. TypeSafe puts it as: a system that cannot express honest uncertainty cannot be trusted with automation.

Three bands: act, check, escalate

A simple starting pattern divides confidence into three ranges:

Band	What to do	Example
High	Act automatically	Route the ticket, tag the item
Middle	Act with a safeguard	Ask the user to confirm, queue for spot checks
Low	Do not act	Send to a person, ask for clarification, fall back to another system

Where you draw the lines depends on the cost of a mistake, so different actions in the same app get different thresholds. TypeSafe’s banking example puts it concretely: showing the wrong balance screen is recoverable, approving a transfer is not, so the transfer needs a much higher bar.

action = response.answers["intent"]

if action.confidence < 0.5:
    hand_to_human()
elif action.choice == "check_balance":
    show_balance()                      # low stakes: act on any confident answer
elif action.choice == "approve_transfer":
    if action.confidence > 0.9:
        confirm_then_execute()          # high stakes: require very high confidence
    else:
        ask_user_to_confirm()

Choosing thresholds

1. Collect real inputs and label the right answer for 50 to 200 of them.
2. Run your questions with a pinned model version (jev-1.13.0).
3. For each candidate threshold, count how many cases are automated and how many of those are wrong.
4. Pick the lines that give an error rate you can live with.
5. Re-check when the model version changes, because a new model can shift the numbers.

When not to use confidence thresholds

TypeSafe’s own troubleshooting notes add two cautions:

• If you only need the best option, take it. A threshold makes sense only when acting on a wrong answer is costly.
• For statistical work, use the probabilities. If you are feeding answers into another model or computing expected values, the full distribution carries more information than one confidence number.

Low confidence is feedback

Consistently low confidence on a question usually means the question, not the input, is unclear: options overlap, levels mix two dimensions, or a yes/no hinges on an undefined word. In our test, “Is the customer asking for money back?” returned exactly 0.50 for a message that never asked for a refund, while a sharper question about the same message returned 0.97. Rewriting the question is often the fix. See Noul, Choice and Score.