Score: Rate Text on Your Own Scale With Jev
Last checked · Independent guide, not affiliated with TypeSafe AI
A Score asks Jev where the state falls on an ordered list of levels you describe, such as 'can wait', 'this week', 'today'. The answer is a probability-weighted score that can land between levels (for example 1.83 on a 0 to 2 scale), plus the probability of each level and a confidence value.
The shape of a Score
Section titled “The shape of a Score”"urgency": { "type": "score", "instructions": "How urgent is this message?", "criteria": ["Can wait a week", "Should be handled this week", "Needs a reply today"]}criteria is an ordered array: the first entry is level 0, the next level 1, and so on. Each level can be a short phrase or a JSON object with a fuller definition.
For the message “Whenever you get a chance, could you look at the duplicate charge? No rush, but I’d like it sorted before the end of the month.”, Jev 1.13 returned:
"urgency": { "type": "score", "score": 0.13, "confidence": 0.8, "legend": { "0": "Can wait a week", "1": "Should be handled this week", "2": "Needs a reply today" }, "probabilities": { "0": 0.87, "1": 0.13, "2": 0 }}Why the score is not a whole number
Section titled “Why the score is not a whole number”The score is the average of the level numbers weighted by their probabilities: 0 × 0.87 + 1 × 0.13 + 2 × 0 = 0.13. A message that is mostly level 2 with some chance of level 1 might come back as 1.83, which is what we got for a login-failure report on a similar three-level urgency scale.
That makes scores easy to sort and threshold, but it does not turn them into measurements. TypeSafe warns that Jev 1.13’s scores are weakly calibrated between levels: do not try to recover an exact quantity by interpolating. Use the score to rank, or compare it against a threshold such as “above 1.5 means today”.
Fields in the answer
Section titled “Fields in the answer”| Field | Meaning |
|---|---|
score |
Probability-weighted position on your levels; can fall between them |
legend |
Your levels by number, echoed back |
probabilities |
The probability of each level (keys are level numbers as strings) |
confidence |
How concentrated the distribution is; low when the levels are ambiguous |
Writing good levels
Section titled “Writing good levels”- Make levels mutually exclusive and ordered. “Low, medium, high” works; “billing, urgent, angry” does not, because those are different dimensions.
- Describe observable differences. “Customers cannot complete payment” is easier to judge than “severe”.
- Use at least two levels. TypeSafe’s docs require two or more. In our test the API accepted a one-level Score and returned a meaningless 0, so check this in your own code.
- One dimension per Score. If a judgment mixes several factors, ask one Score per factor and combine them.
Combining several scores
Section titled “Combining several scores”TypeSafe’s composite scoring pattern splits a broad judgment into independent Scores and weights them in code. For ticket priority, you might score severity, customer frustration and how actionable the report is, normalize each to 0 to 1, and weight them. When your priorities change, you change the weights, not the questions, and you can always see why an item ranked where it did.
answers = response.answerspriority = ( 0.5 * answers["severity"].score / 2 + # three levels: 0 to 2 0.3 * answers["frustration"].score / 2 + 0.2 * answers["actionable"].score / 2)Score versus the other types
Section titled “Score versus the other types”Use a Score when the answer is a position on a spectrum you can describe. If the answer is one of several unordered categories, use a Choice. If you only need yes or no, use a Noul: TypeSafe’s example is that “Is this candidate strong in Python?” works better as a Score with defined skill levels than as a yes/no question about an undefined “strong”.