Confidence scores and thresholds
Confidence scores and thresholds
How the score is calculated, what the 82% and 60% thresholds do, and how to make them stricter per ticket type.
Every answer gets a confidence score from 0 to 100%. It blends three signals that are each calibrated to the full 0–1 range:
| Signal | Weight | What it measures |
|---|---|---|
| Retrieval strength | 45% | How closely the best articles or past resolutions match the question |
| Grounding | 35% | How much of the answer is supported, sentence by sentence, by those sources (courtesy lines like “Let me know if…” are ignored) |
| AI confidence | 20% | How sure the model itself is that the sources answer the question |
An answer that is not about the question is penalised and capped at 50%, however well it copies a source. If the AI itself is at least 82% sure (or your threshold, if higher), the answer is on-topic, at least half supported by sources and backed by a reasonable match, it is sent even when the blended score is a little lower.
The two thresholds
- Auto-send threshold (default 82%): at or above it, the reply goes out on its own (unless assisted mode is on).
- Review threshold (default 60%): between the two, the draft waits in the review queue. Below it, the ticket is triaged to your team.
- 1Open Settings → AutomationSidebar → Settings → Automation. This is the one place to change the auto-send and review thresholds (in %); the Overview page shows them read-only with a link here.
- 2Stricter for some ticket typesSettings → Answer engine → Thresholds by ticket type: set a higher auto-send level (82% or more) for, say, billing or account tickets. Blank means the default applies.
- 3Test the changeRun a regression test after changing a threshold to see which past answers would now be sent, reviewed or triaged.
Tips & tricks
- Auto-send never goes below 82%. Use ticket-type thresholds to be stricter where mistakes cost more — billing, account and legal questions.
- If many good drafts land at 70–80%, the fix is usually a clearer knowledge-base article (higher retrieval strength), not a lower threshold.
- Open a ticket in the ticket log to see the breakdown: low retrieval strength = missing article; low grounding = the answer went beyond the sources.