Confidence Scoring
How TR-AI scores resolution confidence and decides when to auto-reply versus escalate.
Every ticket processed by TR-AI receives a confidence score between 0.0 and 1.0. This score determines whether the ticket is resolved automatically or routed to your triage group.
How the score is calculated
TR-AI uses a two-stage resolution pipeline:
Stage 1 — Vector search
The ticket subject and body are embedded with a local MiniLM sentence model (384 dimensions). TR-AI retrieves the most similar KB chunks and past human resolutions by cosine similarity. The highest similarity among them becomes the best similarity value.
Stage 2 — LLM synthesis
If best similarity ≥ 0.30 and the KB is fully indexed, the top chunks are passed to the LLM (Claude) along with the ticket. The LLM writes a reply from those chunks only and reports how confident it is that they answer the question.
The confidence score
The score blends three signals, each on a 0–1 scale:
- Retrieval strength (45%) — how strongly the best passages match the question. Raw similarity is calibrated: 0.35 or less counts as no match and 0.75 or more as a strong match; the top passage counts for three quarters and the top three together for one quarter, plus a small boost when the article title matches the question.
- Grounding (35%) — how well each sentence of the reply is supported by a sentence in the passages (a close paraphrase counts as fully supported; a claim found nowhere counts as unsupported).
- LLM confidence (20%) — the model's own confidence.
A reply that is not about the question (low reply-to-question similarity) is penalised and capped at 0.50, however well it copies a passage.
needsReindex=true). In these cases the ticket goes to the fallback below and, if that cannot answer either, to triage.When the KB can't answer: live site search, then web search
If your KB and past resolutions have no answer, or their answer scores below your web-search cut-off (default 0.80), TR-AI first runs a live site search of your allowed domains: it reads their sitemaps, help-center search (Zendesk, Freshdesk, Intercom, Help Scout, Discourse) and status page, fetches the few best-matching pages and answers from them with the same scoring and safety checks as a KB answer. This is free — no search fee. Only if that answer is missing or still scores below the cut-off does the paid web search of the same allowlist run, as the last resort, and its answer is used only if it scores higher.
Decision thresholds
| Condition | Outcome | What happens |
|---|---|---|
| confidence ≥ 0.82 (default threshold) | resolved | AI posts reply + closes ticket |
| 0.60 ≤ confidence < 0.82 | review | AI posts its draft as an internal note for an agent to approve |
| confidence < 0.60 | triage | AI posts holding reply + moves to triage group |
| best_similarity < 0.30 | triage (hard) | Ticket goes directly to triage — no LLM call |
| KB reindex running | routed_to_agent | Ticket goes directly to agent — no AI reply |
Reading confidence scores in the Dashboard
The Dashboard → Recent resolutions feed shows the confidence score for each ticket. Scores are displayed as a decimal (e.g. 0.923).
- 0.82–1.00: High confidence — auto-resolved (default threshold)
- 0.60–0.81: Medium confidence — drafted for agent review, close to the threshold
- Below 0.60: Low confidence — triage, KB likely has a gap
Improving confidence scores
For tickets scoring 0.60–0.81
These tickets are close to the auto-resolve threshold. The KB likely has a relevant article but it is not specific enough. Improve the article by:
- Making the title match how customers phrase the question
- Adding more specific steps or values
- Breaking a long article into shorter, focused ones
For tickets scoring below 0.60
Little or no relevant KB content was found. Create an article covering this topic.
The confidence threshold
The default auto-resolve threshold is 0.82, and the default review threshold is 0.60. Both can be changed per account. You can adjust it in the Dashboard → Configuration.
Resolution log fields
{
"externalTicketId": "12345",
"outcome": "resolved",
"confidenceScore": 0.923,
"webSearchUsed": false,
"replyPosted": true,
"source": "zendesk"
}