TR-AI

A/B experiments

A/B experiments

Test a change to thresholds, search or knowledge sources on a share of live tickets and roll out the winner with confidence.

Growth Pro Open in TR-AI6 min read

Simulations replay past tickets; experiments measure real customers. A share of live tickets (5–50%, default 10%) runs with the variant settings, the rest are the control group, and TR-AI compares the two arms.

What you can test

  • Auto-send and review thresholds, and per-category thresholds
  • Web search on/off and its cut-off, live site search on/off
  • The LLM-confidence override
  • Which knowledge sources are used (include or exclude a subset)
Note:Safety settings can never be part of an experiment: output guard, PII handling, escalation rules, assisted mode, URL allowlists and competitor rules always apply to both arms.
  1. 1
    Create
    Dashboard → Experiments → New: name it, choose the settings to change, the traffic share, the minimum tickets per arm (default 100) and the maximum length (default 14 days).
  2. 2
    Start
    Only one experiment runs at a time. Each ticket is assigned to an arm by a hash, so a ticket and its follow-ups always stay in the same arm. Dry runs and simulations never take part.
  3. 3
    Watch the results
    Per arm: auto-resolved rate, review and triage rates, failure rate (reopened, customer replied, low CSAT or a similar new ticket), CSAT, quality score, average confidence, web-search rate and AI calls per ticket — with confidence intervals and significance tests (p < 0.05).
  4. 4
    Roll out or stop
    “Roll out winner” writes the variant settings into your live settings and records a config version, so you can roll back.

Guardrails

  • After at least 50 tickets in each arm, an experiment whose variant fails significantly more often (by 5 percentage points or more) is stopped automatically.
  • Experiments finish on their maximum length even without a clear winner.
  • Experiments add no AI calls — only a cached lookup per ticket.
Tips & tricks
  • Change one thing per experiment; with two changes you won’t know which one helped.
  • Use regression tests or the Simulation Lab first to rule out obvious regressions, then confirm with an experiment on live traffic.
  • Low-volume teams: raise the traffic share to 30–50% or the result will take weeks to become significant.
  • Judge on failure rate as much as auto-resolved rate — a lower threshold always sends more, the question is whether those answers stick.
PreviousKB health, spike detection and churn alertsNextDecision traces