An AI answer is not automatically a resolved customer issue. To measure resolution, define which conversations entered the flow, what evidence confirms the outcome, and when a person took over. Keep confirmed resolution, assumed resolution, human escalation, and abandonment separate. Then inspect repeat contact and answer quality. Otherwise a dashboard may reward customers simply for going quiet.
Use one conversation as the unit and set a fixed observation period. Among conversations involving AI, count outcomes supported by explicit confirmation or a verified action, outcomes inferred only because the customer stopped replying, human handoffs, and unresolved cases. Show all four counts and their denominators. Review samples before treating silence as a successful resolution.
Choose the population before calculating a rate
First decide what management wants to know. Is the question how well AI handles conversations it actually receives, or what fraction of all support conversations AI completes? The first denominator is eligible AI-involved conversations. The second includes the entire scoped support volume, including conversations routed straight to people. Combining these questions can make a routing change look like a quality change.
Intercom's official explanation distinguishes involvement, resolution among AI-involved conversations, and automation across total volume for its product. Those are vendor definitions, not universal standards. Write down your inclusion rule: channels, languages, intents, business hours, test messages, reopened cases, and customers who requested a person before AI could answer. Keep that rule stable across periods.
Count a conversation once even if AI writes several messages. When a customer returns through another channel, verify identity before joining records. Similar names or shared phone numbers do not justify merging two people. The guide to customer identity across channels explains this operational boundary.
Separate an answer from an outcome
An answer proves that the system responded. Resolution means the customer's request was addressed within the agreed scope. Stronger evidence may be an explicit customer confirmation, a completed and verified operational action, or a human review of the case. Many journeys have no explicit confirmation. Label those outcomes assumed instead of confirmed.
Intercom's reporting documentation distinguishes confirmed, assumed, escalated, and abandoned states for Fin. Zendesk's reporting documentation also separates automated resolutions, assisted escalations, and tiers of resolution evidence. Vendor criteria can differ and change. Read their definitions before comparing two percentages that happen to share a name.
Silence is particularly ambiguous. A customer may have found the answer, left because of an interruption, or given up after an unhelpful reply. Automatic closure does not establish satisfaction either. Record what actually happened and retain a repeat-contact window. High-risk requests, disputed charges, health matters, or contract decisions need appropriate human review; a single aggregate rate should never conceal those exceptions.
Work an example a manager can audit
Suppose a support team handled 300 conversations in a period and AI participated in 120. Of those 120, 48 had confirmed resolution, 24 were assumed resolved, 30 went to a person, and 18 had no verifiable conclusion. At period close, assign each conversation to one outcome so the categories do not overlap.
Confirmed resolution among AI-involved conversations is 48 ÷ 120 = 40%. If the team chooses to include assumed outcomes, label the result clearly: 72 ÷ 120 = 60% confirmed-or-assumed conclusions, not 60% proven resolutions. Across all support volume, 72 ÷ 300 = 24% answers a different question. A case completed after a human handoff does not become an AI resolution. These numbers are hypothetical teaching examples, not Signaldesk results or industry targets.
Keep the raw counts alongside the percentages: a rate can rise while the number of useful outcomes falls. Segment by channel and intent only when sample sizes support interpretation. The broader guide to service metrics shows why an isolated average or rate can mislead.
Look for failure hidden by a strong-looking rate
Read a sample of conversations marked both confirmed and assumed. Was the source approved? Did AI understand the actual question? Was an action still pending? Did the customer return with the same issue? Repeat contact alone does not prove AI was wrong, but it should trigger review. The AI conversation audit guide offers observable criteria for evidence, risk, transparency, and handoff quality.
Also inspect who never entered the denominator. If AI receives only simple questions, a high resolution rate can coexist with an overloaded human team. If a new rule escalates earlier, a lower AI resolution rate may reflect better protection rather than poorer service. Monitor involvement, handoff reasons, human wait time, and satisfaction with survey coverage visible. Do not treat a small set of survey responses as everyone’s opinion.
When the review finds a gap, fix its cause before raising the target: outdated knowledge, an incorrectly classified intent, missing customer data, an action requiring authorization, or unclear escalation. Test the revision on a sample, record the owner and version, and confirm the error does not recur. Never ask AI to improvise around missing evidence.
- Define the unit, period, channels, and eligible population before comparing periods.
- Separate involvement, confirmed resolution, assumed resolution, human escalation, and unresolved counts.
- Show numerator and denominator for every rate; identify test traffic and duplicates.
- Preserve outcome evidence and define a repeat-contact observation window.
- Have a person review a sample of apparently successful answers.
- Transfer sensitive requests and explicit human requests with context and accepted ownership.
- Annotate routing, knowledge, and policy changes beside the time series.
- Do not present a resolution rate as satisfaction, cost savings, or revenue without separate proof.