A well-optimized chatbot can deflect an estimated 80% of routine inquiries in the contact center. That means customers with straightforward issues get answers fast, and human agents have more time to spend on complex issues that require critical thinking and empathy.
But what if your chatbot doesn’t perform as expected? What if it routinely gets confused by your customers’ questions, or confidently shares inaccurate information (a risk with GenAI chatbots)? At best, this is annoying for customers and drives more escalations to human agents. At worst, it damages trust and could even have financial or legal consequences (like in the case of the Air Canada chatbot that shared inaccurate information about a bereavement fare and misled a customer into buying a full-price ticket).
Chatbots require continuous monitoring and optimization. If you want your chatbot to be as valuable (and accurate) as possible, you need to measure its performance like you would a human agent. You should pay attention to the issues it handles well, where it’s struggling, and how you can train it to better support your customers.
The basic analytics that come with most chatbot solutions can help you get started by letting you track metrics like chat volume, conversation duration, and escalation rate. However, while these metrics give you the “what,” they don’t give you the “why”--and you need to know the “why” to make meaningful improvements.
Below, we’ll look at the metrics your chatbot provider is likely already giving you, the advanced analytics you should be layering on, and how you can use those analytics to improve the chatbot user experience.
Foundational metrics: What your chatbot platform already tells you
While out-of-the-box analytics will vary by chatbot provider, there are some basic metrics you can expect to see in just about any platform. These metrics are primarily focused on volume, engagement, and success or failure indicators.
Volume and engagement metrics
Session volume: The total number of chat sessions during a defined period. This helps you track chatbot adoption and changes in demand (e.g., does session volume spike due to seasonality, product releases, recalls, etc.).
Messages sent/received: The average number of individual messages exchanged between the customer and chatbot during a session. This can give you a high-level indication of conversation complexity (e.g., how much back-and-forth messaging was required to troubleshoot and resolve the customer’s issue).
Average session duration: How long the conversation takes from start to finish. Short chat durations might indicate that the chatbot efficiently resolved the issue or the customer immediately abandoned the chat, while long durations might suggest that the customer was struggling to get an answer or was stepping away from the chat mid-conversation to handle other tasks.
Success and failure indicators
Success rate/Flow completion rate: The percentage of sessions that ended with a successful resolution (based on predefined criteria). For a flow-based chatbot, this is often defined as the completion of a predefined flow (such as a password reset process). Success might also be defined more broadly as chats in which the user didn’t escalate their issue to a human agent. One challenge is that this metric typically uses a proxy for success (flow completion or no escalation) rather than measuring whether the issue was actually resolved to the customer’s satisfaction.
Error rate/Fallback rate: The percentage of sessions in which the chatbot didn't understand the customer’s question or used its fallback response (e.g., “I can’t answer that question.”) This can help you determine how effective your chatbot is at handling customer queries.
Escalation rate: The percentage of chatbot conversations that are escalated to a human agent. This can help you determine if customers are asking your chatbot questions that require human support. It’s an important baseline metric to pay attention to, as channel switching has been proven to decrease customer loyalty.
Advanced chatbot analytics: Using conversation intelligence for deeper insights
Foundational metrics help you get the lay of the land when it comes to chatbot performance: they give you the volume of chats and certain outcomes, but they don’t tell you what led to those outcomes. To better understand what’s happening in your chatbot interactions—and how you can improve them—you need to use conversation intelligence technology.
Conversation intelligence software analyzes voice and text interactions using machine learning, a type of AI that finds patterns and makes predictions using large data sets. In the case of chatbot conversations, it can analyze chat transcripts to determine things like contact reasons, customer friction points, and causes of chatbot confusion. This helps you uncover targeted opportunities to train your chatbot, improve issue routing and escalation, and address issues upstream.
Analytics based on conversation content
Conversation intelligence software can identify and flag key topics and events in chatbot conversations (in Creovai, we refer to these as “insights categories”). This lets you segment and filter conversations so you can report on both what’s happening and why.
Contact reasons and outcomes
Top contact reasons: Why users are initiating chat sessions. You can track changes in contact reason volume over time to identify seasonal trends, emerging issues, correlation with marketing campaigns, and more.
Top contact reasons with successful resolution: The questions or issues the chatbot consistently solves without escalating to a human agent. With conversation intelligence, you don’t have to limit your “successful resolution” criteria to flow completion or lack of escalation. You can analyze what the customer says to determine if they were satisfied with the resolution.
Top contact reasons leading to escalation: The issues that stump the chatbot and get escalated to a human agent. While there may be some issues that need to be escalated to a human, you’ll also likely discover issues that could be handled by the chatbot with more training.
User experience indicators
Top causes of user effort: Signs of friction during the chat interaction, such as the customer expressing confusion or mentioning that they couldn’t find information on your website. Identifying effort drivers can help you figure out how to improve the chatbot experience and other customer touchpoints.
Top causes of user dissatisfaction: Customer issues or chatbot responses correlated with dissatisfaction. Rather than relying on users to complete a CSAT survey, conversation intelligence software analyzes language throughout the chat interaction and uses AI to predict whether the user would rate themselves as satisfied or dissatisfied.
How to use the analytics you get from conversation intelligence
Once you’ve started analyzing your chatbot interactions with conversation intelligence, it’s time to put that analysis to work.
Optimization strategy 1: Expand your chatbot capabilities
One of the most straightforward places to start is to pinpoint the customer questions or contact reasons that are causing your chatbot to fail. This can be especially useful for generative AI chatbots that have been configured to generate tailored responses based on pre-approved internal knowledge sources and use a fallback response for anything that can’t be answered through those sources.
Here’s how you can improve your chatbot responses with Creovai:
- Create a custom category for your chatbot’s fallback or escalation response. (This will allow you to filter interactions in which the chatbot failed to understand or help the user.)
- Build a report that shows the contact reason categories (by volume) that appear in failed chat interactions.
- Review the contact reasons and determine which ones should be escalated to a human agent vs. handled by the chatbot. Prioritize the contact reasons you need to train your chatbot on based on volume (i.e., tackle the most frequent contact reasons first).
- Update your chatbot’s flows or training data to fill the top knowledge gaps. (Note: The exact training process will depend on the type of chatbot you have and how it is configured.)
- Bonus tactic: Use your best voice interactions to train your chatbot. You can use Creovai to filter or search for voice interactions that address the issues stumping your chatbot (you can further filter to only those interactions with a positive predicted CSAT score or other indicators of a successful outcome). Look at a sampling of these interaction transcripts to determine what your human agents are saying, then build these successful responses into your chatbot training.
Optimization strategy 2: Drive straightforward queries to your chatbot
You can also use Creovai to track the reasons customers are calling you and identify the issues that your chatbot is capable of handling. From there, it’s a matter of educating your customers and encouraging them to use your chatbot.
- Build a report that shows the top contact reason categories for your voice interactions. You can use Creovai’s pre-built categories (e.g., Billing Issue, Reservation Cancellation) and create custom categories for contact reasons that are unique to your business.
- Review the contact reasons and build a list of the ones you know your chatbot is capable of handling.
- Develop messaging to direct customers to your chatbot when they call your contact center about one of those issues that could be automated. For example, you could update the language in your IVR based on your customers’ needs (“If you’d like to check your balance instantly, just type ‘balance’ into our chat widget”). You could also prompt your agents to let customers know they can try the chatbot the next time they have a straightforward issue (e.g., “If you need to get a tracking number from us in the future, you can always use our chatbot to get a fast response”).
Optimization strategy 3: Improve processes upstream
Conversation intelligence can help you identify when customers are talking to your chatbot about issues that could be resolved upstream. By working across teams to address these issues, you can reduce the volume of queries coming to your chatbot and human agents.
- Build a report showing the top causes of customer effort in chatbot interactions. You can use pre-built Creovai effort categories (e.g., Company Communications: Missed Expectations, Confirmation Email Not Received).
- Start by filtering down to a view of all interactions containing the top reason for customer effort. Look for other categories frequently appearing in these interactions (e.g., do customers often mention their shipment didn’t arrive on time in chat conversations with missed expectations?). You can also drill into a sample of transcripts to better understand what’s causing effort.
- Collaborate across teams to proactively address these effort drivers. For example, when shipments are delayed, could you work with your marketing team to send automated emails or SMS notifications to update your customers?
Build performance management into your chatbot strategy
Chatbots are getting increasingly sophisticated thanks to developments in natural language processing and generative AI—and contact center leaders are taking advantage of this. Almost two-thirds (64%) of CX leaders say they plan to increase their investment in evolving their companies’ chatbots in the next year.
But to make that investment pay off, CX and contact center leaders must be able to answer both the “what” and “why” questions about chatbot performance.
Advanced chat analytics let you continuously monitor chatbot performance and discover why interactions succeed or fail. You can use these insights, alongside conversation intelligence from your voice channel, to train your chatbot, guide your customers down a self-service route, and proactively solve issues before they get to the contact center. Ultimately, that means higher first-contact resolution rates for your chat channel, more time for human agents to spend on high-complexity issues, and a lower-effort experience for your customers.
