Copilot Insights (Limited Preview)
Copilot Insights is the analytics dashboard exclusively for the Next-gen Copilot, featuring data solely from this platform. If you are currently using Classic analytics dashboards like Bot Performance Insights or Answers Insights, please note that they do not incorporate data from the Next-gen Copilot.
What is Copilot Insights?
Copilot Insights represents the latest in Moveworks analytics for the Next-Gen Copilot, offering comprehensive visibility into end users' interactions with the Copilot. This includes detailed insights into user adoption rates and Copilot performance, providing a transparent overview for actionable insights.
What are the key components of Copilot Insights?
Interactions
Copilot Insights offers complete transparency into every facet of user interaction with the Copilot, enabling a thorough replay of each engagement for in-depth insights. This encompasses various user interaction types, including free-form text, button clicks, link clicks and Copilot's responses.
User Adoption
Copilot Insights grants visibility into the extent and depth of user adoption for Copilot. Ranging from active user counts to popular plugins, the data is customizable, allowing you to analyze adoption trends across various demographic dimensions.
Performance
Copilot Insights offers an unbiased perspective on Copilot performance. Users can assess the success rates of each plugin call, delve into the underlying causes of unsuccessful interactions, and access in-chat feedback from end users for a comprehensive understanding of system efficiency.
Key definitions
Interactions: all types of engagements end users have with the Copilot. There are 4 types of interactions:
| Interaction Type | Description |
|---|---|
| Free-form text | User types something into the Copilot |
| Button Click | User clicks on a button from a Copilot notification or response. E.g. Clicking on “Approve” of an approval notification |
| Link Click | User clicks on a link of a Copilot response. E.g. A citation link |
| UI Form Submission | User submits a form on the reference pop-up page. The UI forms include submitting a ticket, filling out a catalog form, and reporting in-app free-text feedback. |
Plugin: The next-gen Copilot is a versatile and flexible engine powered by the latest large language models. On top of the Copilot platform, we’ve built plugins to execute specific tasks. If the Copilot engine is the brain of a person, these plugins are the hands and legs that carry out instructions from the Copilot. The Copilot includes over 20 out-of-the-box plugins that provide a wide range of capabilities including search and take action. Below is a list of our out-of-the-box plugins.
| Plugin Name | Description |
|---|---|
| Knowledge Base | Search in the knowledge base and FAQs to find relevant information and summarize the answer |
| Forms | Find forms and let users fill them out in the bot |
| People Registry | Find people from the org chart, including looking up manager, skip level manager, direct reports, and finding their detailed information like department, email address, etc. |
| Room Registry | Find the location of a meeting room in the office |
| Ticket Information | Retrieve all tickets or a specific ticket information |
| Approval Information | Retrieve approval information |
| Owner of Group | Find the owner of a distribution list |
| Group Membership Status | Check whether a user is in the distribution list |
| Account Status | Check whether the user’s account is still active |
| Smart Handoff | Offer handoff options to various domains |
| Create Ticket | File a ticket through the bot |
| Resolve Ticket | Close a ticket |
| Reopen Ticket | Reopen a ticket |
| Add Comment to Ticket | Add comment to a ticket |
| Create Group | Create a new distribution list in chat |
| Add User to Group | Add one or multiple people to a distribution list |
| Remove User from Group | Remove one or multiple people from a distribution list |
| Unlock Account | Unlock account for the user |
| Reset Password | Reset password for the user |
| Reset MFA | Reset MFA for the user |
| Grant Software Access | Grant access to a software |
| Update Approval Record | Approve/deny an approval request in chat |
| Write Email | Compose an email with relevant information |
| Change Preferred Language | Offer to change user’s preferred language |
| Translate Text | Translates arbitrary text in the manner specified by the user. |
In addition to the out-of-the-box plugins, each Creator Studio use case will also appear as a plugin, with the use case name as the plugin name.
Plugin Call: To assist with a user request, the Copilot selects the relevant plugin to execute, this process is a plugin call. There can be multiple plugins called to fulfill a single user request. For example, a user could ask “Can you add my manager to the Sales group?”, and both the People Registry and the Add User to Group plugins are called.
Plugin Status: When a plugin is called, there are 3 final status that help indicate the result of the plugin call, and they are referred to as plugin status. The 3 status are:
| Plugin Status | Description |
|---|---|
| Completed - Successful | The plugin is called, AND it found and summarized a resource or completed a step of action for the end user. |
| Completed - No Result | The plugin is called, but did not find resources from integrated systems. |
| Incomplete | The plugin called, but is unable to complete the workflow. |
What are some key questions I can answer with Copilot Insights?
We will dig into the Copilot Insights user experience by answering each of the key questions below:
What interactions did my user have with the Copilot?
In the example below, observe an upward trend in user interactions with Copilot over the past few weeks. The predominant interaction type is through free-form text, indicating that users prefer engaging with Copilot through chat.
How is my user adoption trending over time?
In the provided example, Copilot usage displayed a gradual upward trend, with a significant spike observed in the current week. This aligns with the user interactions chart above, suggesting that the surge in usage can be attributed to an increased number of users.
What are the most common asks from my user?
In this example, the Knowledge Plugin emerged as the most utilized, signifying that the majority of user requests were related to knowledge searches. Examining the figures, it's noteworthy that 3,000 out of 14,000 knowledge searches did not find resources from the connected systems. For a more in-depth analysis of these user queries, refer to the next question below.
What feedback did my user give?
View end users’ feedback of the Copilot by clicking on “View User Feedback”.
What new knowledge should I write about?
For a more in-depth analysis of knowledge searches with no results, click on 'View Interactions' to open the side panel containing raw interaction data.
Narrow down the user interactions by applying filters of interest. For example, if you want to see users’ knowledge search that did not return any result, apply the following filters:
- Interaction type = ‘Free-form Text’
- Plugin Details = ‘Knowledge Base: Completed - No Result', select all options that contains this value
You can review the “Interaction Content” column to understand what user queries did not receive an answer. This can help inform new knowledge to be added.
What knowledge articles should I update?
You can infer the quality of your knowledge base through end user feedback.
Open the “View User Feedback” table and filter on the unhelpful feedback. You can read through user queries, bot response, user feedback, and the KB articles associated with it. While the user feedback is not directly tied to each knowledge article, you can extract helpful insights from the feedback text.
In addition, you can also download the table into an Excel or BI tool, and group the feedback by article names. If there is one particular article that received a lot of negative feedback, that’s a good signal to update it.
What HR (or any other domain) questions did my users ask about?
You can narrow down the users’ questions with the “Domain” filter. There is a global filter which allows you to apply the filter to every chart on the dashboard, like illustrated below.
Or, you can apply filters directly to the “View Interactions” and “View User Feedback” tables.
Download raw interactions data and feedback data
Click on the download button on the top right corner to download the table as a csv file for more in-depth analysis. Please note the csv file is currently limited to 15K rows.
What did my user ask in the entire conversation?
If you see an interaction that is potentially from a multi-turn conversation, you can download the raw data tables and group interactions with “Conversation ID”. This allows you to see all interactions associated with that conversation. Then, sort the interactions by “Timestamp” to view the interaction in time order.
What are the dimensions of the tables?
View Interactions
| Column Name | Description |
|---|---|
| Timestamp (UTC) | The date and time of the interaction |
| User ID | The user record ID |
| User Email | The user’s email address |
| Conversation ID | The conversation ID of the interaction. Interactions in the same conversation will share the same conversation ID |
| Interaction Type | The type of the interaction, including - Free-form Text: User entered a query in the form of text - Button Click: User clicked on a button from a bot’s response - Link Click: User clicked on a link in the bot’s response - UI Form submission: User submitted a form on the reference pop up page, including ticket filing, form, and in-app feedback |
| Interaction Content | The details of the interaction, such as the user’s input query |
| Bot Response | The Copilot’s response to the user’s interaction, returned in chat. |
| Number of Citations | The total of citations associated with the response |
| KB Names | The knowledge article names cited. Currently, only knowledge articles and FAQs are included. Other citation types, such as external knowledge, files, and forms, are not supported yet. |
| KB ID | The corresponding IDs of the cited knowledge articles |
| Plugin Details | The plugins called for the user’s interaction and the status of the plugin. If there is more than one plugin called for the interaction, the field will be a list of plugins and their status. |
| Department | The user’s department |
| Location | The user’s location |
| Country | The user’s country |
| Domain | The user utterances’ predicted domain |
| Language | The latest language setting of the user on the given date. Note that the language field is not directly tied to each utterance. E.g. On any given day, if the user first interacted in EN and later updated the language to ZH, the language will be set as ZH for all of the utterances on that day. |
View User Feedback
| Column Name | Description |
|---|---|
| Timestamp | The date and time of the interaction |
| User ID | The user record ID |
| User Email | The user’s email address |
| Conversation ID | The conversation ID of the interaction. Interactions in the same conversation will share the same conversation ID |
| User Utterance | The user’s input question to the Copilot |
| Copilot Response | The Copilot’s response to the user’s interaction, returned in chat. |
| Feedback Type | The feedback type: 👍 helpful or 👎 unhelpful |
| Feedback Text | The free-form text user provided |
| KB Name | The knowledge article names cited. Currently, only knowledge articles and FAQs are included. Other citation types, such as external knowledge, files, and forms, are not supported yet. |
| KB ID | The corresponding IDs of the cited knowledge articles |
| Department | The user’s department |
| Location | The user’s location |
| Country | The user’s country |
| Domain | The user utterances’ predicted domain |
| Language | The latest language setting of the user on the given date. Note that the language field is not directly tied to each utterance. E.g. On any given day, if the user first interacted in EN and later updated the language to ZH, the language will be set as ZH for all of the utterances on that day. |
How to get access to Copilot Insights?
Customers who are in Moveworks Next-Gen Copilot Limited Preview/Preview will have access to the Copilot Insights dashboard.
To access Copilot Insights, login to my.moveworks.com, click on “Analytics”, and click on “Copilot Insights” on the left panel of the dashboard.
To add more viewers for Copilot Insights, please follow the steps outlined here Manage Roles and Permissions.
Updated 6 days ago