Copilot Insights (Limited Preview)

What is Copilot Insights?

Copilot Insights represents the latest in Moveworks analytics for the Next-Gen Copilot, offering comprehensive visibility into end users' interactions with the Copilot. This includes detailed insights into user adoption rates and Copilot performance, providing a transparent overview for actionable insights.

What are the key components of Copilot Insights?

Interactions

Copilot Insights offers complete transparency into every facet of user interaction with the Copilot, enabling a thorough replay of each engagement for in-depth insights. This encompasses various user interaction types, including free-form text, button clicks, link clicks and Copilot's responses.

User Adoption

Copilot Insights grants visibility into the extent and depth of user adoption for Copilot. Ranging from active user counts to popular plugins, the data is customizable, allowing you to analyze adoption trends across various demographic dimensions.

Performance

Copilot Insights offers an unbiased perspective on Copilot performance. Users can assess the success rates of each plugin call, delve into the underlying causes of unsuccessful interactions, and access in-chat feedback from end users for a comprehensive understanding of system efficiency.

Key definitions

Interactions: all types of engagements end users have with the Copilot. There are 4 types of interactions:

Interaction TypeDescription
Free-form textUser types something into the Copilot
Button ClickUser clicks on a button from a Copilot notification or response. E.g. Clicking on “Approve” of an approval notification
Link ClickUser clicks on a link of a Copilot response. E.g. A citation link
UI Form SubmissionUser submits a form on the reference pop-up page. The UI forms include submitting a ticket, filling out a catalog form, and reporting in-app free-text feedback.

Plugin: The next-gen Copilot is a versatile and flexible engine powered by the latest large language models. On top of the Copilot platform, we’ve built plugins to execute specific tasks. If the Copilot engine is the brain of a person, these plugins are the hands and legs that carry out instructions from the Copilot. The Copilot includes over 20 out-of-the-box plugins that provide a wide range of capabilities including search and take action. Below is a list of our out-of-the-box plugins.

Plugin NameDescription
Knowledge BaseSearch in the knowledge base and FAQs to find relevant information and summarize the answer
FormsFind forms and let users fill them out in the bot
People RegistryFind people from the org chart, including looking up manager, skip level manager, direct reports, and finding their detailed information like department, email address, etc.
Room RegistryFind the location of a meeting room in the office
Ticket InformationRetrieve all tickets or a specific ticket information
Approval InformationRetrieve approval information
Owner of GroupFind the owner of a distribution list
Group Membership StatusCheck whether a user is in the distribution list
Account StatusCheck whether the user’s account is still active
Smart HandoffOffer handoff options to various domains
Create TicketFile a ticket through the bot
Resolve TicketClose a ticket
Reopen TicketRe-open a ticket
Add Comment to TicketAdd comment to a ticket
Create GroupCreate a new distribution list in chat
Add User to GroupAdd one or multiple people to a distribution list
Remove User from GroupRemove one or multiple people from a distribution list
Unlock AccountUnlock account for the user
Reset PasswordReset password for the user
Reset MFAReset MFA for the user
Grant Software AccessGrant access to a software
Update Approval RecordApprove/deny an approval request in chat
Write EmailCompose an email with relevant information
Change Preferred LanguageOffer to change user’s preferred language
Translate TextTranslates arbitrary text in the manner specified by the user.

In addition to the out-of-the-box plugins, each Creator Studio use case will also appear as a plugin, with the use case name as the plugin name.

Plugin Call: To assist with a user request, the Copilot selects the relevant plugin to execute, this process is a plugin call. There can be multiple plugins called to fulfill a single user request. For example, a user could ask “Can you add my manager to the Sales group?”, and both the People Registry and the Add User to Group plugins are called.

Plugin Status: When a plugin is called, there are 3 final status that help indicate the result of the plugin call, and they are referred to as plugin status. The 3 status are:

Plugin StatusDescription
Completed - SuccessfulThe plugin is called, AND it found and summarized a resource or completed a step of action for the end user.
Completed - No ResultThe plugin is called, but did not find resources from integrated systems.
IncompleteThe plugin called, but is unable to complete the workflow.

What are some key questions I can answer with Copilot Insights?

We will dig into the Copilot Insights user experience by answering each of the key questions below:

What interactions did my user have with the Copilot?

In the example below, observe an upward trend in user interactions with Copilot over the past few weeks. The predominant interaction type is through free-form text, indicating that users prefer engaging with Copilot through chat.

How is my user adoption trending over time?

In the provided example, Copilot usage displayed a gradual upward trend, with a significant spike observed in the current week. This aligns with the user interactions chart above, suggesting that the surge in usage can be attributed to an increased number of users.

What are the most common asks from my user?

In this example, the Knowledge Plugin emerged as the most utilized, signifying that the majority of user requests were related to knowledge searches. Examining the figures, it's noteworthy that 3,000 out of 14,000 knowledge searches did not find resources from the connected systems. For a more in-depth analysis of these user queries, refer to the next question below.

What questions did not get a response from the Copilot?

For a more in-depth analysis of knowledge searches with no results, click on 'View Interactions' to open the side panel containing raw interaction data.

Narrow down the user interactions by applying filters of interest. For example, if you want to see users’ knowledge search that did not return any result, apply the following filters:

  • Interaction type = ‘Free-form Text’
  • Plugin Details = ‘Knowledge Base: Completed - No Result', select all options that contains this value

What feedback did my user give?

View end users’ feedback of the Copilot by clicking on “View User Feedback”.

View the total number of 👍and 👎, as well as the free-form feedback shared by the users.

From Copilot Response column, find the copilot response to the user’s feedback submission “ ✅Thank you for submitting your feedback” as well as the current Copilot message that received the user feedback. Currently these two messages are concatenated together due to our data restrictions. We are actively working on separating them, but we do not want to let that block you from digging into the data.

Download raw interactions data and feedback data

Click on the download button on the top right corner to download the table as a csv file for more in-depth analysis. Please note the csv file is currently limited to 5K rows.

What are the dimensions of the tables?

View Interactions

Column NameDescription
Timestamp (UTC)The date and time of the interaction
User IDThe user record ID
Email AddressThe user’s email address
Interaction TypeThe type of the interaction, including
- Free-form Text: User entered a query in the form of text
- Button Click: User clicked on a button from a bot’s response
- Link Click: User clicked on a link in the bot’s response
- UI Form submission: User submitted a form on the reference pop up page, including ticket filing, form, and in-app feedback
Interaction ContentThe details of the interaction, such as the user’s input query
Bot ResponseThe Copilot’s response to the user’s interaction, returned in chat.
Number of CitationsThe total of citations associated with the response
Plugin DetailsThe plugins called for the user’s interaction and the status of the plugin. If there is more than one plugin called for the interaction, the field will be a list of plugins and their status.
DepartmentThe user’s department
LocationThe user’s location
CountryThe user’s country

View User Feedback

Column NameDescription
TimestampThe date and time of the interaction
User IDThe user record ID
User EmailThe user’s email address
Feedback TypeThe feedback type: 👍 helpful or 👎 unhelpful
Feedback TextThe free-form text user provided
Copilot ResponseThe Copilot’s response after submitting the feedback, concatenated with the Copilot response the feedback was provided to.
(Please note this is a current limitation in our reporting pipeline, and we are actively working on making a cleaner separation between the responses)
DepartmentThe user’s department
LocationThe user’s location
CountryThe user’s country

How to get access to Copilot Insights?

Customers who are in Moveworks Next-Gen Copilot Limited Preview/Preview will have access to the Copilot Insights dashboard.

To access Copilot Insights, login to my.moveworks.com, click on “Analytics”, and click on “Copilot Insights” on the left panel of the dashboard.

To add more viewers for Copilot Insights, please follow the steps outlined here Manage Roles and Permissions.