📘

Copilot Insights is exclusively for Moveworks Copilot

Copilot Insights is the analytics dashboard exclusively for the Moveworks Copilot, featuring data solely from this platform. If you are currently using Classic analytics dashboards like Bot Performance Insights or Answers Insights, please note that they do not incorporate data from the Moveworks Copilot.

Overview

Copilot Insights represents the latest in Moveworks analytics for the Moveworks Copilot, offering comprehensive visibility into end users' interactions with the Copilot. This includes detailed insights into user adoption rates and Copilot performance, providing a transparent overview for actionable insights.

What are the key components of Copilot Insights?

Interactions

Copilot Insights offers complete transparency into every facet of user interaction with the Copilot, enabling a thorough replay of each engagement for in-depth insights. This encompasses various user interaction types, including free-form text, button clicks, link clicks and Copilot's responses.

User Adoption

Copilot Insights grants visibility into the extent and depth of user adoption for Copilot. Ranging from active user counts to popular plugins, the data is customizable, allowing you to analyze adoption trends across various demographic dimensions.

Performance

Copilot Insights offers an unbiased perspective on Copilot performance. Users can assess the success rates of each plugin call, delve into the underlying causes of unsuccessful interactions, and access in-chat feedback from end users for a comprehensive understanding of system efficiency.

Key definitions

Interactions: all types of engagements end users have with the Copilot. There are 4 types of interactions:

Plugin: The Moveworks Copilot is a versatile and flexible engine powered by the latest large language models. On top of the Copilot platform, we’ve built plugins to execute specific tasks. If the Copilot engine is the brain of a person, these plugins are the hands and legs that carry out instructions from the Copilot. The Copilot includes over 20 out-of-the-box plugins that provide a wide range of capabilities including search and take action. Below is a list of our out-of-the-box plugins.

In addition to the out-of-the-box plugins, each Creator Studio use case will also appear as a plugin, with the use case name as the plugin name.

Plugin Call: To assist with a user request, the Copilot selects the relevant plugin to execute, this process is a plugin call. There can be multiple plugins called to fulfill a single user request. For example, a user could ask “Can you add my manager to the Sales group?”, and both the People Registry and the Add User to Group plugins are called.

Plugin Status: When a plugin is called, there are 3 final status that help indicate the result of the plugin call, and they are referred to as plugin status. The 3 status are:

What are some key questions I can answer with Copilot Insights?

We will dig into the Copilot Insights user experience by answering each of the key questions below:

What interactions did my user have with the Copilot?

In the example below, observe an upward trend in user interactions with Copilot over the past few weeks. The predominant interaction type is through free-form text, indicating that users prefer engaging with Copilot through chat.

How is my user adoption trending over time?

In the provided example, Copilot usage displayed a gradual upward trend, with a significant spike observed in the current week. This aligns with the user interactions chart above, suggesting that the surge in usage can be attributed to an increased number of users.

What are the most common asks from my user?

In this example, the Knowledge Plugin emerged as the most utilized, signifying that the majority of user requests were related to knowledge searches. Examining the figures, it's noteworthy that 3,000 out of 14,000 knowledge searches did not find resources from the connected systems. For a more in-depth analysis of these user queries, refer to the next question below.

What are topics in user interactions?

Topics in the context of interactions and feedback table refer to the main subjects of communication between users and the copilot. These topics emerge from free-form text interactions and utterances related to UI Form submissions where users seek specific information or assistance. Powered by Moveworks' entity detection model, topics are categorized into two main types:

1. General Entities: Broad themes that are not specific to any company.
2. Company-Specific Entities: Themes that are tailor-made from entities & terminologies specific to a particular organization.

For more details, see Entity Prediction.

Topic value is set to “n/a” for the general user utterance when the model is not able to find a suitable entity, for example, “Hi” or “Yes”.

The categorization into these topics is initiated when a user interacts with the copilot, asking for help or information about specific items or issues.

The most common topics that users interact about are grouped into four primary themes, which help in understanding the nature of requests:

• Software Requests: Users often inquire about specific software, resulting in the software's name being captured under this topic, for example, “Zoom” or “Microsoft Outlook”.
• Filed Form: This category includes requests for hardware or physical items such as monitors and laptops. This topic also appears on a request with a Form filed. The requested item is specifically noted, for example, “AC adapter” or “New Phone”.
• Ticket Requests: This theme is for interactions related to ticketing operations, encapsulating fixed operations like filing a ticket, fetching user tickets, or adding a comment on a ticket. This topic will appear as “Ticket Operations”.
• Other Requests: This includes any request that does not fit into the above categories, capturing the primary action or query made by the user. An example could be a question on "how to open an email on mobile," which would be categorized under "Email."

You can define new company-specific entities by connecting with your CS team. Once those are added the entity detection model will categorize the user interaction into those entities based on the request.

In the dashboard, one can find topic values in the Interaction data in the side panel.

What feedback did my user give?

View end users’ feedback of the Copilot by clicking on “View User Feedback”.

What new knowledge should I write about?

For a more in-depth analysis of knowledge searches with no results, click on 'View Interactions' to open the side panel containing raw interaction data.

Narrow down the user interactions by applying filters of interest. For example, if you want to see users’ knowledge search that did not return any result, apply the following filters:

• Interaction type = ‘Free-form Text’
• Plugin Details = ‘Knowledge Base: Completed - No Result', select all options that contains this value

You can review the “Interaction Content” column to understand what user queries did not receive an answer. This can help inform new knowledge to be added.

What knowledge articles should I update?

You can infer the quality of your knowledge base through end user feedback.

Open the “View User Feedback” table and filter on the unhelpful feedback. You can read through user queries, bot response, user feedback, and the KB articles associated with it. While the user feedback is not directly tied to each knowledge article, you can extract helpful insights from the feedback text.

In addition, you can also download the table into an Excel or BI tool, and group the feedback by article names. If there is one particular article that received a lot of negative feedback, that’s a good signal to update it.

What HR (or any other domain) questions did my users ask about?

You can narrow down the users’ questions with the “Domain” filter. There is a global filter which allows you to apply the filter to every chart on the dashboard, like illustrated below.

Or, you can apply filters directly to the “View Interactions” and “View User Feedback” tables.

Download raw interactions data and feedback data

Click on the download button on the top right corner to download the table as a csv file for more in-depth analysis. Please note the csv file is currently limited to 15K rows.

What did my user ask in the entire conversation?

If you see an interaction that is potentially from a multi-turn conversation, you can download the raw data tables and group interactions with “Conversation ID”. This allows you to see all interactions associated with that conversation. Then, sort the interactions by “Timestamp” to view the interaction in time order.

What links did my user clicked on?

During interactions, Copilot provides links to cited resources and facilitates a smart handoff within the Copilot response. Currently, user clicks on these Copilot-provided links are recorded as new interaction rows, categorized under a "link click" interaction type.

There are two principal types of links captured:

1. Internal Links - These are links that navigate within the Moveworks domain. Examples include handoff links, citation links, user profile links, etc.
   1. These links have expiration mechanism implemented at moveworks end
      • If a citation link is never clicked, it expires in 3 days
      • When a citation link is clicked, it expires in 5 minutes
      • The pop up page will stay activate as long as you don't close it in browser.
2. External Links - These links direct users to systems outside of Moveworks. Examples include ServiceNow ticket links, SharePoint site links, etc.

Below is a list of internal links provided by Copilot, which can be found under the "link click" interaction types.

Please note this is not a exhaustive list if we make changes to how internal links are formatted we will update the following table

• {unique-id}: This is a unique ID assigned to each URL, differentiating the URL served from one user to another.

• {unique-token}: This is a unique token assigned to each URL. It manages the expiration mechanism for the URLs.

Ticket details in copilot insights

Ticket are filed through the copilot in the following two scenarios

1. User has accessed handoff to file a ticket or directly ask the copilot to file a ticket. This is referred as user initiated ticket
2. Moveworks Copilot has filed the ticket on behalf of the end user to track account unlock request, software access request etc. This is referred as bot initiated ticket

Copilot insights covers ticket details on both of the scenarios. There are 3 different data points which reflect ticket details.

1. Ticket ID - This reflects the ticket number that was filed during the conversation. This data is only populated at the interaction turn where a ticket was filed and the subsequent form submission by the end user.
2. Ticket URL - This reflects the ticket service portal URL. This is hyperlinked with the Ticket ID.
3. Ticket filed - This reflects the scenario, either user filed the ticket by clicking on get help or directly asking the Copilot to file a ticket - Moveworks Copilot has filed the ticket as a receipt to track access requests. This data is populated in all interactions belonging to conversation where a ticket was filed.

How to find all conversations where a ticket was filed by the end user

To find all interactions and conversations where a ticket was filed by the end user. Please use the "Ticket filed" filter. Deselect the "All" option and click on the "User initiated ticket" option. This provides all interactions of all conversations where a ticket was filed by the end user.

How to find all conversations where a ticket was filed by the bot as a receipt

To find all interactions and conversations where a ticket was filed by the bot. Please use the "Ticket filed" filter. Deselect the "All" option and click on the "Bot initiated ticket" option. This provides all interactions of all conversations where a ticket was filed by the bot.

Understanding a conversation where a ticket was filed

Let's go through an example conversation where a ticket has been filed. For demonstration purposes we will leave out few columns from the interaction table. Please note all of the interaction rows in this demonstration are sorted by timestamp field in a descending manner.

What are the dimensions of the tables?

View Interactions

View User Feedback

How to get access to Copilot Insights?

Customers who are in Moveworks Copilot Limited Preview/Preview will have access to the Copilot Insights dashboard.

To access Copilot Insights, login to my.moveworks.com, click on “Analytics”, and click on “Copilot Insights” on the left panel of the dashboard.

To add more viewers for Copilot Insights, please follow the steps outlined here Manage Roles and Permissions.