Overview

The knowledge insights report is available to all customers with access to Copilot Insights. It provides insights into the content served through Copilot as part of the knowledge plugin response. This report covers the usage of the knowledge plugin and details at a citation level.

How to access the report

Knowledge insights is a new report that is present under copilot insights. You can access this dashboard by accessing the plugin insights present in the analytics application on the navigation sidebar

💡

Users with a “Bot analytics admin” or “Bot analytics viewer” role for the analytics application can access this report. Please add the users using the Roles and Permissions tool if they cannot see this report under the copilot insights tab.

Key terms used in the dashboard

Filters and widget types

The Knowledge insights report allows you to filter copilot adoption data for a selected time period, department, location, country and based on user preferred language.

Select date interval for which you want to review the served content details

By default the dashboard show knowledge plugin performance for previous 30 days. This can be changed using the date interval selection. You can select previous months (or) quarters using the selection available in the dropdown.

The dashboard also allows custom date interval selection. You can access that using the “select custom date” button.

💡

Please note, if you are choosing custom dates, the report data will be projected up to one day before the chosen end date.

Select department/ location/ country/ language for which you want review the served content details

Knowledge insights report provides 4 top level filters which can used to filter the data

The department/location/country value is ingested from your external system used in user identity. Please refer here to learn more on user identity ingestion.

Example use-case: I want to see the served content details for US region users

Select the region dropdown and search for US region

Once the filter is applied all metrics will only be shown for the users belonging to US region users

Widget types used in this dashboard

1. Trend chart

Line chart or Bar chart that presents the values of one or more metrics over a period of time, with selectable time intervals (day, week, month). For example: trend of active user over a selected time interval.

The day, week, month breakdown is only visible for a selected time interval

3. Table

A tabular list of one or more dimensions and their metrics. For example: Knowledge gap analysis, this table provides details on the detected topics where knowledge plugin was called by the copilot and there were no results that were served to the end users.

4. Bar chart

A graphical chart that presents the value of one metric for each value of a dimension on horizontal or vertical bar chart. For example: Served content volume, this chart shows the total number of content being served by copilot based on a daily/weekly/monthly breakdown.

Deep dive into widgets

Trend of knowledge base plugin

This widget provides the trend of interactions where a knowledge base plugin was called by the copilot and results where shown to the end users successfully. The trend chart provides a distribution by either day/week/month for the selected time period. This widget helps in understanding how the knowledge base plugin is being utilized by the copilot to help the end users.

Trend of content items cited by the copilot

This widget provides the number of content items cited by the copilot through a bar graph. The bar graph provides a distribution by either day/week/month for the selected time period. This widget enables insights on the volume of content used by the copilot to help the end users.

Breakdown of content items cited by the domain

This widget provides a detailed breakdown on the domain of the interactions where a knowledge base plugin was used to answer the end user query. This domain refers to the content domain which is configured during the ingestion. You can refer to your configured domain in the “Internal sources” section under Moveworks Setup. This widget enables insights on the most used content by domain.

Feedback on served content

End users are able to provide feedback on the served content by utilizing the 👍🏻 and 👎🏻 button and submitting the feedback form. This enables insights on the end user sentiment on the served content and the quality of the content.

Content items cited by copilot.

This widget shows the citations that are served by the copilot as a part of the summarized response. The table provides details on the following points

1. Article name/ FAQ/ File name
2. Total interactions where this citation was utilized by the copilot
3. Number of times users have marked the copilot response as helpful
4. Number of times users have marked the copilot response as unhelpful
5. Number of times users did not provide any feedback on the copilot response
6. Number of times users have filed a ticket by accessing “Handoff” button

This enables insights at a citation level and provide information on what content needs to be updated and is not helpful to the end users.

Topic insights

This widget shows topics detected by the copilot when a knowledge base plugin was used to provide a response to the end user. The topics are detected by the Moveworks entity detection model by parsing the user utterances and generally refer to a software (or) any platform the user is facing issues on. This widget enables insights which are the most popular topics where users are served a knowledge article

🚧

The topic prediction is going through a revamp on the Moveworks end. We are moving the topic prediction at the conversation level which allows a better coverage and precision. This will be reflected in the dashboard in due time. You might see "n/a" currently on the dashboard till the topic prediction is improved

Knowledge gaps

A knowledge gap is defined as interactions where copilot tried to use the knowledge base plugin to provide a response to the end user but was not able to any relevant answers. There are no citations served to the user in this case. This widget enables insights into these topics. The table shows breakdown of the topic name and the total number of interactions where knowledge base plugin wasn’t able to answer the user query. This enables insights on which topic needs more knowledge content and updates to the existing knowledge content.

🚧

The topic prediction is going through a revamp on the Moveworks end. We are moving the topic prediction at the conversation level which allows a better coverage and precision. This will be reflected in the dashboard in due time. You might see "n/a" currently on the dashboard till the topic prediction is improved

What are key questions I can answer using this dashboard?

What are user utterances when a particular knowledge content is served to the end user ?

To understand the user utterances for a knowledge content you need to visit the interactions table. Please find the name of the content item that you are interested in from the “Content items cited by the copilot” table and visit the “ingested knowledge” screen in Moveworks Setup. Search with the content item name and fetch the content ID. Once the content ID is available visit the interactions table and filter based on the content ID to find all interactions where the content item was used by the copilot.

1. Find cases where a ticket is filed as well by the end user

   Users are also able to file a ticket if they want to escalate their issue to the service desk. Utilize the “Ticket filed” filter and select “User initiated ticket” to understand the cases where end users ended up filing a ticket when a particular content item was served.

   

What is the user free-text feedback when a particular knowledge content is served ?

Follow the same process as above till fetching the content ID and visit the feedback table to understand user-free text feedback when a particular knowledge content is served.

What are the users asking when knowledge plugin is not able find any relevant answers ?

Knowledge gaps table enables insights into these cases. Select the topic that you want to learn the user utterances for and visit the interactions table and filter by the selected topic also select the “Plugin details” filter and filter for cases where Knowledge base plugin is incomplete. There are two values that needs to be selected to see the exact knowledge gaps.

1. "Knowledge base : Completed - Success with 0 citations" in this case copilot was able to find knowledge articles but they were not relevant to the user query
2. "Knowledge base : Completed - No results" in this case copilot was not able to find any knowledge article for the user query

Troubleshooting and FAQ’s

What is the data refresh policy on this dashboard?

Copilot conversation data is processed on a daily basis and is updated by 4:00 PM (PT) every day for the previous day's data. The demographic data, such as user department and location, is fetched from the user identity ingestion process, which runs approximately every 4 to 6 hours (Please note: The schedule for the identity ingestion pipeline can be configured differently for your organization).

Why this dashboard is not visible to few users?

Please ensure the user has the “Bot analytics admin” or “Bot analytics viewer” role to access the dashboard. You can provide user the required role using the RBAC tool.

Why the dashboard is taking too much time to load ?

The latency for the dashboard is dependent on the selected time interval. If you see longer load times and failure in loading dashboard please reach out to the Moveworks support team.