Bot Performance Insights (BPI+) FAQ

Q: When is the data available in analytic dashboards?

A: Data will be available in the Moveworks Analytics portal at 4pm PT for reporting on the previous day's data e.g. data for 12/1 will be available by 12/2 4pm PT.

All Moveworks data is processed based on UTC timezone. Taking 2024-05-10 data as an example (which means on dashboards, the data has dt = 2024-05-10) , the following table shows from each region's perspective, what local time range's interactions are included, as well as the SLA from local time zone perspective.

In summary, SLA of reporting data for each UTC day is set at 23 hours after that UTC day ends.

Q: What date period is covered by the data in BPI+, i.e. when does the data start?

A: Data in BPI+ starts on August 18, 2023. Please reach out to Moveworks Support if you'd like to request data for the older period.

Q: How to filter data on time period in BPI+?

A: Date filters are applied independently for each of the tabs Resolution, Acceleration, Adoption. One can use pre-configured filter on a month or a quarter, or pick custom start and end dates of the desired time period.
Note: for customer dates filter, starting date is included and end date is excluded for the date. For example, if the filter is set from May 1 to May 15, the data will include dates from May 1 to May 14.

Q: Why row-level data downloaded from side panel does not match the data displayed in the table in side panel itself?

A: Currently, number of rows available to download is limited to 5000. If the data exceeds this limit, one can use filtering on shorter time frame or other parameters to download smaller, more targeted data set within the limit. Please reach out to Moveworks Support team if you need further help.

Q: Why number of Employees with Bot Resolution exceeds number of Employees with Issues?

A: This may happen if a significant portion of resolved issues are resolved by the Access Account skill through Password Reminders. In this case, the bot prevents an issue from occurring by helping the user proactively. While resolved issues contribute to the number of Employees with Bot Resolution, unaddressed Password Reminder notifications do not contribute to the number of Employees with Issues (notification does not constitute an employee issue).

Q: What is defined as an issue?

A: In general, an issues is a work-related issue that is experienced by an employee. In order to analyze bot usage and performance, we recognize these aspects around concept of issues.

1. Issues that are recorded as tickets and are qualified for resolution by bot.
2. Issues that are described directly in chat that bot attempts to resolve (more specifically, passing resolution skills pre-trigger).
3. Issues that are resolved by one of bot resolution skills.

In various analytic use cases, we leverage these concepts correspondingly. For example, Bot Effectiveness describes what portion of issues that would need resolution by agent is resolved by bot and is calculated as c/(a+c) , thus showing portion of resolved issues out of all issues resolved or ticketed. For insight into issues that are described in bot but later abandoned (neither resolved nor escalated to a ticket), one may need to look into b.

Q: Why there are tickets with Issue Result ”Tickets intercepted by Bot” that are not resolved by any skill?

A: The label ”Tickets intercepted by Bot” stands for the tickets that are polled and qualified for resolution attempt by any of Resolution skills. For a portion of these tickets, if there is a solution to offer, the bot reaches out to the user on the ticket with an appropriate suggestion. If the user follows up on this suggestion then the ticket may be resolved as defined for corresponding Resolution skill.

Q: How can I know what people are asking about, that is not resulting in Answers?

A: That can be done by going to the right-side panel by clicking on the Answers row in resolution breakdown bar chart, setting the filter on Skill Result to “Not Served” and downloading the data export at the bottom of the panel.

Q: Why an utterance and Answers result “Not Served“ would have a knowledge article associated with it?

A: The Skill Result describes the result of how the Answers skill performs, but not necessarily whether a knowledge item was suggested to the user. So in these particular case, it means that the Answers skill found those articles, but they did not pass the threshold nor won over other skills, so the Answers skill did not serve them to the user.

Q: How can I find underperforming articles?

A: You can identify underperforming articles by filtering the Answers side panel by Skill Result=”Engaged On Skill But Not Resolved”, scrolling down and viewing the table data under the pillow button “Article Details”. The data will show user utterances and articles served for cases when users rated the answer as not helpful and/or proceeded to file a ticket for the issue.

Q: How can I track how many time employees file tickets from bot and to which utterances?

A: Under Interactions tab, we show the number of interactions where user submits a ticket in bot. For details, under the Resolution tab, click on “View issues” to see row-level data for each issue with utterance text and a column IS TICKET ESCALATED?.

Q: Is there a way to filter out test data so it doesn’t populate in dashboards and skew analytic metrics?

A: Yes. The testing data is filtered out in two ways:

1. Analytic data is processed starting on the date when the bot is live in the organization. Your Moveworks Customer Success teams owns this process and will set up the launch date accordingly. If the bot was tested on prior dates, that usage data will not appear on the dashboards.
2. Users who are testing bot functionality, can be tagged as TESTER in the user roster. Usage data from these users is filtered out from the data shown in analytic dashboards for all times.

Q: [Tableau BPI] What does label n/a mean in the breakdown of active users by platform where they interact with the bot?

A: For small portion of users who only click on a link without having sent a message nor pressing a button to the bot, the platform can not be detected due to technical limitation and is set to value n/a

Q: How do labels in Answers Insights Tableau dashboard correspond to labels in BPI+ Answers side panel?

A: In BPI+, result of the Answers skill is described by columns “SKILL RESULT” (also available as a filter), "IS AGENT HANDOFF?”, ”IS TICKET ESCALATED?” and ”ISSUE RATING”. This allows us to characterize the results in a more granular manner then the single label “Result” which is used in the Answers Insights Tableau dashboard.

Q: How can I calculate article Effectiveness from BPI+?

A: Article Effectiveness shows how effective an article is in resolving employees’ issues, specifically, how often the article resolves an issue comparing to how often the article is served to the user. For example, is a snippet “How to install Zoom?” is served in response to 10 issues, and in 3 cases user rates it as not helpful, while in 7 cases issues are resolved (user does not rate is as not helpful and does not file a ticket/agent handoff), effectiveness is equal to 7/10=70%.

By definition, article Effectiveness is calculated as the ratio of number of cases with positive or no rating to number of all cases with user engagement. This corresponds to the ratio of number of issues with Skill Result “Resolved by Skill” to the number of issues with Skill Result equal to “Resolved by Skill” or “Engaged On Skill But Not Resolved”:

Article Effectiveness = (“Resolved by Skill”) / (“Resolved by Skill” + “Engaged On Skill But Not Resolved”)

This metric can be reviewed in the Answers skill side panel in the chart “Top knowledge served”, or calculated on the data downloaded from “Article Details” table data.

Q: How can I calculate topic Coverage from BPI+?

A: Topic Coverage shows how well the bot is able to serve knowledge for particular category of employees issues.

By definition, topic coverage is defined as the ratio of number of times when an article is served to the number of all attempts to do so. This corresponds to the ratio of number of cases with Skill Result equal to any of “Resolved by Skill”, “Engaged On Skill But Not Resolved” or “Engaged In Bot But Not On Skill”, to the number of cases with any Skill Result (“Resolved by Skill”, “Engaged On Skill But Not Resolved” or “Engaged In Bot But Not On Skill” or “Not Served”).
For example, if for 10 issues with the topic “Zoom” the bot serves articles for 4 issues and does not serve any articles for the other 6 issues, Coverage is equal to 4/10=40%.

Topic Coverage = (“Resolved by Skill” + “Engaged On Skill But Not Resolved” + “Engaged In Bot But Not On Skill”) / (“Resolved by Skill” + “Engaged On Skill But Not Resolved” + “Engaged In Bot But Not On Skill” + “Not Served”)

This metric can be calculated on the data downloaded from “Article Details” table data from Answers skill side panel.

Q: How can I see the trend of number of issues resolved or not resolved by Answers in BPI+?

A: In the Answers skill side panel, scroll down to see the trend “How is the usage of Answers trending over time?”. This bar chart shows number of Issues Resolved and Issues Unresolved by Answers. These two metrics correspond to metrics Resolution Provided and Answers Provided in Tableau dashboard Answers Insights.

Q: How metric Tickets with Approvals processed in Tableau BPI dashboard is different from BPI+?

A: In Tableau dashboard, value of Tickets with Approvals processed accounts for approvals processed for the first time only. In BPI+, the metrics accounts for all cases when approval is processed in the bot, including those when approval was already processed earlier by another approver. Therefore, value of Tickets with Approvals processed in BPI+ may be higher.

Q: How can I see usage of external vs internal Knowledge in BPI+?

Please filter on “Doc Source” in the side panel for Answers. This field is also available in the row-level data under “Article Details” pillow button.

How can I see which Domain drives more adoption in bot? In BPI+ there is no filter Domain on Adoption tab.

“Domain” is in fact an attribute of an issue, not a user. Same user can ask any of IT- or HR- or Finance-related questions. This is why in BPI+ we removed domain as a breakdown for users as it is not applicable cleanly.
We can still reason about which domain drives adoption by looking at distribution of issues by Domain in Resolution tab.

Why the value of employee’s department (location, country) is set to "n/a"?

This is a UX treatment (abbreviation of "not available") to indicate that there is no value in the field that is mapped to department (location, country) in analytics. User attributes such as department, location or country are reported as they are set up in the customer user systems data that is ingested by Moveworks systems.