BPI+ Metrics Dictionary
Resolution metrics
| Metric name ..................................... | Metric description ...................................................... | Where to find the metric ................................................. | Example SQL | SQL for SFTP data |
|---|---|---|---|---|
| Number of issues resolved by bot | Number of issues resolved by the bot, including helpful or not rated Answers, submitted Forms and Access issues resolved. This metric shows how many issues were resolved by one of the bot resolution skills: - Answers: Bot provided answers that were rated helpful or not rated. This excludes answers that were either rated negatively or followup up with a ticket filed or agent handoff; - Account Access Enabled: Bot allowed employees to reset passwords or unlock accounts; - Group-related Operations Provided: Bot created or changed membership for groups and distribution lists; - Software Access Enabled: Bot provided access to a software application; - Forms Filled: Employees filled a form or structured request recommended by bot. | B- top metric: - time trend - breakdown by contact type - breakdowns by skill, topic, route, ticket avoided, department, language, location, domainTab Resolution → side panels in Skill breakdown:- top metric | SELECT COUNT(global_issue_uuid)AS "Distinct Issues Resolved by Bot" FROM bpi_issue WHERE ((is_resolved_by_bot = 'true')) | select count(*) from {{date}}_daily_detailed_report where skill_category not in ( 'Interaction', 'Lookups') and skill_detail<>'AskMe rated Not Helpful' |
| Mean time to resolution, minutes | The average time between when an issue was created and when it was resolved in the bot. It is calculated as time interval in minutes, from the time when an issues was created, to the time when bot provides resolution of the issue. Outlier values over 1440 minutes (24 hours) and issues from bot notification origin are excluded to decrease metric volatility. | Tab Resolution: - top metric - time trend - breakdowns by department, language, location, domain | SELECT AVG(num_mins_to_resolution) AS "Mean Time to Resolution" FROM bpi_issue WHERE ((num_mins_to_resolution IS NOT NULL) AND (num_mins_to_resolution < 1440) AND (dim_issue_origin != 'BOT_NOTIFICATION')) | select avg(minutes_to_resolution) from {{date}}_daily_detailed_report where skill_category not in ( 'Interaction', 'Lookups') and skill_detail<>'AskMe rated Not Helpful' and minutes_to_resolution is not null (newer SFTP versions) |
| Number of employees with bot resolution | Number of employees who have at least one issue resolved by the bot in the selected time period. | Tab Resolution: - top metric - top side panel - time trend - breakdowns by department, language, location, domain | SELECT COUNT(DISTINCT user_record_id)AS "Employees with Resolution" FROM bpi_issue WHERE ((is_resolved_by_bot = 'true')) | select count(distinct user_id) from {{date}}_daily_detailed_report where skill_category not in ( 'Interaction', 'Lookups') and skill_detail<>'AskMe rated Not Helpful' (newer SFTP versions) |
| Number of issues resolved or ticketed | Number of issues that are either resolved by bot or are recorded as tickets qualified for bot resolution. Issues can originate from DM, from shared IT channel, from a ticket polled by bot, or from a notification proactively sent by bot. Tickets qualified for resolution typically exclude tickets that the bot is unable to resolve, such as system generated tickets, tickets that are not for internal support, and tickets that are outside of the domains supported by the bot. | Tab Resolution: - top metric | SELECT COUNT(global_issue_uuid) AS "Total Issues Submitted" FROM bpi_issue WHERE ((is_qualified_for_resolution_denominator = 'true')AND ((is_resolved_by_bot = 'true') OR (ticket_id IS NOT NULL))) | Not available |
| Breakdown "How effective is the bot at resolving issues?” | The chart shows issues resolved or ticketed broken down by contact type. Value of contact type is set to the bot name if the ticket is created from bot, or to “Shared Chat Channel” if it is created in bot channel. Otherwise it is set to the contact type as indicated in the ticketing system. If it is not present, the value is set to either “Service Portal” (or similar) or to numeric contact type (for some ITSM systems). | Tab Resolution: - breakdown by contact type | SELECT dim_contact_type, COUNT(global_issue_uuid)AS "Total Issues Submitted" FROM bpi_issue WHERE ((is_qualified_for_resolution_denominator = 'true')AND ((is_resolved_by_bot = 'true')OR (ticket_id IS NOT NULL))) group by dim_contact_type | Not available |
| Number of unique employees with issues resolved or ticketed | Number of employees with issues that are either resolved by bot or are recorded as tickets qualified for bot resolution. Issues can originate from DM, from shared IT channel, from a ticket polled by bot, or from a notification proactively sent by bot. | Tab Resolution: - top metric | SELECT COUNT(DISTINCT user_record_id)AS "Employees with Issues Resolved or Ticketed" FROM bpi_issue WHERE ((is_qualified_for_resolution_denominator = 'true') AND ((is_resolved_by_bot = 'true') OR (ticket_id IS NOT NULL))) | Not available |
| Bot Effectiveness | Bot effectiveness shows how effective the bot is in resolving employee issues. Bot effectiveness ratio is equal to percentage of issues resolved by bot, out of all issues that are either resolved by bot or are ticketed to be further handled by agents. More specifically, numerator is defined as number of issues raised in bot and resolved by bot and denominator is number of issues described in bot, with either a resolution or a filed ticket. Denominator does not include questions that were rated negative but no ticket was filed, since this scenario does not call for further help on the issue. Bot effectiveness = (number of issues raised in bot and resolved by bot) / (number of issues described in bot, with either a resolution or a filed ticket) | Tab Resolution: - subtitle in the bar chart showing breakdown by contact type | SELECT 1.0* COUNT(CASE WHEN ((is_resolved_by_bot = 'True'))THEN global_issue_uuid END) / COUNT(global_issue_uuid) AS "Bot Effectiveness" FROM bpi_issue WHERE ((is_qualified_for_resolution_denominator = 'true')AND ((is_resolved_by_bot = 'true')OR (ticket_id IS NOT NULL)) AND (is_created_by_bot = 'true')) | Not available |
| Number of issues submitted by users List of issues in side panel under “View Issues” button | Number of issues consists of two parts: (1) issues described in bot and (2) tickets polled and qualified for resolution by bot. More specifically: 1. Issues that are described in bot are considered qualified for resolution if any of resolution skills attempts to resolve the issue (as opposed to be non-applicable). For example, if user describes an issue with access or troubleshoots for an issue with an app, such issues are qualified for resolution because intent of these issues triggers bot’s resolution skills. If user types a non-issue utterance like ‘Hi’ or ‘Cancel’, these issues are not counted toward this metric. 2. poled tickets that are filtered according to the preset filter to be qualified for resolution by bot. | Tab Resolution: - top side panel - breakdowns by topic, department, language, location, domain, “ticket avoided” | SELECT COUNT(global_issue_uuid) AS "Total Issues" FROM bpi_issue WHERE dim_issue_result is not null | Not available |
| Number of issues: breakdown by Route | Breakdown by Route shows issues submitted by users by the issue origin. - ”Messages to Bot” are issues described by users directly in DM - ”Tickets intercepted by Bot” are tickets that were polled and qualified for resolution attempt by any of Resolution skills. For portion of these tickets, if there is a solution to offer, bot reaches out to user on the ticket with an appropriate suggestion. - ”Messages to Bot - Channel Utterances” are issues described in shared IT channel - “Proactive Messages by Bot” are issues proactively raised by bot via notification | Tab Resolution: - breakdowns by route | SELECT COUNT(global_issue_uuid) AS "Total Issues" FROM bpi_issue WHERE dim_issue_result is not null | Not available |
| Number of issues with solution offered | Number of issues with solution offered by one or more of the bot resolution skills. For example, if user describes an issue and in response bot offers articles by Answers and suggests a Form to file, this issue in counted toward this metric. | Tab Resolution: - top side panel - breakdowns by topic, route, department, language, location, domain, “ticket avoided” | SELECT COUNT(global_issue_uuid) FROM bpi_issue WHERE dim_issue_result IN (’Bot Offered Solution’, ‘Bot Solutions Engaged’, ‘Issue Resolved’) | Not available |
| Number of issues with users engaging with solution | Number of issues where bot provided one or more solutions that the user is engaging with in the bot by clicking on corresponding link or button. For example, if user describes an issue for whic bot offers articles by Answers and suggests a Form to file, and user clicks on an article link, then this issue is counted toward this metric. | Tab Resolution: - top side panel - breakdowns by topic, route, department, language, location, domain, “ticket avoided” | SELECT COUNT(global_issue_uuid) FROM bpi_issue WHERE dim_issue_result IN (‘Bot Solutions Engaged’, ‘Issue Resolved’) | Not available |
| Number of issues: breakdown by issue result (Issue Result = "Issue Submitted by User”, “Bot Offered Solution”, "Bot Solutions Engaged”, ”Issue Resolved”) | Breakdown of issues by “Issue result” shows exclusive buckets of issues depending on the outcome. Issue Result = "Issue Submitted by User”: issue was submitted by user but bot did not offer any solutions Issue Result = “Bot Offered Solution”: bot offered a solution to the issue but user did not engage with it Issue Result = "Bot Solutions Engaged”: user engages with solution offered by bot but bot does not resolve the issue Issue Result = ”Issue Resolved: bot resolves the issue with one or more resolution skills | Tab Resolution: - top side panel - breakdowns by topic, route, department, language, location, domain, “ticket avoided” | SELECT dim_issue_result, COUNT(global_issue_uuid) FROM bpi_issue WHERE dim_issue_result is not null GROUP BY dim_issue_result | Not available |
| Number of utterances in shared IT channel | Number of Channel utterances shows how many of issues described in shared IT channel bot attempted to resolve with one of resolution skills. Utterances that are not meant to be resolved (such as utterances typed into the channel by IT agent or utterances in threads) are not counted toward this metric. | Tab Resolution → side panel for Messages to Bot - Channel Utterances in Route breakdown:- top metric | SELECT COUNT(global_issue_uuid) AS "Num Channel Issues" FROM bpi_issue WHERE ((dim_issue_origin = 'CHANNEL_UTTERANCE') AND (has_resolution_skill_pass_pretrigger = 'true')) | select count(*) from {{today's date}}_daily_channel_issues |
| Number of Bot reach-outs in shared IT channel | Number of Bot reach-outs shows for how many issues described in shared IT channel bot has offered a solution from one of resolution skills. For example, if user described as issue in IT channel and bot suggested a solution to them in DM, this issue is counted toward this metric, even if user does not follow up with the bot in DM. | Tab Resolution → side panel for Messages to Bot - Channel Utterances in Route breakdown:- top metric | SELECT COUNT(global_issue_uuid)AS "Num Channel Reachouts" FROM bpi_issue WHERE ((dim_issue_origin = 'CHANNEL_UTTERANCE') AND (num_resolution_skill_offer_won >= 1)) | select count(*) from {{today's date}}_daily_channel_issues |
Skill metrics
| Metric name ..................................... | Metric description ...................................................... | Where to find the metric ................................................. | Example SQL | SQL for SFTP data |
|---|---|---|---|---|
| Number of issues with Skill Result=”Served but not engaged in Bot” | Bot suggests a solution by the skill but user does not engage with it. For example, bot offers a Form but user does not return to bot or returns to bot with another issue | Tab Resolution → side panel for Answers: - top metric - breakdowns by knowledge served, topics identified - time trend Tab Resolution → side panel for Form Filling: - breakdowns by forms served, topics identified - time trend Tab Resolution → side panel for Access Software: - breakdown by software provisioned - time trend Tab Resolution → side panel for Access Groups: - breakdown by subskill- time trend Tab Resolution → side panel for Access Account: - breakdown by subskill- time trend Tab Resolution → side panel for Messages to Bot - Channel Utterances: - breakdown by skill, topic identified - time trend | SELECT count(*) FROM bpi_issue_skill WHERE dim_skill_result = 'Served But Not Engaged In Bot' | Not available |
| Number of issues with Skill Result=”Engaged in Bot But Not in Skill” | Bot provides a solution by Resolution skill but user does not explore the solution. However user responds in generic way, for example, user does not click into suggested Answers link but clicks on Cancel button. Another scenario is when bot offers solutions by other skills as well and user engages with one of these other skills rather than with the skill relevant to this metric. | Tab Resolution → side panel for Answers: - top metric - breakdowns by knowledge served, topics identified - time trend Tab Resolution → side panel for Form Filling: - breakdowns by forms served, topics identified - time trend Tab Resolution → side panel for Access Software: - breakdown by software provisioned - time trend Tab Resolution → side panel for Access Groups: - breakdown by subskill- time trend Tab Resolution → side panel for Access Account: - breakdown by subskill- time trend Tab Resolution → side panel for Messages to Bot - Channel Utterances: - breakdown by skill, topic identified - time trend | SELECT count(*) FROM bpi_issue_skill WHERE dim_skill_result = 'Engaged In Bot But Not On Skill' | Not available |
| Number of issues with Skill Result=”Engaged on Skill But Not Resolved” | User engages with a solution offered by particular skill but does not proceed to full resolution by this skill. For example, bot provides a solution by Answers and user clicks on the link but then rates it as not helpful or proceeds to file a ticket | Tab Resolution → side panel for Answers: - top metric - breakdowns by knowledge served, topics identified - time trend Tab Resolution → side panel for Form Filling: - breakdowns by forms served, topics identified - time trend Tab Resolution → side panel for Access Software: - breakdown by software provisioned - time trend Tab Resolution → side panel for Access Groups: - breakdown by subskill- time trend Tab Resolution → side panel for Access Account: - breakdown by subskill- time trend Tab Resolution → side panel for Messages to Bot - Channel Utterances: - breakdown by skill, topic identified - time trend | SELECT count(*) FROM bpi_issue_skill WHERE dim_skill_result = 'Engaged On Skill But Not Resolved' | Not available |
| Number of issues with Skill Result=”Resolved By Skill” | Issue is resolved by one of the bot resolution skills: - Answers: Bot provided answers that were rated helpful or not rated. This excludes answers that were either rated negatively or followup up with a ticket filed or agent handoff; - Account Access Enabled: Bot allowed employees to reset passwords or unlock accounts; - Group-related Operations Provided: Bot created or changed membership for groups and distribution lists; - Software Access Enabled: Bot provided access to a software application; - Forms Filled: Employees filled a form or structured request recommended by bot. | Tab Resolution → side panel for Answers: - top metric - breakdowns by knowledge served, topics identified - time trend Tab Resolution → side panel for Form Filling: - breakdowns by forms served, topics identified - time trend Tab Resolution → side panel for Access Software: - breakdown by software provisioned - time trend Tab Resolution → side panel for Access Groups: - breakdown by subskill- time trend Tab Resolution → side panel for Access Account: - breakdown by subskill- time trend Tab Resolution → side panel for Messages to Bot - Channel Utterances: - breakdown by skill, topic identified - time trend | SELECT count(*) FROM bpi_issue_skill WHERE dim_skill_result = 'Resolved By Skill' | Not available |
| Number of issues by Skill Result | “Resolved by Skill” - Bot resolves issue using the corresponding Resolution skill ”Engaged on Skill But Not Resolved” - User engage with a solution offered by particular skill but does not proceed to full resolution by this skill. For example, bot provides a solution by Answers and user clicks on the link but then rates it as not helpful or proceeds to file a ticket ”Engaged in Bot But Not in Skill” - Bot provides a solution by Resolution skill but user does not explore the solution but responds in generic way. For example, user does not click into suggested Answers link and procedes to file a ticket. ”Served but not engaged in Bot” - Bot suggests a solution by the skill but user does not engage with it. For example, bot offers a Form but user does not file it | Tab Resolution → side panel for Answers: - top metric - breakdowns by knowledge served, topics identified - time trend Tab Resolution → side panel for Form Filling: - breakdowns by forms served, topics identified - time trend Tab Resolution → side panel for Access Software: - breakdown by software provisioned - time trend Tab Resolution → side panel for Access Groups: - breakdown by subskill- time trend Tab Resolution → side panel for Access Account: - breakdown by subskill- time trend Tab Resolution → side panel for Messages to Bot - Channel Utterances: - breakdown by skill, topic identified - time trend | SELECT dim_skill_name AS "Skill Name", COUNT(CASE WHEN ((dim_skill_result = 'Resolved By Skill')) THEN global_issue_uuid END)AS "Resolved By Skill", COUNT(CASE WHEN ((dim_skill_result = 'Engaged On Skill But Not Resolved'))THEN global_issue_uuid END)AS "Engaged On Skill But Not Resolved", COUNT(CASE WHEN ((dim_skill_result = 'Engaged In Bot But Not On Skill'))THEN global_issue_uuid END)AS "Engaged In Bot But Not On Skill", COUNT(CASE WHEN ((dim_skill_result = 'Served But Not Engaged In Bot'))THEN global_issue_uuid END)AS "Served But Not Engaged In Bot" FROM bpi_issue_skill WHERE ((dim_skill_result IS NOT NULL) AND (dim_skill_result != 'Not Served')) | Not available |
| Number of employees with bot resolution by specific resolution skill | Number of employees who had any of their issues resolved by one of resolution skills: Answers, Form Filling, Access Software, Access Account or Access Groups | Tab Resolution → side panels for Skill breakdown: - top metric | SELECT dim_skill_name, COUNT(DISTINCT user_record_id)AS "Employees with Resolution"FROM bpi_issue_skill WHERE ((is_resolved_by_skill = 'true')) group by dim_skill_name | select skill_category, count(distinct user_id) from {{date}}_daily_detailed_report where skill_category not in ( 'Interaction', 'Lookups') and skill_detail<>'AskMe rated Not Helpful' group by skill_category (newer SFTP versions) |
| Number of Answers snippets served | Number of Answers snippets served shows total number of times any knowledge snippet was offered to employee as a solution of their issues. For example, if same snippet was served in response to 5 different issues, it will add 5 to this metric. | Tab Resolution → side panel for Answers: - top metric | SELECT COUNT(offer_item_id) AS "Number of Served Offer Items" FROM bpi_issue_skill_item WHERE ((dim_skill_result IS NOT NULL) AND (dim_skill_result != 'Not Served') AND (dim_skill_name = 'Answers')) |
Acceleration metrics
| Metric name ..................................... | Metric description ...................................................... | Where to find the metric ................................................. | Example SQL | SQL for SFTP |
|---|---|---|---|---|
| Number of unique tickets that users interacted with in the bot | Number of unique tickets for which an employee performed one of the interactions (including submitting a ticket, closing a ticket, adding a comment, clicking on a ticket weblink, clicking on “Got it, Thanks”, nudging agent). Note that this metric does not count tickets related to Approvals processed in the bot in order to provide separation from the metric Tickets with approvals processed. | Tab Acceleration: - top metric - time trend | SELECT COUNT(DISTINCT ticket_id)AS "Issue Count" FROM bpi_interaction WHERE dim_interaction_category IN ('Click On Got It Thanks','Live Chat Clicked','Ticket Closed In Bot','Ticket Submitted In Bot','Comment Added','Nudge Agent','Ticket Weblink Clicked','Check Status','Other Interaction') | SELECT COUNT(DISTINCT ticket_id)AS "Issue Count"FROM {{date}}_daily_detailed_report WHERE skill_category='Interaction’ and skill_detail in ('Click On Got It Thanks','Live Chat Clicked','Ticket Closed In Bot','Ticket Submitted In Bot','Comment Added','Nudge Agent','Ticket Weblink Clicked','Check Status','Other Interaction') |
| Number of tickets interactions | Number of ticket interaction performed through the bot by employees (i.e. submit a ticket, close a ticket, add a comment, click on a weblink, click on “Got it, Thanks”, click to get directed to live chat, check status, nudge agent). | Tab Acceleration: - breakdowns by interaction category, platform, department, language, location, domain | SELECT COUNT(proc_link_root_uuid) AS "Interactions Count" FROM bpi_interaction WHERE dim_interaction_category IN ('Click On Got It Thanks','Live Chat Clicked','Ticket Closed In Bot','Ticket Submitted In Bot','Comment Added','Nudge Agent','Ticket Weblink Clicked','Check Status','Other Interaction') | SELECT COUNT(*)AS "Interactions Count"FROM {{date}}_daily_detailed_report WHERE skill_category='Interaction’ and skill_detail in ('Click On Got It Thanks','Live Chat Clicked','Ticket Closed In Bot','Ticket Submitted In Bot','Comment Added','Nudge Agent','Ticket Weblink Clicked','Check Status','Other Interaction') |
| Number of employees who interacted with ticket through the bot | Employees who accessed tickets through the bot and performed ticket operations (i.e. submit a ticket, close a ticket, add a comment, click on a weblink, click on “Got it, Thanks”, click to get directed to live chat, check status). | Tab Acceleration: - top metric | SELECT COUNT(DISTINCT user_record_id) AS "Distinct User Count" FROM bpi_interaction WHERE dim_interaction_category IN ('Click On Got It Thanks','Live Chat Clicked','Ticket Closed In Bot','Ticket Submitted In Bot','Comment Added','Nudge Agent','Ticket Weblink Clicked','Check Status','Other Interaction') | SELECT COUNT(DISTINCT user_id)AS "Issue Count"FROM {{date}}_daily_detailed_report WHERE skill_category='Interaction’ and skill_detail in ('Click On Got It Thanks','Live Chat Clicked','Ticket Closed In Bot','Ticket Submitted In Bot','Comment Added','Nudge Agent','Ticket Weblink Clicked','Check Status','Other Interaction') |
| Total tickets with approvals | Number of tickets with Approval requests that were polled by bot. | Tab Acceleration: - top metric - top side panel - breakdowns by type, department, language, location, domain | SELECT COUNT(DISTINCT approval_parent_id) AS "Distinct Approval Parent ID Count" FROM clt_issue_approvals | Not available |
| Median time to approval | The median time between when an approval was requested and when it was approved or denied in the bot, in minutes. | Tab Acceleration: - top metric | SELECT median(num_mins_to_resolution) AS "Median Time to Approval" FROM clt_issue_approvals WHERE ((is_resolved_by_bot = 'true')AND (num_mins_to_resolution IS NOT NULL)AND (is_first_approval_resolution = 'true')) | select median(minutes-to_resolution) from {{date}}_daily_detailed_report where skill_detail=’Tickets approved' |
| Tickets with approvals processed | Number of tickets with Approval requests that are approved or rejected by employees using the bot | Tab Acceleration: - top metric - time trend - breakdowns by type, department, language, location, domain | SELECT COUNT(DISTINCT CASE WHEN ((is_resolved_by_bot = 'true'))THEN approval_parent_id END) AS "Approvals Processed" FROM clt_issue_approvals | select count(distinct ticket_id) from {{date}}_daily_detailed_report where skill_detail=’Tickets approved' |
| % approvers engaged | Ratio of the number of unique users who approved or rejected an approval to the number of unique approvers requested for an approval polled by bot | Tab Acceleration: - time trend | SELECT to_char(date_trunc("week", Date(dt)), 'yyyy-MM-dd')AS "Week", COUNT(DISTINCT CASE WHEN ((is_resolved_by_bot = 'true'))THEN approval_parent_id END) AS "Approvals Processed", COUNT(DISTINCT CASE WHEN ((is_resolved_by_bot = 'true'))THEN user_record_id END) AS "Users with Resolved Approvals", COUNT(DISTINCT user_record_id) AS "Total Users" FROM clt_issue_approvals | Not available |
| Tickets triaged | Number of tickets triaged by Moveworks bot | Tab Acceleration: - top metric - side panel top metric - side panel time trend | [SELECT COUNT(DISTINCT ticket_id)AS "Distinct Tickets Count" FROM bpi_triage_record WHERE ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope') AND (dim_reason_for_not_triage IN ('NO_REASON', 'NO_CHANGE')) AND (dim_predicted_field_type != 'Field not Mapped') )] | [SELECT COUNT(DISTINCT ticket_id)AS "Distinct Tickets Count"FROM {{date}}_daily_triage_record WHERE ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_reason_for_not_triage IN ('NO_REASON', 'NO_CHANGE'))AND (dim_predicted_field_type != 'Field not Mapped') )] |
| Tickets triaged correctly | The number of tickets that were Transferred, Routed, or Categorized, without any later manual correction. - Routed: Tickets routed to the appropriate support group. This generally refers to tickets that are updated on a routing field, such as Assignment Group and Components - Transferred: Call tickets transferred to another table. This generally refers to tickets that are updated on a call type field, usually getting classified as either Incident or Request - Categorized: Tickets labeled. This generally refers to tickets that are updated on non-routing descriptive fields, such as Category, Subcategory, and Configuration Item | Tab Acceleration: - top metric - side panel top metric - side panel time trend | SELECT COUNT(DISTINCT ticket_id) AS "Distinct Tickets Count" FROM bpi_triage_record WHERE ((dim_predicted_field_type = 'Transferred')OR ((dim_predicted_field_type IN ('No Change', 'Routed', 'Categorized'))AND (dim_triage_result_type IN ('Correct', 'Correct (no change)')))) | SELECT COUNT(DISTINCT ticket_id)AS "Distinct Tickets Count"FROM {{date}}_daily_triage_record WHERE ((dim_predicted_field_type = 'Transferred')OR ((dim_predicted_field_typeIN ('No Change', 'Routed', 'Categorized'))AND (dim_triage_result_typeIN ('Correct', 'Correct (no change)')))) |
| Triage precision | Ratio of the number of tickets triaged correctly to the number of tickets triaged and closed | Tab Acceleration: - side panel top metric - side panel time trend | SELECT COUNT(DISTINCT CASE WHEN ((dim_predicted_field_type = 'Transferred')OR ((dim_predicted_field_type IN ('No Change', 'Routed', 'Categorized'))AND (dim_triage_result_type IN ('Correct', 'Correct (no change)'))))THEN ticket_id END)AS "Tickets Triaged Correctly" , COUNT(DISTINCT CASE WHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_reason_for_not_triage IN ('NO_REASON', 'NO_CHANGE'))AND (dim_predicted_field_type != 'Field not Mapped')AND (dim_triage_result_type IN ('Correct', 'Incorrect', 'Correct (no change)', 'Incorrect (no change)')))THEN ticket_id END)AS "Tickets Triaged and Closed" FROM bpi_triage_record | SELECTCOUNT(DISTINCT CASEWHEN ((dim_predicted_field_type = 'Transferred')OR ((dim_predicted_field_typeIN ('No Change', 'Routed', 'Categorized'))AND (dim_triage_result_typeIN ('Correct', 'Correct (no change)'))))THEN ticket_idEND)AS "Tickets Triaged Correctly" ,COUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_reason_for_not_triageIN ('NO_REASON', 'NO_CHANGE'))AND (dim_predicted_field_type != 'Field not Mapped')AND (dim_triage_result_typeIN ('Correct', 'Incorrect', 'Correct (no change)', 'Incorrect (no change)')))THEN ticket_idEND)AS "Tickets Triaged and Closed"FROM {{date}}_daily_triage_record |
| Triage coverage | Ratio of the number of tickets triaged and closed to the number of tickets eligible for triage | Tab Acceleration: - side panel top metric - side panel time trend | SELECT COUNT(DISTINCT CASE WHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_reason_for_not_triageIN ('NO_REASON', 'NO_CHANGE'))AND (dim_predicted_field_type != 'Field not Mapped'))THEN ticket_id END) AS "Tickets Triaged",COUNT(DISTINCT ticket_id)AS "Tickets Eligible for Triage"FROM bpi_triage_recordWHERE ((dim_predicted_field_type != 'Not Relevant') AND (dim_predicted_field_type != 'Out of Scope')) | SELECTCOUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_reason_for_not_triageIN ('NO_REASON', 'NO_CHANGE'))AND (dim_predicted_field_type != 'Field not Mapped'))THEN ticket_id END) AS "Tickets Triaged",COUNT(DISTINCT ticket_id)AS "Tickets Eligible for Triage"FROM {{date}}_daily_triage_record WHERE ((dim_predicted_field_type != 'Not Relevant') AND (dim_predicted_field_type != 'Out of Scope')) |
| Triage precision and precision reassigned | Precision = Tickets Triaged Correctly / Tickets Triaged and Closed Precision Reassigned = Tickets Triaged Correctly Reassigned / Tickets Triaged and Closed Reassigned | Tab Acceleration: - side panel breakdowns - side panel time trend | SELECT COUNT(DISTINCT CASE WHEN ((dim_predicted_field_type = 'Transferred')OR ((dim_predicted_field_type IN ('No Change', 'Routed', 'Categorized'))AND (dim_triage_result_typeIN ('Correct', 'Correct (no change)'))))THEN ticket_id END)AS "Tickets Triaged Correctly",COUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_reason_for_not_triage IN ('NO_REASON', 'NO_CHANGE'))AND (dim_predicted_field_type != 'Field not Mapped')AND (dim_triage_result_typeIN ('Correct', 'Incorrect', 'Correct (no change)', 'Incorrect (no change)')))THEN ticket_id END)AS "Tickets Triaged and Closed",COUNT(DISTINCT CASEWHEN ((dim_predicted_field_type = 'Transferred')OR ((dim_predicted_field_typeIN ('No Change', 'Routed', 'Categorized'))AND (dim_triage_result_type = 'Correct')))THEN ticket_id END)AS "Tickets Triaged Correctly Reassigned" ,COUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_reason_for_not_triage IN ('NO_REASON'))AND (dim_predicted_field_type != 'Field not Mapped')AND (dim_triage_result_type IN ('Correct', 'Incorrect')))THEN ticket_idEND)AS "Tickets Triaged and Closed Reassigned"FROM bpi_triage_record | SELECTCOUNT(DISTINCT CASEWHEN ((dim_predicted_field_type = 'Transferred')OR ((dim_predicted_field_typeIN ('No Change', 'Routed', 'Categorized'))AND (dim_triage_result_typeIN ('Correct', 'Correct (no change)'))))THEN ticket_id END)AS "Tickets Triaged Correctly",COUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_reason_for_not_triage IN ('NO_REASON', 'NO_CHANGE'))AND (dim_predicted_field_type != 'Field not Mapped')AND (dim_triage_result_typeIN ('Correct', 'Incorrect', 'Correct (no change)', 'Incorrect (no change)')))THEN ticket_id END)AS "Tickets Triaged and Closed",COUNT(DISTINCT CASEWHEN ((dim_predicted_field_type = 'Transferred')OR ((dim_predicted_field_typeIN ('No Change', 'Routed', 'Categorized'))AND (dim_triage_result_type = 'Correct')))THEN ticket_id END)AS "Tickets Triaged Correctly Reassigned" ,COUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_reason_for_not_triage IN ('NO_REASON'))AND (dim_predicted_field_type != 'Field not Mapped')AND (dim_triage_result_type IN ('Correct', 'Incorrect')))THEN ticket_idEND)AS "Tickets Triaged and Closed Reassigned"FROM {{date}}_daily_triage_record |
| Triage coverage and coverage reassigned | Coverage = Tickets Triaged / Tickets Eligible for Triage Coverage Reassigned = Tickets Triaged Reassigned / Tickets Eligible for Triage Reassigned | Tab Acceleration: - side panel breakdowns - side panel time trend | SELECTCOUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_reason_for_not_triageIN ('NO_REASON', 'NO_CHANGE'))AND (dim_predicted_field_type != 'Field not Mapped'))THEN ticket_id END)AS "Tickets Triaged",COUNT(DISTINCT ticket_id)AS "Tickets Eligible for Triage",COUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_start_value != dim_final_value)AND (dim_reason_for_not_triage = 'NO_REASON')AND (dim_predicted_field_type != 'Field not Mapped'))THEN ticket_idEND) AS "Tickets Triaged Reassigned",COUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_start_value != dim_final_value))THEN ticket_idEND)AS "Tickets Eligible for Triage Reassigned"FROM bpi_triage_recordWHERE ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')) | SELECTCOUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_reason_for_not_triageIN ('NO_REASON', 'NO_CHANGE'))AND (dim_predicted_field_type != 'Field not Mapped'))THEN ticket_id END)AS "Tickets Triaged",COUNT(DISTINCT ticket_id)AS "Tickets Eligible for Triage",COUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_start_value != dim_final_value)AND (dim_reason_for_not_triage = 'NO_REASON')AND (dim_predicted_field_type != 'Field not Mapped'))THEN ticket_idEND) AS "Tickets Triaged Reassigned",COUNT(DISTINCT CASEWHEN ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')AND (dim_start_value != dim_final_value))THEN ticket_idEND)AS "Tickets Eligible for Triage Reassigned"FROM {{date}}_daily_triage_record WHERE ((dim_predicted_field_type != 'Not Relevant')AND (dim_predicted_field_type != 'Out of Scope')) |
Adoption metrics
| Metric name | Metric description | Where to find the metric | Example SQL | SQL for SFTP data |
|---|---|---|---|---|
| Bot-Active Employees List of users in side panel under “View Users” button | Employees who have actively used the bot (who typed in the bot, clicked on a button in the bot, or clicked on a link generated by the bot) in the selected time range | Tab Adoption: - top metric - top side panel - time trend - breakdowns by department, language, location | SELECT COUNT(DISTINCT user_record_id)AS "Bot Active Employees"FROM user_summary_daily WHERE ((is_bot_active_user = 'true')AND (is_tester = 'false')) | select count(distinct user_id) from {{date}}_daily_active_users |
| Employee Who Submitted Issues or Tickets Equivalent to Employees with Issues | Number of employees who created tickets in ITSM system or described an issue in the bot | Tab Adoption: - top metric - time trend - breakdowns by department, language, location | SELECT COUNT(DISTINCT user_record_id)AS "Employees with Issues"FROM user_summary_dailyWHERE ((is_bot_first_eligible = 'true')AND (is_tester = 'false') ) | Not available |
| Number of conversations | The total number of conversations that an employee engaged with the bot; one conversation is defined by an employee-to-bot conversation interrupted by no more than 30 minutes | Tab Adoption: - top metric - time trend - breakdowns by department, language, location | SELECT SUM(num_tot_user_sessions)AS "Number of Conversations"FROM user_summary_daily WHERE ((is_tester = 'false')) | Not available |
| Number of First Time Users | Users who became active in bot for the first time in the selected time range (typed in the bot, clicked on a button in the bot, or clicked on a link generated by the bot) | Tab Adoption: - time trend - list of users in side panel under “View Users” button | SELECT COUNT(DISTINCT user_record_id)AS "Bot Active Employees"FROM user_summary_daily WHERE ((is_first_time_user = 'true')AND (is_tester = 'false')) | Not available |
Row-level data columns
| Column name | Column description |
|---|---|
| ARTICLE ADDED DATE | Date of addition/update of knowledge snippet used by Answers skill |
| ARTICLE ID | ID of the knowledge article that has the snippet used by Answers skill. For example, for ServiceNow Knowledge Base article ID looks like KB0012345. |
| ARTICLE TITLE | Title of the knowledge snippet used by Answers skill |
| CHANNEL ID | ID of Slack channel (if the feature is enabled for the org) where issue was described by user |
| CHANNEL NAME | Name of Slack channel (if the feature is enabled for the org) where issue was described by user |
| COUNTRY CODE | As indicated for the user in user roster |
| DEPARTMENT | As indicated for the user in user roster |
| DOC DOMAIN | Domain of the knowledge snippet used by Answers skill, as configured for the knowledge. For example, if knowledge is curated by HR owner, domain is HR. |
| DOC SOURCE | Source of the knowledge snippet used by Answers skill which can be one of “Internal Knowledge Base”, “Moveworks FAQ Sheet”, “Moveworks External Knowledge” or “Internal File” |
| DOMAIN | Domain of Enterprise services representing nature of the issue. For example, inquiries on laptop replacement as in IT domain, whereas questions about PTO balance are in HR domain. |
| ENTITY | Entity, or topic, is ML-predicted label succinctly describing nature of the issue. For example, for an issue “I need a license for Zoom”, topic is “zoom”. For more details, see Entity Prediction. |
| IS AGENT HANDOFF? | True if for the given issue, user clicks on a button or link to be redirected to Live Chat or other Agent Handoff |
| IS RESOLVED BY BOT? | True if the issue was resolved by one of the bot’s Resolution skills |
| IS TICKET ESCALATED? | True if for the given issue, user files a ticket from bot |
| ISSUE ORIGIN | Describes origin of issues that were described in DM (”Messages to Bot”), filed as a ticket (”Tickets intercepted by Bot”), described in shared IT channel (”Messages to Bot - Channel Utterances”), or proactively raised by bot via notification (”Proactive Messages by Bot”) |
| ISSUE RATING | Indicates whether user gave positive or negative rating to the bot’s solution, by clicking on buttons “Helpful” or “Not Helpful” |
| ISSUE RESULT | See above in Resolution metrics |
| LANGUAGE | Bot language setting |
| LOCATION | As indicated for the user in user roster |
| RESOLVED BY SKILL(S) | List of Resolution skills that bot used to resolve the issue |
| SKILL NAME | One of the Bot resolution skills: Answers, Form Filling, Access Software, Access Account and Access Groups |
| SKILL RESULT | See above in Skill metrics |
| TICKET ID | ID of ITSM ticket relevant to the issue. For example, if the issue was originally filed as a ticket and this ticket was polled by bot, ticket id reflects the ID of the original ticket. Or, if user describes an issue in bot and proceeds to file a ticket for this issue, ticket id reflects the ID of the ticket filed from bot. |
| USER RECORD ID | Numeric anonymized user ID, generated by Moveworks system |
| USER TEXT | Text of issue description, either as typed in bot, or as entered as ticket description in ticketing system. |
Updated 4 months ago