BPI+ Metrics Dictionary

Resolution metrics

Metric name
.....................................
Metric description
......................................................
Where to find the metric
.................................................
Example SQL
Number of issues resolved by botNumber of issues resolved by the bot, including helpful or not rated Answers, submitted Forms and Access issues resolved.
This metric shows how many issues were resolved by one of the bot resolution skills: Answers, Form Filling, Access Software, Access Account and Access Groups. An issues can be described in DM, filed as a ticket, described in shared IT channel, or proactively raised by bot via notification.
Tab Resolution:
- top metric
- time trend
- breakdown by contact type
- breakdowns by skill, topic, route, ticket avoided, department, language, location, domain

Tab Resolution → side panels in Skill breakdown:
- top metric
SELECT COUNT(global_issue_uuid)
AS "Distinct Issues Resolved by Bot"
FROM bpi_issue
WHERE ((is_resolved_by_bot = 'true'))
Mean time to resolution, minutesThe average time between when an issue was created and when it was resolved in the bot. It is calculated as time interval in minutes, from the time when an issues was created, to the time when bot provides resolution of the issue.
Outlier values over 1440 minutes (24 hours) are excluded to decrease metric volatility.
Tab Resolution:
- top metric
- time trend
- breakdowns by department, language, location, domain
SELECT AVG(num_mins_to_resolution)
AS "Mean Time to Resolution"
FROM bpi_issue
WHERE ((num_mins_to_resolution IS NOT NULL)
AND (num_mins_to_resolution < 1440)
AND (dim_issue_origin != 'BOT_NOTIFICATION'))
Number of employees with bot resolutionNumber of employees who have at least one issue resolved by the bot in the selected time period.Tab Resolution:
- top metric
- top side panel
- time trend
- breakdowns by department, language, location, domain
SELECT COUNT(DISTINCT user_record_id)
AS "Employees with Resolution"
FROM bpi_issue
WHERE ((is_resolved_by_bot = 'true'))
Number of issues resolved or ticketedNumber of issues that are either resolved by bot or are ticketed to be further handled by agents. Issues can be described in DM, filed as a ticket, described in shared IT channel, or proactively raised by bot via notification.Tab Resolution:
- top metric
- breakdown by contact type
SELECT COUNT(global_issue_uuid)
AS "Total Issues Submitted" FROM bpi_issue WHERE ((is_qualified_for_resolution_denominator = 'true')
AND ((is_resolved_by_bot = 'true')
OR (ticket_id IS NOT NULL)))
Number of unique employees with issues resolved or ticketedNumber of employees with issues that are either resolved by bot or are ticketed to be further handled by agents. Issues can be described in DM, filed as a ticket, described in shared IT channel, or proactively raised by bot via notification.Tab Resolution:
- top metric
SELECT COUNT(DISTINCT user_record_id)
AS "Employees with Issues Resolved or Ticketed"
FROM bpi_issue
WHERE ((is_qualified_for_resolution_denominator = 'true')
AND ((is_resolved_by_bot = 'true')
OR (ticket_id IS NOT NULL)))
Bot EffectivenessBot effectiveness shows how effective the bot is in resolving employee issues. Bot effectiveness ratio is equal to percentage of issues resolved by bot, out of all issues that are either resolved by bot or are ticketed to be further handled by agents.
More specifically, numerator is defined as number of issues raised in bot and resolved by bot and denominator is number of issues described in bot, with either a resolution or a filed ticket.
Tab Resolution:
- subtitle in the bar chart showing breakdown by contact type
SELECT 1.0* COUNT(CASE
WHEN ((is_resolved_by_bot = 'True'))
THEN global_issue_uuid
END) / COUNT(global_issue_uuid)
AS "Bot Effectiveness"
FROM bpi_issue
WHERE ((is_qualified_for_resolution_denominator = 'true')
AND ((is_resolved_by_bot = 'true')
OR (ticket_id IS NOT NULL))
AND (is_created_by_bot = 'true'))
Number of issues qualified for resolution by botNumber of all issues in scope: issues described in bot and intercepted tickets qualified for resolution by bot.
- Issues that are described in bot are considered qualified for resolution if any of resolution skills attempts to resolve the issue (as opposed to be non-applicable).
- Intercepted tickets are filtered according to the preset filter to be qualified for resolution by bot.
Tab Resolution:
- top side panel
- breakdowns by topic, route, department, language, location, domain
SELECT COUNT(global_issue_uuid)
AS "Total Issues"
FROM bpi_issue
Number of issues with solution offeredNumber of issues with solution offered by one of the bot resolution skills. Once bot offers a solution, user may or may not engage in conversation with the bot on the same issue. Further, if bot offers solutions by several skills, user may engage with only some of the skills offered. Finally, bot may resolve the issue with one or more skills.Tab Resolution → side panel for Answers:
- top metric
- breakdowns by knowledge served, topics identified
- time trend

Tab Resolution → side panel for Form Filling:
- breakdowns by forms served, topics identified
- time trend

Tab Resolution → side panel for Access Software:
- breakdown by software provisioned
- time trend

Tab Resolution → side panel for Access Groups:
- breakdown by subskill
- time trend

Tab Resolution → side panel for Access Account:
- breakdown by subskill
- time trend

Tab Resolution → side panel for Messages to Bot - Channel Utterances:
- breakdown by skill, topic identified
- time trend
SELECT dim_skill_name
AS "Skill Name",
COUNT(CASE
WHEN ((dim_skill_result = 'Resolved By Skill'))
THEN global_issue_uuid
END)
AS "Resolved By Skill",
COUNT(CASE
WHEN ((dim_skill_result = 'Engaged On Skill But Not Resolved'))
THEN global_issue_uuid
END)
AS "Engaged On Skill But Not Resolved",
COUNT(CASE
WHEN ((dim_skill_result = 'Engaged In Bot But Not On Skill'))
THEN global_issue_uuid
END)
AS "Engaged In Bot But Not On Skill",
COUNT(CASE
WHEN ((dim_skill_result = 'Served But Not Engaged In Bot'))
THEN global_issue_uuid
END)
AS "Served But Not Engaged In Bot"
FROM bpi_issue_skill
WHERE ((dim_skill_result IS NOT NULL)
AND (dim_skill_result != 'Not Served'))
Number of utterances in shared IT channelNumber of Channel utterances shows how many of issues described in shared IT channel bot attempted to resolve with one of resolution skillsTab Resolution → side panel for Messages to Bot - Channel Utterances in Route breakdown:
- top metric
SELECT COUNT(global_issue_uuid)
AS "Num Channel Issues"
FROM bpi_issue
WHERE ((dim_issue_origin = 'CHANNEL_UTTERANCE')
AND (has_resolution_skill_pass_pretrigger = 'true'))
Number of Bot reach-outs in shared IT channelNumber of Bot reach-outs shows for how many issues described in shared IT channel bot has offered a solution from one of resolution skillsTab Resolution → side panel for Messages to Bot - Channel Utterances in Route breakdown:
- top metric
SELECT COUNT(global_issue_uuid)
AS "Num Channel Reachouts"
FROM bpi_issue
WHERE ((dim_issue_origin = 'CHANNEL_UTTERANCE')
AND (num_resolution_skill_offer_won >= 1))
Number of employees with bot resolution by specific resolution skillNumber of employees who had any of their issues resolved by one of resolution skills: Answers, Form Filling, Access Software, Access Account or Access GroupsTab Resolution → side panels for Skill breakdown:
- top metric
SELECT COUNT(DISTINCT user_record_id)
AS "Employees with Resolution"
FROM bpi_issue_skill
WHERE ((is_resolved_by_skill = 'true'))
Number of employees with bot resolution by Access Account skillNumber of employees who had any of their issues resolved by Access AccountTab Resolution → side panel for Access Account:
- top metric
- breakdowns by department, location, language, domain
SELECT COUNT(DISTINCT user_record_id)
AS "Employees with Resolution"
FROM bpi_issue_skill
WHERE ((is_resolved_by_skill = 'true')
AND (dim_skill_name = 'Access Account'))
Number of Answers snippets servedNumber of Answers snippets served shows total number of times any knowledge snippet was offered to employee as a solution of their issuesTab Resolution → side panel for Answers:
- top metric
SELECT COUNT(offer_item_id)
AS "Number of Served Offer Items"
FROM bpi_issue_skill_item
WHERE ((dim_skill_result IS NOT NULL)
AND (dim_skill_result != 'Not Served')
AND (dim_skill_name = 'Answers'))

Acceleration metrics

Metric name
.....................................
Metric description
......................................................
Where to find the metric
.................................................
Example SQL
Number of unique tickets that users interacted with in the botNumber of unique tickets for which an employee performed one of the interactions (including submitting a ticket, closing a ticket, adding a comment, clicking on a ticket weblink, clicking on “Got it, Thanks”, nudging agent).Tab Acceleration:
- top metric
- time trend
- breakdowns by interaction category, platform, department, language, location, domain
SELECT COUNT(DISTINCT ticket_id)
AS "Issue Count"
FROM bpi_interaction
WHERE ((dim_interaction_category
IN ('Click On Got It Thanks',
'Live Chat Clicked',
'Ticket Closed In Bot',
'Ticket Submitted In Bot',
'Comment Added',
'Nudge Agent',
'Ticket Weblink Clicked',
'Check Status',
'Other Interaction'))
AND ((dim_interaction_category !=
'Check Status')
OR (ticket_id IS NOT NULL)))
Number of tickets interactionsNumber of ticket interaction performed through the bot by employees (i.e. submit a ticket, close a ticket, add a comment, click on a weblink, click on “Got it, Thanks”, click to get directed to live chat, check status, nudge agent).Tab Acceleration:
- breakdowns by interaction category, platform, department, language, location, domain
SELECT COUNT(proc_link_root_uuid)
AS "Interactions Count"
FROM bpi_interaction
WHERE ((dim_interaction_category
IN ('Click On Got It Thanks',
'Live Chat Clicked',
'Ticket Closed In Bot',
'Ticket Submitted In Bot',
'Comment Added',
'Nudge Agent',
'Ticket Weblink Clicked',
'Check Status',
'Other Interaction'))
AND ((dim_interaction_category
!= 'Check Status')
OR (ticket_id IS NOT NULL)))
Number of employees who interacted with ticket through the botEmployees who accessed tickets through the bot and performed ticket operations (i.e. submit a ticket, close a ticket, add a comment, click on a weblink, click on “Got it, Thanks”, click to get directed to live chat, check status).Tab Acceleration:
- top metric
SELECT COUNT(DISTINCT user_record_id)
AS "Distinct User Count"
FROM bpi_interaction
WHERE ((dim_interaction_category
IN ('Click On Got It Thanks',
'Live Chat Clicked',
'Ticket Closed In Bot',
'Ticket Submitted In Bot',
'Comment Added',
'Nudge Agent',
'Ticket Weblink Clicked',
'Check Status',
'Other Interaction'))
AND ((dim_interaction_category
!= 'Check Status')
OR (ticket_id IS NOT NULL)))
Total tickets with approvalsNumber of tickets with Approval requests that were intercepted by bot.Tab Acceleration:
- top metric
- top side panel
- breakdowns by type, department, language, location, domain
SELECT COUNT(DISTINCT approval_parent_id)
AS "Distinct Approval Parent ID Count"
FROM clt_issue_approvals
Median time to approvalThe median time between when an approval was requested and when it was approved or denied in the bot, in minutes.Tab Acceleration:
- top metric
SELECT APPROX_PERCENTILE(num_mins_to_resolution, 0.5)
AS "Median Time to Approval"
FROM clt_issue_approvals
WHERE ((is_resolved_by_bot = 'true')
AND (num_mins_to_resolution IS NOT NULL)
AND (is_first_approval_resolution = 'true'))
Tickets with approvals processedNumber of tickets with Approval requests that are approved or rejected by employees using the boTab Acceleration:
- top metric
- time trend
- breakdowns by type, department, language, location, domain
SELECT COUNT(DISTINCT CASE
WHEN ((is_resolved_by_bot = 'true'))
THEN approval_parent_id
END)
AS "Approvals Processed"
FROM clt_issue_approvals
% approvers engagedRatio of the number of unique users who approved or rejected an approval to the number of unique approvers requested for an approval intercepted by botTab Acceleration:
- time trend
SELECT to_char(date_trunc("week", Date(dt)), 'yyyy-MM-dd')
AS "Week",
COUNT(DISTINCT CASE
WHEN ((is_resolved_by_bot = 'true'))
THEN approval_parent_id
END)
AS "Approvals Processed",
COUNT(DISTINCT CASE
WHEN ((is_resolved_by_bot = 'true'))
THEN user_record_id
END) AS "Users with Resolved Approvals",
COUNT(DISTINCT user_record_id)
AS "Total Users"
FROM clt_issue_approvals
Tickets triagedNumber of tickets triagedTab Acceleration:
- top metric
- side panel top metric
- side panel time trend
[SELECT COUNT(DISTINCT ticket_id)
AS "Distinct Tickets Count"
FROM bpi_triage_record
WHERE ((dim_predicted_field_type != 'Not Relevant')
AND (dim_predicted_field_type != 'Out of Scope')
AND (dim_reason_for_not_triage
IN ('NO_REASON', 'NO_CHANGE'))
AND (dim_predicted_field_type != 'Field not Mapped') )]
Tickets triaged correctlyThe number of tickets that were Transferred, Routed, or Categorized, without any later manual correctionTab Acceleration:
- top metric
- side panel top metric
- side panel time trend
SELECT COUNT(DISTINCT ticket_id)
AS "Distinct Tickets Count"
FROM bpi_triage_record
WHERE ((dim_predicted_field_type = 'Transferred')
OR ((dim_predicted_field_type
IN ('No Change', 'Routed', 'Categorized'))
AND (dim_triage_result_type
IN ('Correct', 'Correct (no change)'))))
Triage precisionRatio of the number of tickets triaged correctly to the number of tickets triaged and closedTab Acceleration:
- side panel top metric
- side panel time trend
SELECT
COUNT(DISTINCT CASE
WHEN ((dim_predicted_field_type = 'Transferred')
OR ((dim_predicted_field_type
IN ('No Change', 'Routed', 'Categorized'))
AND (dim_triage_result_type
IN ('Correct', 'Correct (no change)'))))
THEN ticket_id
END)
AS "Tickets Triaged Correctly" ,
COUNT(DISTINCT CASE
WHEN (
(dim_predicted_field_type != 'Not Relevant')
AND (dim_predicted_field_type != 'Out of Scope')
AND (dim_reason_for_not_triage
IN ('NO_REASON', 'NO_CHANGE'))
AND (dim_predicted_field_type != 'Field not Mapped')
AND (dim_triage_result_type
IN ('Correct', 'Incorrect', 'Correct (no change)', 'Incorrect (no change)'))
)
THEN ticket_id
END)
AS "Tickets Triaged and Closed"
FROM bpi_triage_record
Triage coverageRatio of the number of tickets triaged and closed to the number of tickets eligible for triageTab Acceleration:
- side panel top metric
- side panel time trend
SELECT
COUNT(DISTINCT CASE
WHEN ((dim_predicted_field_type != 'Not Relevant')
AND (dim_predicted_field_type != 'Out of Scope')
AND (dim_reason_for_not_triage
IN ('NO_REASON', 'NO_CHANGE'))
AND (dim_predicted_field_type != 'Field not Mapped'))
THEN ticket_id END) AS "Tickets Triaged",
COUNT(DISTINCT ticket_id)
AS "Tickets Eligible for Triage"
FROM bpi_triage_record
WHERE ((dim_predicted_field_type != 'Not Relevant') AND (dim_predicted_field_type != 'Out of Scope'))
Triage precision and precision reassignedPrecision = Tickets Triaged Correctly / Tickets Triaged and Closed

Precision Reassigned = Tickets Triaged Correctly Reassigned / Tickets Triaged and Closed Reassigned
Tab Acceleration:
- side panel breakdowns
- side panel time trend
SELECT
COUNT(DISTINCT CASE
WHEN ((dim_predicted_field_type = 'Transferred')
OR ((dim_predicted_field_type
IN ('No Change', 'Routed', 'Categorized'))
AND (dim_triage_result_type
IN ('Correct', 'Correct (no change)'))))
THEN ticket_id END)
AS "Tickets Triaged Correctly",

COUNT(DISTINCT CASE
WHEN ((dim_predicted_field_type != 'Not Relevant')
AND (dim_predicted_field_type != 'Out of Scope')
AND (dim_reason_for_not_triage IN ('NO_REASON', 'NO_CHANGE'))
AND (dim_predicted_field_type != 'Field not Mapped')
AND (dim_triage_result_type
IN ('Correct', 'Incorrect', 'Correct (no change)', 'Incorrect (no change)')))
THEN ticket_id END)
AS "Tickets Triaged and Closed",
COUNT(DISTINCT CASE
WHEN ((dim_predicted_field_type = 'Transferred')
OR ((dim_predicted_field_type
IN ('No Change', 'Routed', 'Categorized'))
AND (dim_triage_result_type = 'Correct')))
THEN ticket_id END)
AS "Tickets Triaged Correctly Reassigned" ,
COUNT(DISTINCT CASE
WHEN ((dim_predicted_field_type != 'Not Relevant')
AND (dim_predicted_field_type != 'Out of Scope')
AND (dim_reason_for_not_triage IN ('NO_REASON'))
AND (dim_predicted_field_type != 'Field not Mapped')
AND (dim_triage_result_type IN ('Correct', 'Incorrect')))
THEN ticket_id
END)
AS "Tickets Triaged and Closed Reassigned"

FROM bpi_triage_record
Triage coverage and coverage reassignedCoverage = Tickets Triaged / Tickets Eligible for Triage

Coverage Reassigned = Tickets Triaged Reassigned / Tickets Eligible for Triage Reassigned
Tab Acceleration:
- side panel breakdowns
- side panel time trend
SELECT
COUNT(DISTINCT CASE
WHEN ((dim_predicted_field_type != 'Not Relevant')
AND (dim_predicted_field_type != 'Out of Scope')
AND (dim_reason_for_not_triage
IN ('NO_REASON', 'NO_CHANGE'))
AND (dim_predicted_field_type != 'Field not Mapped'))
THEN ticket_id END)
AS "Tickets Triaged",
COUNT(DISTINCT ticket_id)
AS "Tickets Eligible for Triage",

COUNT(DISTINCT CASE
WHEN ((dim_predicted_field_type != 'Not Relevant')
AND (dim_predicted_field_type != 'Out of Scope')
AND (dim_start_value != dim_final_value)
AND (dim_reason_for_not_triage = 'NO_REASON')
AND (dim_predicted_field_type != 'Field not Mapped'))
THEN ticket_id
END) AS "Tickets Triaged Reassigned",

COUNT(DISTINCT CASE
WHEN ((dim_predicted_field_type != 'Not Relevant')
AND (dim_predicted_field_type != 'Out of Scope')
AND (dim_start_value != dim_final_value))
THEN ticket_id
END)
AS "Tickets Eligible for Triage Reassigned"

FROM bpi_triage_record
WHERE ((dim_predicted_field_type != 'Not Relevant')
AND (dim_predicted_field_type != 'Out of Scope'))

Adoption metrics

Metric name
.....................................
Metric description
......................................................
Where to find the metric
.................................................
Example SQL
Bot-Active EmployeesEmployees who have actively used the bot (who typed in the bot, clicked on a button in the bot, or clicked on a link generated by the bot) in the selected time rangeTab Adoption:
- top metric
- top side panel
- time trend
- breakdowns by department, language, location
SELECT COUNT(DISTINCT user_record_id)
AS "Bot Active Employees"
FROM user_summary_daily
WHERE ((is_bot_active_user = 'true')
AND (is_tester = 'false'))
Employees with IssuesNumber of employees who created tickets in ITSM system or described an issue in the botTab Adoption:
- top metric
- time trend
- breakdowns by department, language, location
SELECT COUNT(DISTINCT user_record_id)
AS "Employees with Issues"
FROM user_summary_daily
WHERE ((is_bot_first_eligible = 'true')
AND (is_tester = 'false') )
Number of conversationsThe total number of conversations that an employee engaged with the bot; one conversation is defined by an employee-to-bot conversation interrupted by no more than 30 minutesTab Adoption:
- top metric
- time trend
- breakdowns by department, language, location
SELECT SUM(num_tot_user_sessions)
AS "Number of Conversations"
FROM user_summary_daily WHERE ((is_tester = 'false'))