📘

This is applicable for Moveworks AI Assistant and Moveworks Classic

Triage is a backend service that works independent of the frontend experience

Overview

For issues that cannot be resolved instantly, Moveworks Triage can categorize and route these tickets to ensure that they reach the service desk team best equipped to resolve the issue. The Moveworks Triage reviews new incoming tickets and analyzes the ticket’s fields using natural language understanding (NLU). The bot predicts what values the ticket’s fields should have and then depending on the bot’s confidence level, it will automatically update the various fields (such as Assignment Group, Category, Subcategory, etc.) in the ticket.

By doing this, Triage can improve the rate at which tickets are resolved by making the categorization of tickets autonomous and reduce amount of tickets that get categorized incorrectly

How does Triage work?

Intercept

Triage intercepts and analyzes new tickets coming into your ITSM system. The bot is configured to determine which initial assignment group it needs to look at for new incoming tickets. Once a newly created ticket is found, Triage sees if it can predict the ticket fields.

📘

Note: Triage only intercepts tickets coming from the self service portal, email, or the bot. Phone tickets are not eligible for Triage.

Predict

Triage will use NLU to analyze the ticket description and fields to determine if it can make a prediction it is confident in. The bot’s confidence must equal or exceed the set confidence threshold in order for it to make changes to the ticket. For example, if you’ve set the confidence threshold to 67%, the bot will only make updates to the ticket if it is at least 67% confident in its prediction.

If the bot is not confident enough in its predictions it will leave the ticket as is and let the service desk agent update the ticket fields.

Update

If the bot is confident in its predictions, then it will automatically update each of the ticket fields it has been trained to predict. For example, if the bot has been trained on assignment group and category models, meaning it has learned from two different datasets, then the model will be able to update both of those fields.

The most common fields Triage can predict are:

• assignment group
• category
• subcategory
• cmdb_ci
• component (Jira)
• hr service

How does Triage make predictions?

In order for Triage to make predictions, Moveworks uses model training on labeled data, specifically tickets that have fields populated and in a CLOSED/RESOLVED state. With this data, Moveworks uses a supervised learning model to train the Triage model.

🚧

Note: Because Triage is reliant on the data its model is trained on, it’s important to provide the best data available to train the bot on.

Model Training

Model training is the process of feeding a machine learning model with data in order for it to learn to identify ticket fields confidently.

Supervised Learning

The supervised learning model is trained on both input and output data. The model learns the relationship between input and output data and uses these known relationships to predict future, unknown outputs. For example, if we train a model, and it sees that often when the employee files a ticket with the short description and description containing topics relating to VPN, the tickets tends to close in the VPN Helpdesk assignment group. So when we deploy this model, when a new ticket (input) comes in related to VPN, the model should be able to follow the input-output relationship it learned from and predict VPN Helpdesk (output).

Triage Performance

After the Triage model has been trained, it’s important to establish a framework for monitoring performance. The evaluation of Triage's performance involves the assessment of two primary metrics:

Coverage

In essence, coverage refers to the proportion of tickets on which Triage has made predictions compared to the total number of tickets received.

For instance, let us consider the example of Company.com, which experienced the influx of 1000 tickets within the previous month. Out of these 1000 tickets, Triage made predictions for 300 of them. This implies that Triage covered 300 out of 1000 tickets, indicating a coverage rate of 30% for Company.com's tickets.

Mathematically, coverage (C) can be calculated as:

C = Number of tickets covered / Total number of tickets = 300 / 1000 = 0.30

It is important to note that at this point, we have solely focused on Triage's prediction coverage and have not yet evaluated its accuracy.

Precision

Precision refers to the percentage of correct predictions made by Triage among all the tickets on which predictions were made.

How do we define "correct" or "incorrect" predictions?

To establish the criteria for determining whether a prediction is correct or incorrect, we consider the following:

• A prediction is considered correct if Triage predicts that the assignment group for a ticket, such as INC12345, is "VPN Helpdesk," and the ticket is indeed closed or resolved within the "VPN Helpdesk" group. In this case, our prediction is accurate.
  • Prediction: Ticket closed/resolved in "VPN Helpdesk"
• On the other hand, a prediction is considered incorrect if Triage predicts that the assignment group for a ticket, such as INC12345, is "VPN Helpdesk," but the ticket is actually closed or resolved within the "SAP Helpdesk" group. In this scenario, our prediction is deemed inaccurate.
  • Prediction: Ticket closed/resolved ≠ "VPN Helpdesk"

For example, let's consider Company.com, which received a total of 1000 tickets in the past month. Out of these 1000 tickets, Triage made predictions for 300 of them. Thus, we can infer that Triage "covered" 300 out of 1000 tickets, representing a coverage rate of 30% for [Company.com]'s tickets.

Furthermore, out of the 300 tickets covered, Triage's predictions were correct 150 times, indicating a precision rate of 50%.

P = 150/300 = 0.50

How to enable Triage?

While Moveworks Triage offers powerful capabilities, it is not a one-size-fits-all solution. To determine whether your organization’s system is eligible for Triage, please reach out to your Customer Success team, who will guide you through the Triage Eligibility questionnaire. This questionnaire collects necessary information to assess whether this skill is suitable for your specific environment and needs.

Once the relevant information is collected, the Triage Data Science team will leverage the your ticketing data to train a set of models tailored to your specific requirements. This training process ensures that Triage is optimized to provide accurate predictions and effective ticket categorization based on the unique characteristics and patterns within your organization’s ticketing system.

By following this careful evaluation and training process, Moveworks ensures that Triage is deployed appropriately and delivers the desired benefits.

Configuration options

Confidence Threshold

Adjusting the Confidence Threshold allows for customizing the level of certainty required for the bot to predict and route tickets. However, it's important to consider the tradeoff between coverage and precision when modifying this threshold.

Lowering the Confidence Threshold increases coverage by enabling the bot to make predictions and route a larger number of tickets. However, this can potentially decrease the accuracy of the bot's predictions, leading to more incorrect assignments.

On the other hand, raising the Confidence Threshold improves the precision of the bot's predictions. By setting a higher threshold, the bot only predicts when it is more confident, resulting in more accurate routing and reduced misclassification. However, this may reduce coverage, as fewer tickets meet the threshold for automatic routing.

Blocklist and Allowlist

Moveworks provides the flexibility to implement field blocklists and allowlists, enabling selective prediction capabilities for Triage.

Blocklist

The blocklist feature allows the specification of certain fields for which Triage will abstain from making predictions. By including fields in the blocklist, organizations can define which fields should not be subject to prediction by the bot.

Allowlist

Conversely, the allowlist feature allows organizations to identify a specific set of fields for which Triage is exclusively permitted to make predictions. Any field not included in the allowlist will be restricted from receiving predictions.

AutoML

AutoML is an optional configuration step in the process that enables automated retrains every month

Automatic Retrains

At the beginning of each month, AutoML will check to see if the minimum ticket volume threshold of at least 5,000 tickets polled (e.g. passes the triage ticket filter) has been met since the launch of Triage.

If so, then AutoML aggregates all polled tickets that passes the triage ticket filter, up to the last 12 months, and retrains a new model. To validate the new model, AutoML samples 10% of the polled tickets and analyzes the model prediction between the current and new model in a "side-by-side" evaluation.

If there is an improvement in key Triage metrics, e.g. Precision and Coverage, the new model is promoted into production. This entire process is done without customer intervention.

Manual Retrain

For organizations with smaller ticket volumes below the minimum ticket volume threshold, Moveworks can manually complete the retrain as aforementioned. Given the work required, Moveworks only recommends doing this 6 months after launch.

FAQ

Q: Which ITSM does Triage support?

A: Triage is supported by all ITSM integrations. See our integration documentation for the full list of supported ITSM integrations.

Q: What does Triage use to make predictions?

A: By default, Triage uses description and short description as the inputs for training and predictions. Based on your organization's business processes, additional metadata fields such as location can be also taken into account.

Q: How long does it take for Triage to route tickets?

A: Triage can route tickets at a minimum of 30 seconds, and a maximum of 1 minute. Triage can also be configured to route the ticket on the initial creation of the ticket itself. This means when the ticket is created, it will have the correct values on creation.

Q: How many tickets does it take to train a Triage model?

A: To ensure a robust and accurate Triage model, a minimum of 10,000 tickets is required for training.

• Why? The larger the dataset used for training, the lower the variance and the higher the accuracy we can achieve.

Training the model with a low volume of tickets carries the risk of "over-fitting" the deep learning models to the training data. This means the models may become overly biased towards the specific patterns of the training set, resulting in unreliable predictions when faced with new tickets in a production environment.

Q: How does the model “learn”?

A: Triage is a machine learning model that requires continuous learning and improvement. To continue enhancing its performance, it's important for Service Desk Agents to reroute any incorrectly predicted tickets. When Triage makes incorrect predictions and tickets are rerouted by agents, this feedback is automatically collected and used for future retraining. Retraining the model with new data helps it "relearn" and improve its accuracy for future predictions.

Q: What fields does the customer need to provide the CS team?

A: Follow the instructions in Getting ready for Triage

Please contact your CS team about any questions on how to securely upload these data.