The Moveworks Answers skill enables the Moveworks bot to answer a user’s questions directly in chat, by understanding the questions and responding instantly with answers containing the most relevant article snippets, links to articles in the knowledge base, or links to external articles taken from trusted knowledge sources. The content of these answers come from the knowledge bases that your team already manages in repositories like ServiceNow Knowledge Management, Microsoft SharePoint, Confluence and many more.

The Answers skill is composed of these key features:

• Prioritized Answer — The bot is able to answer the user’s question by extracting a snippet from a knowledge base article that can solve their issue.
• Related Answers — The bot is able answer the user’s question by linking to a knowledge base article that can solve their issue.
• External Answers — The bot is able answer the user’s question by linking to an external article from the official customer support documentation page of a product that can solve their issue.

Prioritized Answer

When users pose a question to the Moveworks bot, the bot will understand what is being asked and can respond with an article and snippet that the bot is confident can best answer the user’s question. The user can click on link in the bot’s message to be taken to the full article or they can click on Show related articles to see other articles the bot found that is relevant to the user’s question.

Users can also click on 👍 Yes or 👎 No to give feedback, or they can click on File Ticket to file a ticket directly in chat to ask for more help.

📘

Note: Feedback from users is taken into account in periodic model updates as one of the signals to relevance. However, this does not always mean that the article will be prevented from being shown in the near future, as feedback is only taken into consideration after it reaches a statistically significant level. This also ensures that a single user or small number of users are not able to influence the behavior for the broader organization.

See our Profile Boosting guide for more information on the Moveworks Prioritized Answers feature.

Related Answers

When users pose a question to the Moveworks bot, the bot will understand what is being asked and it will search through your knowledge base to find the best answer to the question. If the bot cannot find a top answer that it is confident can address the user’s question, like in the example above, the bot will provide the user with links to multiple articles in your knowledge base that it is confident may answer the user’s question. Users can then click on the links provided, and be redirected to the article’s page in the knowledge base in a new window.

📘

Note: Answers must meet a minimum relevancy to be served. If no answers meet this requirement, the bot may choose to show responses from other skills such as Forms or inform the user that no relevant solution was found, and that they may file a ticket for further support.

External Answers

When users pose a question to the Moveworks bot that explicitly references an application, bot will understand what is being asked and can respond with links to external articles that best answers the question. External articles are taken from trusted external knowledge sources.

See our External Answers guide for more information on the Moveworks’ External Answers skill.

How does Answers work?

When a user asks the Moveworks bot a question, the bot takes this query and uses its Search skill, to search through knowledge articles taken from your organization’s internal knowledge base, and external knowledge pulled from trusted sources.

Once the bot is done it creates a list of articles or “answers” that it thinks is capable of answering the question. The bot also gives each answer a confidence score. A confidence score is a numerical value representing the bot’s confidence in whether this answer will resolve the user’s question. Scoring allows the bot to take the other answers it found into account. So a very high-scoring answer will exclude other answers, which ensure users are not distracted with lower relevance answers.

The bot takes this list of answers and ranks them by their confidence score. Once the answers are ranked, the bot sees if any of the answers passed the display threshold. The display threshold is the minimum confidence score an answer must achieve to be eligible of being shown to a user. If no answers found pass the display threshold, the bot will not show the user any of the answers. Next, the bot sees if any of the answers passes the prioritized answer threshold.

The prioritized answers threshold is a threshold that is set at a high confidence score. If one or more answers has a confidence score above the prioritized answer threshold, the bot will take the answer with the highest confidence score and send the user a message with that answer and a button that says Show related articles. If the user clicks on Show related articles, the bot will send the other articles it found that passed the display threshold. The bot tries to send the user a prioritized answer when it can so that users can see the article most likely to answer their query first.

If no answers passed the prioritized answers threshold but some answers did pass the display threshold, then the bot will send the user a message saying that it found these related articles that may answer the question being asked and it will send the user the answers that passed the display threshold.

Answers Knowledge Base integrations

In order to provide your users with the best and most relevant answers, the Moveworks bot ingests articles from your internal knowledge bases and FAQs.

Ingesting article content

To populate your organization's set of answers, the Moveworks service ingests articles from your internal knowledge bases (KBs) and FAQs. Your set of answers is private to your organization and can only be viewed by your users. During ingestion, long articles are divided into shorter articles, called snippets, that are more likely to answer an employee's exact query. Moveworks natural language understanding (NLU) routines identify word embeddings in each article.

Moveworks NLU does this by being context aware, evaluating each word and sequence of words to discern its meaning with respect to the sentence, heading, and it appears in. It knows synonyms and understands alternate phrasings, so that queries and articles are treated as language instead of just keywords. Simple keyword search fails when the words in the query don't match the words used in the KB article, but semantic search consistently returns good results, even when the user uses alternate names not present in the article, or when they phrase their question in an unexpected way. For example, with semantic search, a question like, “How do I setup vpn on my iPhone?” will retrieve the correct article, even if that article uses the term mobile device instead of iPhone.

Finding the answer

When a user asks the bot a question, Moveworks employs semantic understanding to match the most suitable article or content with the query. This also means that for every unique query, your bot analyzes the query and all available knowledge articles to find the ones that best answer the query every time. To find relevant answers, the bot uses a retrieval and ranking phase.

The retrieval phase uses a high-performance index to quickly filter a set of candidates from a large volume of answers. From the set of filtered candidates, the Moveworks ranking phase scores each candidate using a large set of signals that include the similarity of the question’s written content to the content of each candidate answer; relation to standard rephrasing of the question; relation to synonyms for words in the question; and its relevance with respect to other things we know about the entities being asked about (for example, the ranking routine knows that both “Visio” and “LucidChart” refer to diagramming tools).

For every utterance-document pair, a score is generated that reflects the system’s confidence in the match. Scoring is a comparative action that, among other inputs, takes the other candidate answers into account. This means that a very high-scoring answer will tend to exclude other answers, which helps ensure the user is not distracted with lower relevance answers.

Moveworks compares each answer’s score with an internal threshold to ensure that we return useful answers while not distracting the user with potentially irrelevant answers. The bot also “snippetizes the answer to ensure the user can find the solution to their query as easily as possible.

Snippetizing

For longer articles, Moveworks divides the article into multiple snippets that can be used to answer individual questions. We call this snippetizing - showing the relevant snippet in bot.

When a user asks the bot a questions, the Moveworks bot then searches across all articles to find the most relevant snippets.

For best results, write articles which snippetize well by following the guidelines below:

1. Where possible, divide your article into sections.
2. Title each section using a clear, specific heading.
3. Ensure each section can stand on its own when served as a snippet.

Snippet character limits

For KB articles and FAQs, the maximum number of characters that can be shown in a snippet is 500 characters. The title of the article does not count towards the maximum number of characters.

📘

Note: A snippet is not always exactly 500 characters. Snippets end at the first space after 500 characters, this is done so that the last word in a snippet is not cut off.

How Answers improves over time

Moveworks Answers becomes more accurate based on implicit and explicit feedback from the users. For example, an explicit signal is registered when a user is unhappy with an answer Moveworks provided and clicks the thumbs-down button. If a user files a ticket because they still need help after reading an answer Moveworks provided, we track that as an implicit signal that the user did not get a helpful answer. In addition to these user-provided signals, our internal service-quality reviews provide explicit feedback to improve our answer relevance models.

Note that user feedback is one of the many signals used for determining relevance of an article for a query, but it is not the only one.

• When we update our search models regularly (a few times a year), these feedback signals are taken into account.
  If an article is presented and downvoted, its relevance score on a scale of 0 to 1 may decrease in proportion to the number of negative feedback signals.
• However, this will generally only affect whether the article is shown or not if the article was a marginal result, i.e. low relevance, to begin with.
• This ensures that the right combination of machine learning-based relevance and user feedback is employed to find the best answers to questions.

Language learning occurs continuously, with natural language understanding models trained daily on your enterprise-specific utterances and KB contents. NLU models are also trained using aggregated, anonymized domain-specific data across all customers, ensuring that improvements are available to every customer while maintaining privacy and security.

Moveworks helps you address gaps in KB content

Machine learning makes the Moveworks Answers service more accurate over time, but ultimately the bot’s answering ability is limited to the set of answers available in your knowledge base. To address missing knowledge, Moveworks identifies knowledge gaps. That is, we identify user questions that Moveworks was unable to fully answer, and Moveworks Customer Success reporting makes your help desk team aware of topics where Moveworks found no relevant articles.

When an irrelevant or unhelpful answer is served, the user typically clicks thumbs-down to mark it as “Not Helpful,” or they file a ticket to send their question to a service desk agent. The Moveworks Answers service flags these interactions so that missing and inadequate answers can be addressed with knowledge base updates. At a regular interval, Moveworks notifies your IT organization of questions that went unanswered due to possible gaps in the knowledge base.

FAQ

Q: What types of knowledge base articles are supported?

A: KBs are ingested into the bot experience through a service. There are three KB types that we support:

1. Internal: Moveworks can ingest answers from sources such as : ServiceNow, Confluence Wiki, Zendesk, Cherwell, Igloo, LumApps, Salesforce, SharePoint, Github, Google Drive, and many more...
2. External: External Knowledge is knowledge crawled by Moveworks from official support documentation for various products.
3. FAQ: Customers can fill out a Google Sheet with two columns: Question and Answer. This FAQ document is then ingested into the Answers Enterprise Cache.