Conversations API overview

The Conversations API is a REST interface for creating and interacting with Moveworks Assistant conversations from your own application. Your service sends a user’s message, the assistant plans and executes actions against connected systems, and you receive the assistant’s response back, either by polling or over a streaming connection.

Every request runs on behalf of a specific user, so responses reflect that user’s roster identity, permissions, connected content, and conversation history.

What you can build with it

The API is intentionally general-purpose. Three common patterns:

• Custom user-facing chat surfaces. Embed the assistant in a product you already own, such as a customer portal, an internal web app, a mobile app, or a voice or messaging surface. Users sign in with your identity provider, and your application exchanges their token for a Moveworks bearer that scopes each request to the signed-in user.
• Backend and workflow integrations. Trigger assistant conversations from a service that already knows which user each action is on behalf of. Examples include ambient monitoring, batch processing, ticket-driven automation, or systems that need to consume assistant output programmatically. Authentication is handled by a backend credential rather than an interactive sign-in.
• Evaluation and quality workflows. Programmatically send curated test queries through the assistant to measure answer quality, citation accuracy, plugin behavior, and regressions over time.

Choosing an authentication path

Every conversation runs as a specific user. How your application obtains a token that identifies that user depends on how your integration is architected.

All three paths land at the same Moveworks token endpoint (POST /oauth/v1/token) and produce a bearer token you use on every Conversations API request.

Interaction patterns

Once authenticated, every conversation follows the same shape:

1. Create a conversation with POST /conversations. You get back a conversation_id that scopes all subsequent activity.
2. Send a message with POST /conversations/{conversation_id}/responses (for polling) or POST /conversations/{conversation_id}/responses/stream (for streaming).
3. Read the response by polling until the status is COMPLETED, or by consuming Server-Sent Events on the streaming endpoint.
4. Continue the conversation by sending more messages on the same conversation_id. Conversation history is preserved server-side.

Streaming is recommended for user-facing surfaces where perceived latency matters. Polling is simpler to implement and appropriate for backend integrations where progressive rendering is not needed. Both patterns are documented alongside their endpoints in the API reference.

Running evaluations with the Conversations API

A common use of the Conversations API is running an evaluation dataset against an assistant, either as a regression check when you change assistant configuration or as an ongoing quality signal.

Because evaluations typically need to run many rows on behalf of one or more test users without an interactive browser flow, the service-to-service authentication path is the natural fit. The same backend credential mints a bearer per user, and you cycle through your dataset row by row.

Evaluation dataset structure

A working evaluation dataset is a flat collection of rows in whatever format suits your team (spreadsheet, JSON Lines, database table). Each row represents one query you want to evaluate. Fields that tend to be useful:

A well-formed dataset is a real product artifact; it takes iteration to build. Start with 20 to 50 queries covering your primary use cases and grow from there. Treat the dataset as versioned in the same way you version code.

Per-row execution flow

For each row in the dataset, your runner:

1. Choose the user. Either the per-row assigned user, or a default test user, or the next in a rotation, depending on how your dataset is structured.
2. Obtain a bearer token for that user. If you cache per-user bearers, reuse the cached one until it expires; otherwise mint a fresh one.
3. Create a conversation with a title that references the row (for example, eval:{test_case_id}). This isolates the row in your Moveworks conversation history and makes debugging easier.
4. Send the query on that conversation. Use polling for simplicity; streaming works too if you want to inspect intermediate reasoning.
5. Wait for completion with a per-row timeout (30 to 120 seconds is typical). If the response times out or fails, record that outcome rather than retrying indefinitely; persistent slowness is itself a signal worth capturing.
6. Capture the outputs you care about: the assistant’s final text, the citations, and any metadata (which plugins ran, the conversation ID for later inspection, latency).
7. Score the response using whatever combination of scorers you’ve chosen (see below).
8. Persist the row’s result: pass or fail, per-scorer values, the raw response, and enough context to reproduce or investigate later.

Scoring approaches

There is no single right way to score assistant responses. The scorers below are common building blocks, and most working eval setups combine several of them.

Exact-match or substring. Deterministic, cheapest, most brittle. Best for factual answers with a canonical form (“your PTO balance is X hours”, “the ticket status is Approved”). Fragile against phrasing changes, so it works for a narrow class of queries.

Fuzzy similarity. A string-similarity metric (Levenshtein, token overlap, embedding cosine) with a threshold. Tolerates surface phrasing differences. Cheaper than an LLM judge but blind to whether the meaning is correct. A semantically wrong response can score highly if it shares vocabulary with the expected answer.

LLM-as-judge. Send the query, the expected answer, and the actual response to a language model with a rubric (“rate 1-5 on faithfulness to expected answer; a 4 or 5 is acceptable”). Handles paraphrasing and reasoning well but is non-deterministic, costs per-row API spend, and needs prompt tuning to be stable. Two starter guardrails: run each row through the judge twice and average, and periodically spot-check judge decisions against a human reviewer to catch drift.

Citation overlap. Compare the set of URLs the assistant cited against the expected set. Two useful metrics:

• Recall (did the assistant cite the sources it should have?): appropriate when extra citations are acceptable.
• F1 (recall plus penalty for extra or wrong citations): appropriate when you want to catch over-citation.

URL comparison is where most citation scoring gets subtle. Trailing slashes, casing, environment subdomains, and portal alias parameters (like ServiceNow’s id=kb_article_view vs id=kb_article) all refer to the same resource but look different as strings. Normalize before comparing, and treat ServiceNow URLs by logical record (env subdomain and view alias stripped) rather than exact match. Do not rewrite URLs in-place; only normalize for comparison.

Plugin / action assertions. For rows where the expected behavior is “the assistant should invoke plugin X” or “the assistant should file a ticket in Y form,” inspect the tools/plugins that ran during the conversation rather than the final text. This is the right approach for action-taking cases where “correct” is defined by what happened, not by what was said.

Human review. A subset of runs (or all rows on a fresh dataset) get reviewed by a human. Expensive but authoritative. Common patterns: full human review on the first N runs of a new dataset to calibrate automated scorers, then automated scoring with sampled human spot-checks on subsequent runs.

Most production eval setups combine an answer-quality scorer (LLM judge or fuzzy similarity) with a citation scorer, and require both to pass for the row to count as passing. Threshold choices for what “passing” means for each scorer are yours to set based on what actionable signal you want.

Running as multiple users

If different rows in your dataset should run as different users (for example, testing locale-specific knowledge, role-based permissions, or persona-specific content), two patterns work well:

• Per-row user assignment. Each row names the user it should run as. Your runner mints a bearer for that user and uses it for that row only. Falls back to a default user for rows that don’t name one. Best when you want one combined result set with attribution per row.
• Round-robin per user. Every user runs the full dataset. Each user produces their own output file, plus a roll-up comparison. Best when you’re comparing behavior across personas rather than testing individual queries per user.

Both patterns use the same service-to-service credential; only the sub claim changes per user when signing the assertion.

Practical considerations

• Runs count against the same rate limits as production traffic. Plan concurrency and dataset size around your tenant’s request budget, and back off on 429 responses. Concurrency of 2 to 4 in-flight rows and a sliding-window rate cap around 120 requests per minute is a safe starting point; test in a small batch before running a full dataset.
• Runs create real conversations in your Moveworks tenant. Some customers isolate evaluations in a dedicated sandbox tenant or against a dedicated set of test users to keep evaluation traffic out of production analytics. Others use their production tenant intentionally so the evaluation reflects real-world configuration.
• Assistant behavior is not deterministic. Two runs of the same query can produce different phrasings, different plugin choices, and occasionally different citations. Build your scoring logic to tolerate variation on non-semantic dimensions (word choice) while catching regressions on semantic dimensions (wrong answer, missing citation, wrong plugin). If you need to detect small regressions, average multiple runs per row rather than treating a single run as authoritative.
• Latency varies per query. Some queries return in a few seconds; some involve multiple plugin calls and take longer. Budget a per-row timeout (30 to 120 seconds is typical) rather than waiting indefinitely, and record the timeout outcome as a distinct signal from a scored failure.
• Keep raw responses. Store the full request and response for every row, not just the pass/fail verdict. When a scorer produces a surprising result, you want the raw response, the reasoning trace, and the conversation ID available for inspection. A per-row JSONL log alongside your summary report tends to be the right shape.
• Version the dataset. As you add and refine cases, the dataset drifts. Tagging or versioning the dataset lets you interpret run-to-run comparisons correctly (a pass-rate drop can come from real regressions or from adding harder cases).

Next steps

• Pick the authentication path that matches your integration and follow its setup guide.
• Once you have a bearer, walk the API reference to see every endpoint, request shape, and response schema.
• Confirm the smoke-test flow at the end of each auth guide works end to end before wiring the API into production traffic.