🚧

Limited Preview Capability

This capability is currently in Limited Preview. You can learn more in our community.

Overview

The Code Interpreter Plugin can write code to analyze data related to a user's request and provide accurate & verifiable calculation.

Examples

Ask Quantitative Questions

If the user asks a quantitative question, the Code Interpreter write a script to calculate the result.

The Assistant can also write these scripts by combining results across multiple plugin responses. In the example below, equity data from Workday is combined with stock data from Perplexity to calculate the value of an option.

Analyze Live & Permission-Aware Data

Unlike data that's been ported to a data lake, the Code Interpreter works with Agent Studio plugins to analyze live & permission-aware data.

One customer built the following use case that uses stock grant information from Workday, and stock price information from Perplexity. As a result, employees can always ask how much their next grant is worth without contacting a compensation specialist.

Process Large Responses

In the following example, the Assistant has access to a plugin that retrieves ALL Account objects from Salesforce. However, if you asked an LLM to summarize hundreds of records with hundreds of fields – it would struggle to do so effectively.

However, since the Code Interpreter can process many records rapidly (similar to Excel or a calculator), it can provide reliable results while reasoning over a large volume of data.

Citations

Every Code Interpreter execution is accompanied by the code blocks that contributed to its calculation. If you click on the reference popup, you can see the code that was written.

Limitations

1. Plugin responses only. This will not work with capabilities like BriefMe (e.g. with Excel or CSV files).
2. Plugin responses sizes The maximum amount of data that can be returned is 400KB. If more than 400KB are returned, we will attempt to compress it. In practice, compression can convert ~3MB of Salesforce records to fit in our Working Memory.
3. No visualization. Code Interpreter will not produce charts, tables, graphs, or downloadable CSVs.
4. Semi-structured data. If a plugin response contains a text field with 2000+ characters, we disable Code Interpreter processing. This prevents our Reasoning Engine from attempting to write python code to analyze a knowledge article.
5. Limited code execution environment. Currently our Code Interpreter plugin only has access to a subset of Python – intended to keep your data secure. As a result, it lacks access to more advanced data processing libraries. We are working on bringing full Python to the Code Interpreter so it can do more advanced analyses – follow our roadmap item in community.

Evaluation & Performance

We've tuned our Code Interpreter plugin to work very well with enterprise data. As we improve the core architecture of this plugin, we will continue to unlock more advanced types of questions.

To help you understand the types of queries that will work well with our Code Interpreter, we've broken down their characteristics into...

• 🟢 Expected to perform very well
• 🟡 May perform well
• 🔴 Not available

If your organization needs one of the below, please share your details as a product idea in our Community!

• Number of Sources
  • 🟢 Data provided by the user
  • 🟢 Data returned by a single plugin
  • 🟡 Data returned by multiple plugins
• Record Types
  • 🟢 Homogenous record structures
    • i.e. every record in the response has a shared structure, e.g. SFDC Accounts
  • 🔴 Heterogenous record structures
    JSON

    [
    	{
    		"resource_id": "5678s9ds8ds9as78d9",
    		"type": "item",
    		"item": {
    			/* item attributes */
    		}
    	},
    	{
    		"resource_id": "d45f6g78dsf21389ds",
    		"type": "category",
    		"category": {
    			/* category attributes */
    		}
    	}
    ]
    

  • 🔴 Non-lists of records
• Computation
  • 🟢 Aggregation (count, sum, average, median, mode, percentile, etc.)
  • 🟡 Conditional Aggregation (x += object.attr1 if condition(object) else object.attr2)
    • Example: “What our total ARR target for end of Q1” → If account.type == "Prospect" then add “ARR”, but if account.type == "Customer then add “Growth ARR”
    • It will likely perform this, but it may not state its assumptions back to the user.
  • 🔴 Forecasting (linear regression)
  • 🔴 Correlation
  • 🔴 Anomaly detection
• Group-by Aggregation & Filtering
  • 🟢 Small Enum Facets
    • Example: "How many customers are in Tele com" → will match to Telecom and do the grouping
  • 🟡 Big Enum Facets
    • Example: "How many support tickets were handled by Jenn" → if there are thousands of users, might struggle to match "Jenn"
  • 🟡 Computed Facets
    • Example: "What was our retention rate last half for the All Star accounts?" → If All Star is determined by customer.arr > 100,000 && customer.num_skus > 10 && customer.num_licenses > 4000, then Code Interpreter may struggle to come up with that calculation by itself.
  • 🔴 Semantic Facets
    • Example: "Give me a count of the most popular themes of objections raised for opportunities closed last quarter" → Assuming "objections" is not a dropdown, the Code Interpeter has no ability to "label" and then group.
• Presentation
  • 🟢 Computed Result
  • 🟢 Code Citations
  • 🟢 Citations of original records
  • 🟡 Natural language descriptions of the code
  • 🔴 CSV download
  • 🔴 Visualization (tables / charts)