Palmyra X4 actions

Introducing actions with Palmyra X4

Sam Julien

FIG. 01 Describe the tools → Plan the call → Check against the schema → Execute and answer
Identifier
WRITER Research
Published
Oct 9, 2024
Authors
1
Type
Model
Area
Enterprise-optimized models

In one paragraph

Palmyra X4 adds native tool calling. Tools are described in JSON schema, a graph-based RAG tool is built in, and WRITER apps can be called as tools. The model scores 78.76% overall accuracy across tool-calling benchmarks.

overall tool-calling accuracy
78.76%
overall tool-calling accuracy
AST accuracy
87.93%
AST accuracy
execution accuracy
88.27%
execution accuracy
token context window
128k
token context window

FIG. 02: Mechanism

How it works

  1. Describe the tools

    The developer sends a tools array with the chat request. Each tool has a name, a description and a JSON schema for its parameters.

  2. Plan the call

    The model decides whether a tool is needed. If one is, it returns the function name and arguments that fit the request.

  3. Check against the schema

    The call must name a declared tool, and its arguments must match the declared types and required fields. This is what BFCL's AST accuracy measures.

  4. Execute and answer

    The application runs the function and passes the result back, and the model writes the final reply. The built-in graph RAG tool runs on WRITER's side and returns cited sources.

FIG. 03: Interactive figure

Plan the call, check it, run it

Pick a request to see which tool the model calls and with what arguments, then break the schema to watch the AST check stop a bad call before it runs.

Loading interactive demo…

FIG. 04: Notes

Summary

Palmyra X4 can take actions. Developers describe tools as a JSON-schema array, and the model decides when to call them and with what arguments. A graph-based RAG tool is built in, and WRITER no-code apps can be invoked as tools.

The tool-use training data was synthesized by Instruct-Adapt-X, a proprietary model that uses an early-stopping mechanism developed at WRITER. (Our Becoming Self-Instruct paper studies early stopping for instruction tuning more generally.)

Results

  • Tool calling: 78.76% overall accuracy, 87.93% AST accuracy and 88.27% execution accuracy.
  • HELM: 86.1% on HELM Lite and 81.3% on HELM MMLU.