- Identifier
- WRITER Research
- Published
- Oct 9, 2024
- Authors
- 1
- Type
- Model
- Area
- Enterprise-optimized models
In one paragraph
Palmyra X4 adds native tool calling. Tools are described in JSON schema, a graph-based RAG tool is built in, and WRITER apps can be called as tools. The model scores 78.76% overall accuracy across tool-calling benchmarks.
- overall tool-calling accuracy
- 78.76%
- overall tool-calling accuracy
- AST accuracy
- 87.93%
- AST accuracy
- execution accuracy
- 88.27%
- execution accuracy
- token context window
- 128k
- token context window
FIG. 02: Mechanism
How it works
Describe the tools
The developer sends a tools array with the chat request. Each tool has a name, a description and a JSON schema for its parameters.
Plan the call
The model decides whether a tool is needed. If one is, it returns the function name and arguments that fit the request.
Check against the schema
The call must name a declared tool, and its arguments must match the declared types and required fields. This is what BFCL's AST accuracy measures.
Execute and answer
The application runs the function and passes the result back, and the model writes the final reply. The built-in graph RAG tool runs on WRITER's side and returns cited sources.
FIG. 03: Interactive figure
Plan the call, check it, run it
Pick a request to see which tool the model calls and with what arguments, then break the schema to watch the AST check stop a bad call before it runs.
Loading interactive demo…
FIG. 04: Notes
Summary
Palmyra X4 can take actions. Developers describe tools as a JSON-schema array, and the model decides when to call them and with what arguments. A graph-based RAG tool is built in, and WRITER no-code apps can be invoked as tools.
The tool-use training data was synthesized by Instruct-Adapt-X, a proprietary model that uses an early-stopping mechanism developed at WRITER. (Our Becoming Self-Instruct paper studies early stopping for instruction tuning more generally.)
Results
- Tool calling: 78.76% overall accuracy, 87.93% AST accuracy and 88.27% execution accuracy.
- HELM: 86.1% on HELM Lite and 81.3% on HELM MMLU.