- Identifier
- WRITER Research
- Published
- Apr 28, 2025
- Authors
- WRITER
- Type
- Model
- Area
- Enterprise-optimized models
In one paragraph
A 1M-token context model for agents. It uses hybrid attention, mixing linear and softmax attention, with mixture-of-experts routing. It reads a full 1M-token prompt in about 22 seconds and makes multi-turn function calls in about 300 ms.
- token context window
- 1M
- token context window
- to process a 1M-token prompt
- ~22 s
- to process a 1M-token prompt
- multi-turn function calls
- ~300 ms
- multi-turn function calls
- per 1M input tokens
- $0.60
- per 1M input tokens
FIG. 02: Mechanism
How it works
Mix two kinds of attention
Palmyra X5 blends linear attention with standard softmax attention. Linear attention keeps a fixed-size state, so its cost grows in step with prompt length instead of with its square.
Keep softmax where precision matters
Softmax attention compares every token with every earlier one, which is precise but quadratic. A hybrid keeps it in only part of the network, so most of the long-context cost disappears.
Route each token to a few experts
Mixture-of-experts layers send each token to a small set of expert subnetworks chosen by a router. Only those experts run, so compute per token stays well below the full parameter count.
Serve it fast and cheap
WRITER reports reading a full 1M-token prompt in about 22 seconds, multi-turn function calls in about 300 ms, and a list price of $0.60 per million input tokens.
FIG. 03: Interactive figure
Why a million tokens stays affordable.
Drag the prompt length toward 1M and change how many layers keep softmax attention to see where quadratic attention cost goes, then pick a token to see which experts it is routed to.
Loading interactive demo…
FIG. 04: Notes
Summary
Palmyra X5 is built for agentic workloads that need to hold whole codebases, document sets or long multi-step histories in context. It uses a hybrid attention mechanism that blends linear and softmax attention to reach 1M-token inputs efficiently, and mixture-of-experts routing that sends each token to a subset of experts.
Results
- OpenAI MRCR (8-needle): 19.1%, against 20.25% for GPT-4.1 and 17.63% for GPT-4o, at roughly 3–4× lower cost per token than GPT-4.1.
- BigCodeBench (Full, Instruct): 48.7.
- BBH: 70.99%.
- GPQA: 47.20%.
- MMLU-Pro: 65.02%.
- MATH (hard): 71.57%.