Palmyra X5

Palmyra X5: The 1M-token enterprise LLM

FIG. 01 Mix two kinds of attention → Keep softmax where precision matters → Route each token to a few experts → Serve it fast and cheap
Identifier
WRITER Research
Published
Apr 28, 2025
Authors
WRITER
Type
Model
Area
Enterprise-optimized models

In one paragraph

A 1M-token context model for agents. It uses hybrid attention, mixing linear and softmax attention, with mixture-of-experts routing. It reads a full 1M-token prompt in about 22 seconds and makes multi-turn function calls in about 300 ms.

token context window
1M
token context window
to process a 1M-token prompt
~22 s
to process a 1M-token prompt
multi-turn function calls
~300 ms
multi-turn function calls
per 1M input tokens
$0.60
per 1M input tokens

FIG. 02: Mechanism

How it works

  1. Mix two kinds of attention

    Palmyra X5 blends linear attention with standard softmax attention. Linear attention keeps a fixed-size state, so its cost grows in step with prompt length instead of with its square.

  2. Keep softmax where precision matters

    Softmax attention compares every token with every earlier one, which is precise but quadratic. A hybrid keeps it in only part of the network, so most of the long-context cost disappears.

  3. Route each token to a few experts

    Mixture-of-experts layers send each token to a small set of expert subnetworks chosen by a router. Only those experts run, so compute per token stays well below the full parameter count.

  4. Serve it fast and cheap

    WRITER reports reading a full 1M-token prompt in about 22 seconds, multi-turn function calls in about 300 ms, and a list price of $0.60 per million input tokens.

FIG. 03: Interactive figure

Why a million tokens stays affordable.

Drag the prompt length toward 1M and change how many layers keep softmax attention to see where quadratic attention cost goes, then pick a token to see which experts it is routed to.

Loading interactive demo…

FIG. 04: Notes

Summary

Palmyra X5 is built for agentic workloads that need to hold whole codebases, document sets or long multi-step histories in context. It uses a hybrid attention mechanism that blends linear and softmax attention to reach 1M-token inputs efficiently, and mixture-of-experts routing that sends each token to a subset of experts.

Results

  • OpenAI MRCR (8-needle): 19.1%, against 20.25% for GPT-4.1 and 17.63% for GPT-4o, at roughly 3–4× lower cost per token than GPT-4.1.
  • BigCodeBench (Full, Instruct): 48.7.
  • BBH: 70.99%.
  • GPQA: 47.20%.
  • MMLU-Pro: 65.02%.
  • MATH (hard): 71.57%.