Prompt Compression

Overview

Prompt Compression reduces input token count by 5-14x on multi-turn chat conversations before sending them to LLM providers. It uses LLMLingua-2, an extractive token classification approach that identifies and removes redundant tokens while preserving the original text — no paraphrasing, no hallucination risk.

Compression runs in parallel with guardrails scanning and model routing, adding no latency to the request pipeline. It applies to /v1/chat/completions requests with multiple messages and to the state of /v1/systemone decision requests. Single-turn completions and embeddings are passed through unmodified.

For /v1/systemone, only the state is compressed. A text state is compressed as one text. In a structured (JSON) state, each string value is compressed separately, and keys, numbers and the structure are left untouched. Questions, instructions and option descriptions are never compressed. The same enable switches (global and per client), compression rate, minimum word count, quoted-text protection and [abridged] notice apply. min_messages and preserve_recent do not apply, because a decision request has no message history.

Key benefits:

5-14x input token reduction (empirically benchmarked)
Zero hallucination risk — extractive only, keeps original tokens verbatim
Fast inference: 20-80ms on CPU for typical prompts
Parallel execution in the request pipeline
Per-client granular control over compression behavior

LLMLingua-2 Engine

LLMLingua-2 is a token classification model that assigns a "preserve" probability to each token in the input. Tokens below the threshold are dropped, and the remaining tokens are reassembled into coherent text.

Unlike generative compression (which paraphrases), extractive compression never introduces new tokens — only original tokens from your messages appear in the compressed output.

Extractive Token Classification

The compression process:

Tokenize — Input messages are tokenized using the model's tokenizer
Encode — A single forward pass through the BERT encoder produces per-token embeddings
Classify — A classification head assigns preserve/drop probability to each token
Filter — Tokens below the compression rate threshold are dropped
Reconstruct — Remaining tokens are joined back into text

This runs as a single forward pass through a small BERT model (not a generative LLM), making it extremely fast — typically 20-80ms on CPU.

Model Options

Four ONNX models are available, downloaded from HuggingFace on first use:

Model	Size	Speed	Quality	Best For
TinyBERT	57 MB	Fastest	Good	High-volume, cost-sensitive workloads
MobileBERT	99 MB	Fast	Better	Default — good balance of speed and quality
BERT-base	710 MB	Moderate	High	Quality-sensitive applications
XLM-RoBERTa-large	2.2 GB	Slower	Best	Multilingual content, maximum quality

Models are cached locally after download and use ONNX Runtime for inference (no PyTorch required).

Compression Pipeline

The compression pipeline is integrated into the chat request flow:

Request arrives
    ├── Guardrails scan (original messages) ──┐
    ├── Decision routing    ──────────────────┤ parallel
    └── Prompt compression ───────────────────┘
                │
                ▼
    Compressed messages sent to provider

Key pipeline behaviors:

Guardrails always scan original messages — safety checks see uncompressed content
Tool calls and tool results are never compressed — structured data is preserved exactly
System prompts optionally compressed — disabled by default to preserve instruction fidelity
Recent messages preserved — the most recent N messages are kept uncompressed (default: 4)
Fallback on failure — if compression fails or is disabled, original messages pass through

Configuration

Prompt Compression is configured globally and can be overridden per client.

Compression Rate

The compression rate controls how aggressively tokens are dropped:

Rate	Token Reduction	Use Case
0.8	~20% removed	Light compression, maximum fidelity
0.5	~50% removed	Default — good balance
0.3	~70% removed	Aggressive, best for long conversations
0.1	~90% removed	Maximum compression, may lose nuance

Lower values mean more aggressive compression. The default rate of 0.5 typically achieves 5x token reduction.

Message Settings

Setting	Default	Description
`min_messages`	6	Minimum message count before compression activates
`preserve_recent`	4	Number of most recent messages kept uncompressed
`compress_system`	false	Whether to compress system prompts

The min_messages threshold prevents compression on short conversations where it provides little benefit. preserve_recent ensures the AI always has full context for the most recent exchange.

Per-Client Override

Each client can override the global Prompt Compression settings:

Inherit (default) — Uses global compression settings
Enabled — Forces compression on with custom rate and thresholds
Disabled — No compression for this client

Per-client overrides allow you to enable aggressive compression for cost-sensitive automated pipelines while keeping compression off for interactive coding sessions where every token matters.

Context Management

Memory