OMNI Pilot Report: Can Compressing Tool Outputs Make Coding Agents Faster?

5/14/2026·View as Markdown

OMNI is a forked version of Fajar Hidayat’s OMNI extension (original) that adds the OMNI extension implementation for Pi Coding Agent. This pilot ran against the v0.6.0-pi-alpha.2 tag.

The idea behind OMNI is simple: take those massive, noisy outputs from tools like find or rg and distill them into something much leaner before the agent sees them. There’s nothing more annoying than an agent getting lost in 500 lines of git logs — so does the compression actually help?

I ran a controlled 30-run A/B pilot (5 tasks × 2 conditions × 3 iterations) to find out.

Spoiler: OMNI compresses input context well — sometimes dramatically — but whether that translates to real savings depends entirely on whether the agent stays focused or goes on an exploratory tangent.

The Setup #

Conditions #

ID	OMNI	Description
C1	❌	Baseline — no OMNI extension
C2	✅	OMNI extension loaded

Workload #

5 tasks spanning code understanding, debugging, and implementation, all targeting the OMNI fork’s Rust codebase:

Task	Type	Difficulty	Est. time	What the agent was asked
CU-01	Code Understanding	Easy	2-3 min	Find and explain the distiller implementation for git output in `src/distillers/git.rs` — identify the main function and describe its filtering logic
CU-06	Code Understanding	Medium	5-7 min	Trace the development history of the distillation pipeline feature using `git log`, `rg`, and related commands — synthesize a summary of when pipeline was introduced, scoring evolution, and key files changed
DB-01	Debugging	Medium	5-7 min	Run `cargo test`, identify any failing tests, trace the root cause, fix it, and verify the fix passes
IM-02	Implementation	Medium	5-7 min	Create a custom TOML filter at `~/.omni/filters/npm_install.toml` that strips progress bars, keeps warnings/errors, and collapses verbose dependency logs — verify with `omni learn --verify`
IM-05	Implementation	Medium	5-7 min	Implement `omni diff --json` in `src/cli/diff.rs` that returns a JSON comparison of raw vs distilled output including metrics (token savings, compression ratio)

Matrix #

5 tasks × 2 conditions × 3 iterations = 30 runs
Model: mimo-v2.5 (Xiaomi MiMo-V2.5) — running on Xiaomi MiMo Open Platform (use code YTKZ66 for ¥10 API credits)
Completion: 100% in both conditions

How I Designed the Test #

A critical design decision: the same system prompt was used for both conditions.

The agent was never told about OMNI. It simply had access to tools (search, read_file, write_file, bash) and was asked to complete each task efficiently. OMNI operated as transparent middleware — the agent called search, got filtered results, and had no idea which condition it was in.

Exact system prompt used for every run:

You are a coding agent working on a Rust project. You have access to these tools: search, read_file, write_file, bash. Complete the task using the most efficient approach. Do not ask clarifying questions unless absolutely necessary.

Task prompts (listed above) were also identical across conditions. I wanted to test the tools, not the agent’s ability to follow instructions — so the system prompt stayed exactly the same for every run.

The Results: What the Data Tells Us #

Averages Across All 30 Runs #

Metric	C1	C2	Delta (C2 vs C1)
Wall clock (s)	275.7	253.3	-8.1%
Total tool calls	17.2	25.0	+45.3%
Search tool calls	3.6	7.6	+111.1%
Total tokens	296,246.9	338,010.2	+14.1%
Input tokens	27,702.5	19,005.4	-31.4%
Context pressure	457.2	244.0	-46.6%
Turns	12.3	18.5	+49.8%

What This Means #

Compression works. The -46.6% reduction in context pressure is the clearest signal in the dataset. OMNI genuinely shrinks what the agent has to chew through.
But trajectories expanded. C2 also triggered more turns (+49.8%) and more tool calls (+45.3%). The agent became more exploratory — almost like the lower cognitive load freed it up to wander.
Net token cost increased. The extra exploration outweighed the per-step compression on aggregate. Total tokens went up 14.1%.

How It Actually Works (The Chain Reaction) #

The chain of effects looks like this:

plaintext

Large raw tool output          OMNI distillation
  (find/rg/git log)    ──────────────────►  Compact context payload
                                              -31.4% input tokens
                                              -46.6% context pressure
                                                    │
                                                    ▼
                                          Agent trajectory
                                              │
                          ┌───────────────────┴───────────────────┐
                          ▼                                       ▼
                    Stable agent                             Exploratory agent
                    Fewer [turns],                              More [turns],
                    potential net savings                     increased tool use
                          │                                       │
                          ▼                                       ▼
                    Net savings                              Net increase
                    (DB-01, CU-06)                            (CU-01, IM-02, IM-05)

The Split: Where OMNI Helps and Where It Doesn’t #

Averages hide a crucial story. OMNI’s impact varies dramatically by task:

Task	Total token delta	Direction
DB-01	-52.5%	✅ Big savings
CU-06	-28.6%	✅ Moderate savings
IM-05	+7.0%	❌ Slight increase
IM-02	+36.1%	❌ Increase
CU-01	+75.0%	❌ Large increase

Two tasks (DB-01, CU-06) show clear wins — OMNI helped the agent find answers faster with less churn. Three tasks went in the opposite direction, driven mainly by longer, more exploratory trajectories.

Why the Difference? #

Tasks where the agent already had a clear strategy (debugging a known pattern in DB-01, understanding a focused code area in CU-06) benefited from OMNI’s compression. Tasks where the agent was more uncertain (CU-01, IM-02, IM-05) led to more searching — and OMNI’s compression couldn’t compensate for the extra turns.

What We Know (and Don’t Know Yet) #

✅ Proven #

OMNI strongly reduces context payload from tool output.
On certain tasks (DB-01, CU-06), OMNI can materially improve token usage.

❌ Not yet proven #

Consistent aggregate token reduction across mixed workloads.
Stable runtime improvement under high non-determinism.

What We Can Do #

Being Intentional About Phases #

The pilot data reveals a striking pattern: OMNI’s impact correlates with the type of cognitive work the agent is doing, not just the task category. This connects directly to something I’ve written about before — the importance of separating exploration, planning, and execution into distinct threads (Evolution of Coding Agent Workflows).

Back then, I advocated for “one thread, one objective” — keeping planning, execution, and exploration in separate sessions. The OMNI pilot data now provides empirical evidence for why this matters: the tools that help in one phase can actively hurt in another.

The Phase-Aware Pattern #

Drawing from Mario Zechner’s “Prompts are Code” framework and the structured coding workflow research (Think-Plan-Execute pattern), we can map OMNI’s behavior to cognitive phases:

Phase	What Happens	OMNI?	Why
Exploration	Agent maps the codebase, searches broadly, follows tangents	❌ OFF	Compression may starve the agent of serendipitous signals. Curiosity needs raw context.
Planning	Agent synthesizes findings into a structured approach	⚡ OPTIONAL	Depends on whether the plan is context-heavy (git log synthesis → ON) or conceptual (→ OFF).
Execution	Agent implements changes with a clear target	✅ ON	The agent knows what to do — compression removes noise, reduces context pressure, keeps it focused.

The Data Validates the Pattern #

Looking at the pilot results through this lens:

DB-01 (-52.5%): Debugging a known pattern — pure execution. OMNI thrived.
CU-06 (-28.6%): Synthesizing git history — structured execution with a clear endpoint. OMNI helped.
CU-01 (+75.0%): “Find and explain” — exploratory, agent went on tangents. OMNI compressed inputs but the agent compensated by searching more.

The pattern is clear: when the agent knows where it’s going, compression accelerates. When it’s still figuring things out, compression may constrain the very exploration that leads to understanding.

The Formula, Extended #

In the earlier article, I proposed this formula for effective AI-assisted coding:

right task × right agent × right thread length × right prompt contract

The OMNI pilot suggests an addition:

right task × right agent × right thread length × right prompt contract × right phase configuration

That last variable — phase configuration — is where middleware like OMNI belongs. Not as an always-on layer, but as a phase-aware tool that adapts to the cognitive mode of the current thread.

Practical Implications #

For tool developers: Build phase-aware middleware. Detect when the agent is in exploration vs. execution mode (e.g., via tool call patterns — high search/find frequency = exploration) and toggle compression dynamically.
For users: Be explicit about phases in your prompts and session management. “Explore this codebase and find…” vs. “Implement X based on this plan…” signals different cognitive modes. This isn’t just a prompt strategy — it’s a tooling configuration strategy.
For benchmark design: Test OMNI on execution-heavy and exploration-heavy task suites separately. Aggregate metrics mask the phase-dependent behavior. My pilot’s -8.1% aggregate improvement hides both -52.5% wins and +75.0% losses.

The Bigger Picture #

The thread-based workflow I adopted from Amp wasn’t just about keeping context clean — it was an implicit acknowledgment that different phases need different cognitive environments. OMNI’s data makes this explicit: the same middleware that accelerates execution can sabotage exploration.

The next round of benchmarks should test this directly: same tasks, but with OMNI toggled based on detected phase. That’s the experiment that could turn this pilot signal into a design principle.

Poke Around the Data Yourself #

The full pilot data and analysis pipeline are open source:

GitHub repo: wayanjimmy/omni-pilot
Raw trace data (30 runs): traces/pilot.30runs.jsonl
Analysis script: scripts/analyze-pilot.py
Full report (markdown): traces/pilot-report.md

To reproduce locally:

bash

git clone https://github.com/wayanjimmy/omni-pilot.git
cd omni-pilot
uv run python scripts/analyze-pilot.py traces/pilot.30runs.jsonl

Glossary of AI Coding Terms #

Terms used in this report that you may encounter when working with coding agents. Where a term has a matching entry in Matt Pocock’s Dictionary of AI Coding, a link is provided inline above.

Distillation: Compressing large tool outputs (like find, rg, git log) into compact representations before injecting them into the agent's context. OMNI's core mechanism — the opposite of raw pass-through.
Context pressure: A measure of how full the context window is relative to its capacity. High context pressure means more of the window is consumed, increasing the risk of attention degradation (the model losing sight of earlier information). Calculated here as total input tokens divided by the model's context window size.
Middleware: A software layer that intercepts and transforms data flowing between components. OMNI operates as transparent middleware here: it sits between the agent's tool calls and the model provider request, compressing tool outputs without the agent knowing.
Agent trajectory: The path an agent takes through tool calls, turns, and decisions while completing a task. A "stable" trajectory finds the answer in few steps; an "exploratory" one branches out with more searches and iterations.
A/B pilot: A controlled experiment comparing two conditions (A = baseline, B = treatment) across identical tasks. Here: 5 tasks run with and without OMNI, repeated 3 times each for statistical power.

Want to build your own charts from this data? The structured JSON is here.