--- title: "OMNI Pilot Report: Can Compressing Tool Outputs Make Coding Agents Faster?" description: "I spent some time running 30 controlled tests to see if OMNI's tool-output distillation actually moves the needle on coding agent performance. Spoiler: it depends." publishedAt: 2026-05-14 locale: en urlSlug: omni-pilot-report isDraft: false defaultLocale: en --- [OMNI](https://github.com/wayanjimmy/omni) is a forked version of Fajar Hidayat's OMNI extension ([original](https://github.com/fajarhide/omni)) that adds the OMNI extension implementation for [Pi Coding Agent](https://pi.dev). This pilot ran against the [`v0.6.0-pi-alpha.2`](https://github.com/wayanjimmy/omni/releases/tag/v0.6.0-pi-alpha.2) tag. The idea behind OMNI is simple: take those massive, noisy outputs from tools like `find` or `rg` and distill them into something much leaner before the [agent](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Agent.md) sees them. There's nothing more annoying than an agent getting lost in 500 lines of git logs — so does the compression actually help? I ran a controlled **30-run A/B pilot** (5 tasks × 2 conditions × 3 iterations) to find out. > **Spoiler:** OMNI compresses input context well — sometimes dramatically — but whether that translates to real savings depends entirely on whether the agent stays focused or goes on an exploratory tangent. --- ## The Setup ### Conditions | ID | OMNI | Description | |----|------|-------------| | C1 | ❌ | Baseline — no OMNI extension | | C2 | ✅ | OMNI extension loaded | ### Workload 5 tasks spanning code understanding, debugging, and implementation, all targeting the OMNI [fork's Rust codebase](https://github.com/wayanjimmy/omni): | Task | Type | Difficulty | Est. time | What the agent was asked | |------|------|------------|-----------|--------------------------| | CU-01 | Code Understanding | Easy | 2-3 min | Find and explain the distiller implementation for git output in `src/distillers/git.rs` — identify the main function and describe its filtering logic | | CU-06 | Code Understanding | Medium | 5-7 min | Trace the development history of the distillation pipeline feature using `git log`, `rg`, and related commands — synthesize a summary of when pipeline was introduced, scoring evolution, and key files changed | | DB-01 | Debugging | Medium | 5-7 min | Run `cargo test`, identify any failing tests, trace the root cause, fix it, and verify the fix passes | | IM-02 | Implementation | Medium | 5-7 min | Create a custom TOML filter at `~/.omni/filters/npm_install.toml` that strips progress bars, keeps warnings/errors, and collapses verbose dependency logs — verify with `omni learn --verify` | | IM-05 | Implementation | Medium | 5-7 min | Implement `omni diff --json` in `src/cli/diff.rs` that returns a JSON comparison of raw vs distilled output including metrics (token savings, compression ratio) | ### Matrix - 5 tasks × 2 conditions × 3 iterations = **30 runs** - [Model](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Model.md): [`mimo-v2.5`](https://platform.xiaomimimo.com?ref=YTKZ66) (Xiaomi MiMo-V2.5) — running on [Xiaomi MiMo Open Platform](https://platform.xiaomimimo.com?ref=YTKZ66) (use code `YTKZ66` for ¥10 API credits) - Completion: **100% in both conditions** ### How I Designed the Test A critical design decision: **the same [system prompt](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/System%20prompt.md) was used for both conditions.** The agent was never told about OMNI. It simply had access to [tools](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Tool.md) (`search`, `read_file`, `write_file`, `bash`) and was asked to complete each task efficiently. OMNI operated as transparent middleware — the agent called `search`, got filtered results, and had no idea which condition it was in. **Exact system prompt used for every run:** > You are a coding agent working on a Rust project. > You have access to these tools: search, read_file, write_file, bash. > Complete the task using the most efficient approach. > Do not ask clarifying questions unless absolutely necessary. Task prompts (listed above) were also identical across conditions. I wanted to test the **tools**, not the agent's ability to follow instructions — so the system prompt stayed exactly the same for every run. --- ## The Results: What the Data Tells Us ### Averages Across All 30 Runs | Metric | C1 | C2 | Delta (C2 vs C1) | |---|---:|---:|---:| | Wall clock (s) | 275.7 | 253.3 | **-8.1%** | | [Total tool calls](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Tool%20call.md) | 17.2 | 25.0 | **+45.3%** | | Search [tool calls](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Tool%20call.md) | 3.6 | 7.6 | **+111.1%** | | [Total tokens](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Token.md) | 296,246.9 | 338,010.2 | **+14.1%** | | [Input tokens](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Input%20tokens.md) | 27,702.5 | 19,005.4 | **-31.4%** | | Context pressure | 457.2 | 244.0 | **-46.6%** | | [Turns](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Turn.md) | 12.3 | 18.5 | **+49.8%** | ### What This Means 1. **Compression works.** The -46.6% reduction in context pressure is the clearest signal in the dataset. OMNI genuinely shrinks what the agent has to chew through. 2. **But trajectories expanded.** C2 also triggered more [turns](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Turn.md) (+49.8%) and more [tool calls](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Tool%20call.md) (+45.3%). The agent became more exploratory — almost like the lower cognitive load freed it up to wander. 3. **Net token cost increased.** The extra exploration outweighed the per-step compression on aggregate. Total tokens went up 14.1%. --- ## How It Actually Works (The Chain Reaction) The chain of effects looks like this: ``` Large raw tool output OMNI distillation (find/rg/git log) ──────────────────► Compact context payload -31.4% input tokens -46.6% context pressure │ ▼ Agent trajectory │ ┌───────────────────┴───────────────────┐ ▼ ▼ Stable agent Exploratory agent Fewer [turns], More [turns], potential net savings increased tool use │ │ ▼ ▼ Net savings Net increase (DB-01, CU-06) (CU-01, IM-02, IM-05) ``` --- ## The Split: Where OMNI Helps and Where It Doesn't Averages hide a crucial story. OMNI's impact varies dramatically by task: | Task | Total token delta | Direction | |-----:|---:|:---:| | DB-01 | **-52.5%** | ✅ Big savings | | CU-06 | **-28.6%** | ✅ Moderate savings | | IM-05 | +7.0% | ❌ Slight increase | | IM-02 | +36.1% | ❌ Increase | | CU-01 | +75.0% | ❌ Large increase | Two tasks (DB-01, CU-06) show clear wins — OMNI helped the agent find answers faster with less churn. Three tasks went in the opposite direction, driven mainly by longer, more exploratory trajectories. ### Why the Difference? Tasks where the agent already had a clear strategy (debugging a known pattern in DB-01, understanding a focused code area in CU-06) benefited from OMNI's compression. Tasks where the agent was more uncertain (CU-01, IM-02, IM-05) led to more searching — and OMNI's compression couldn't compensate for the extra [turns](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Turn.md). --- ## What We Know (and Don't Know Yet) ### ✅ Proven - OMNI strongly reduces context payload from tool output. - On certain tasks (DB-01, CU-06), OMNI can materially improve token usage. ### ❌ Not yet proven - Consistent aggregate token reduction across mixed workloads. - Stable runtime improvement under high [non-determinism](https://github.com/mattpocock/dictionary-of-ai-coding/blob/main/dictionary/Non-determinism.md). ## What We Can Do ### Being Intentional About Phases The pilot data reveals a striking pattern: OMNI's impact correlates with the *type of cognitive work* the agent is doing, not just the task category. This connects directly to something I've written about before — the importance of separating exploration, planning, and execution into distinct threads ([Evolution of Coding Agent Workflows](/en/posts/evolution-of-coding-agent-workflow)). Back then, I advocated for **"one thread, one objective"** — keeping planning, execution, and exploration in separate sessions. The OMNI pilot data now provides *empirical evidence* for why this matters: **the tools that help in one phase can actively hurt in another.** ### The Phase-Aware Pattern Drawing from Mario Zechner's "Prompts are Code" framework and the structured coding workflow research (Think-Plan-Execute pattern), we can map OMNI's behavior to cognitive phases: | Phase | What Happens | OMNI? | Why | |---|---|---|---| | **Exploration** | Agent maps the codebase, searches broadly, follows tangents | ❌ OFF | Compression may starve the agent of serendipitous signals. Curiosity needs raw context. | | **Planning** | Agent synthesizes findings into a structured approach | ⚡ OPTIONAL | Depends on whether the plan is context-heavy (git log synthesis → ON) or conceptual (→ OFF). | | **Execution** | Agent implements changes with a clear target | ✅ ON | The agent knows what to do — compression removes noise, reduces context pressure, keeps it focused. | ### The Data Validates the Pattern Looking at the pilot results through this lens: - **DB-01 (-52.5%)**: Debugging a known pattern — pure execution. OMNI thrived. - **CU-06 (-28.6%)**: Synthesizing git history — structured execution with a clear endpoint. OMNI helped. - **CU-01 (+75.0%)**: "Find and explain" — exploratory, agent went on tangents. OMNI compressed inputs but the agent compensated by searching *more*. The pattern is clear: **when the agent knows where it's going, compression accelerates. When it's still figuring things out, compression may constrain the very exploration that leads to understanding.** ### The Formula, Extended In the earlier article, I proposed this formula for effective AI-assisted coding: > **right task × right agent × right thread length × right prompt contract** The OMNI pilot suggests an addition: > **right task × right agent × right thread length × right prompt contract × right phase configuration** That last variable — phase configuration — is where middleware like OMNI belongs. Not as an always-on layer, but as a **phase-aware tool** that adapts to the cognitive mode of the current thread. ### Practical Implications 1. **For tool developers**: Build phase-aware middleware. Detect when the agent is in exploration vs. execution mode (e.g., via tool call patterns — high `search`/`find` frequency = exploration) and toggle compression dynamically. 2. **For users**: Be explicit about phases in your prompts and session management. "Explore this codebase and find..." vs. "Implement X based on this plan..." signals different cognitive modes. This isn't just a prompt strategy — it's a *tooling configuration* strategy. 3. **For benchmark design**: Test OMNI on execution-heavy and exploration-heavy task suites separately. Aggregate metrics mask the phase-dependent behavior. My pilot's -8.1% aggregate improvement hides both -52.5% wins and +75.0% losses. ### The Bigger Picture The thread-based workflow I adopted from Amp wasn't just about keeping context clean — it was an implicit acknowledgment that **different phases need different cognitive environments**. OMNI's data makes this explicit: the same middleware that accelerates execution can sabotage exploration. The next round of benchmarks should test this directly: same tasks, but with OMNI toggled based on detected phase. That's the experiment that could turn this pilot signal into a design principle. --- ## Poke Around the Data Yourself The full pilot data and analysis pipeline are open source: - **GitHub repo:** [`wayanjimmy/omni-pilot`](https://github.com/wayanjimmy/omni-pilot) - **Raw trace data (30 runs):** [`traces/pilot.30runs.jsonl`](https://github.com/wayanjimmy/omni-pilot/blob/main/traces/pilot.30runs.jsonl) - **Analysis script:** [`scripts/analyze-pilot.py`](https://github.com/wayanjimmy/omni-pilot/blob/main/scripts/analyze-pilot.py) - **Full report (markdown):** [`traces/pilot-report.md`](https://github.com/wayanjimmy/omni-pilot/blob/main/traces/pilot-report.md) To reproduce locally: ```bash git clone https://github.com/wayanjimmy/omni-pilot.git cd omni-pilot uv run python scripts/analyze-pilot.py traces/pilot.30runs.jsonl ``` --- ## Glossary of AI Coding Terms Terms used in this report that you may encounter when working with coding agents. Where a term has a matching entry in [Matt Pocock's Dictionary of AI Coding](https://github.com/mattpocock/dictionary-of-ai-coding), a link is provided inline above.
Distillation
Compressing large tool outputs (like find, rg, git log) into compact representations before injecting them into the agent's context. OMNI's core mechanism — the opposite of raw pass-through.
Context pressure
A measure of how full the context window is relative to its capacity. High context pressure means more of the window is consumed, increasing the risk of attention degradation (the model losing sight of earlier information). Calculated here as total input tokens divided by the model's context window size.
Middleware
A software layer that intercepts and transforms data flowing between components. OMNI operates as transparent middleware here: it sits between the agent's tool calls and the model provider request, compressing tool outputs without the agent knowing.
Agent trajectory
The path an agent takes through tool calls, turns, and decisions while completing a task. A "stable" trajectory finds the answer in few steps; an "exploratory" one branches out with more searches and iterations.
A/B pilot
A controlled experiment comparing two conditions (A = baseline, B = treatment) across identical tasks. Here: 5 tasks run with and without OMNI, repeated 3 times each for statistical power.
--- *Want to build your own charts from this data? The structured JSON is [`here`](https://github.com/wayanjimmy/omni-pilot/blob/main/traces/pilot-report-data.json).*