Results
Date of run: 21 Oct 2026. Costs in US dollars per task. “Cache cold” means the system prompt and playbook are not yet cached; “cache warm” means an earlier run already cached them. Simulated
| Task | Agent steps | Baseline cost | NeoAI, cache cold | NeoAI, cache warm |
|---|---|---|---|---|
| Market-entry research brief | 10 → 7 | $0.0499 | $0.0189 (−62%) | $0.0155 (−69%) |
| Bug fix in an unfamiliar codebase | 10 → 7 | $0.0489 | $0.0191 (−61%) | $0.0157 (−68%) |
| Churn analysis memo from 3 data exports | 9 → 8 | $0.0470 | $0.0210 (−55%) | $0.0176 (−62%) |
| Task | Baseline tokens | NeoAI tokens | Change |
|---|---|---|---|
| Market-entry research brief | 95,718 | 26,914 | −72% |
| Bug fix in an unfamiliar codebase | 96,704 | 27,834 | −71% |
| Churn analysis memo from 3 data exports | 91,428 | 32,319 | −65% |
The optimised runs make more model calls than the baseline (12, 13 and 12 against 10, 10 and 9) because some steps run on a small model. They still use far fewer tokens, because the frontier model sees a shorter context.
What was replayed
Each task is a fixed trajectory of agent steps: a tool call or a model action, a tool output of a stated size, and a short piece of model reasoning. The replay builds the exact context each step would send, step by step, then counts the tokens. No model is called, and the text inside each tool output is synthetic filler, so only its length matters.
- Market-entry research brief: choose two markets to enter first for a B2B data-labelling service. Searches, page fetches, a comparison in Python, and a written brief.
- Bug fix in an unfamiliar codebase: find why a CSV export drops rows after 10,000 and make the tests pass.
- Churn analysis memo: explain a Q3 churn rise from three data exports and recommend three actions.
How the optimised run differs from the baseline
- A short playbook of about 70 tokens is added to the system prompt, standing in for a recipe. In the replay it steers the agent away from steps marked as known dead ends, so those steps are skipped.
- Simple steps go to a small model. A step marked as simple enough sees only the task and its own tool output, and returns a short distilled note instead of the raw output.
- Shorter history. The frontier model's running history holds those distilled notes rather than full tool outputs.
- The frontier model handles the rest. Judgment steps, such as the comparison and the final write-up, stay on the frontier tier.
Assumptions and prices
| Item | Value used |
|---|---|
| Tokenizer | tiktoken, o200k_base |
| Frontier price (per million tokens) | $2.00 input, $0.10 cached input, $10.00 output |
| Small price (per million tokens) | $0.10 input, $0.01 cached input, $0.50 output |
| Price source and date | OpenAI standard list prices, 21 Oct 2026 |
| System prompt and tool schemas | 1,682 tokens, identical in both runs |
| Per-step reasoning | 120 words of model output |
| Final deliverable | 350 words |
| Tool output sizes | 200 to 2,500 words per step, stated per step |
| Prompt caching | Applied to both pipelines once the unchanged prefix reaches 1,024 tokens |
| Dead ends and simple steps | Marked by us per step. These are assumptions. |
What this replay does not show
- It does not measure quality. A cheaper run that gives a worse answer would look better here. Only a pilot measures quality pass rate.
- The dead ends and simple steps are assumed, not discovered. The savings depend on a recipe that can really identify them on your tasks.
- It is three tasks, not a sample of your workload. Your results will differ, in either direction.
- It is not a customer result. NeoAI has no published customer results yet.
That is why the pilot exists: a baseline and a NeoAI period on your own evaluation set, with quality measured against your acceptance criteria. The script and its raw output are available on request: email contact@neoai.solutions.