I Ran 106 Coding-Agent Sessions to Test Whether Faster LLM Inference Actually Helps

Everyone is competing on tokens per second.

But for autonomous coding agents, I think the more useful question is:

Does faster inference actually help you ship more?

I got early access to Xiaomi MiMo-V2.5-Pro-UltraSpeed and ran it through the same autonomous coding workflow I had already been using with standard MiMo-V2.5-Pro.

This was not a synthetic prompt benchmark. The agent worked on a real production codebase: reading files, planning changes, writing code, running builds, debugging failures, and committing working updates.

I compared:

• 62 runs on standard MiMo-V2.5-Pro
• 44 runs on MiMo-V2.5-Pro-UltraSpeed
• Same agent framework
• Same codebase
• Similar production task types
• Fixed agent windows of roughly 30–35 minutes

The practical result

The headline is simple:

UltraSpeed reduced average agent-run time by 37% while producing a similar amount of output on comparable production work.

That matters. But it does not mean that a model capable of 1,000+ tok/s suddenly makes an agent 10× more productive.

Why 1,000 tok/s does not become 1,000 tok/s in an agent

In isolation, UltraSpeed can generate extremely quickly. But generation is only one part of an agent loop.

A real coding agent also spends time on:

1. Reading context and prior tool output
2. Planning the next action
3. Generating a response or code change
4. Writing files
5. Running commands, builds, and tests
6. Reading failures and iterating

In my UltraSpeed sessions, a typical run had around 60 turns and generated roughly 397 output tokens per turn.

At 1,000 tok/s, that generation phase is only around 0.4 seconds.

The rest of the turn is context processing, tool execution, planning, and waiting on the environment.

That is why median end-to-end throughput came out at 95 tok/s, rather than anywhere near 1,000 tok/s.

For interactive chat, raw generation speed can dominate the experience.

For autonomous coding agents, it is one part of a larger system.

Where faster inference did help

The gains were still meaningful.

Faster time-to-first-token

On cached contexts, UltraSpeed often started responding in 2–3 seconds instead of around 3–5 seconds.

That does not sound dramatic in one interaction. Across 60+ turns, it compounds.

Better performance on long code-heavy outputs

The biggest gains showed up when the agent generated larger code blocks. UltraSpeed’s P90 effective throughput was 147 tok/s versus 63 tok/s on standard Pro.

That makes individual implementation steps feel materially faster.

More useful work inside fixed windows

This was the outcome I cared about most.

In a fixed 30-minute window, the faster setup usually completed around 5–6 runs instead of 3–4.

That is a much more useful metric than a headline tokens-per-second number.

The trade-off: speed costs more

UltraSpeed was not free performance.

Average cost per run was higher:

• Standard Pro: $2.92/run
• UltraSpeed: $4.19/run

So the decision depends on what constrains you.

Use the faster model when:

• You run fixed-duration agent windows
• You care about CI/CD turnaround
• You are operating a multi-step autonomous workflow
• Developer time is more valuable than model spend

Use the cheaper model when:

• You are not time-constrained
• You care mostly about minimizing spend per run
• A few extra minutes per task do not matter

My takeaway for agent builders

Raw tok/s is not useless, but it is often a marketing metric before it is a productivity metric.

For agentic coding, I would track:

• completed runs per hour
• useful commits per session
• wall-clock time to successful completion
• cost per completed run
• tool execution bottlenecks
• cache hit rate and prefill behaviour

The question is not:

How fast can the model emit tokens?

It is:

How many useful things can the whole system finish per hour?

In this workflow, faster inference helped a lot. Just not in the simplistic 10× way the raw speed number might imply.

I published the full write-up with methodology, limitations, and the technical details behind UltraSpeed here:

MiMo UltraSpeed for Agentic Coding: 106 Sessions Tested

Disclosure: Xiaomi provided early access to MiMo UltraSpeed for testing. The workflow, measurements, analysis, and conclusions are my own.