The Local Stack Is Shipping Decision Models Now

The next important local model may not be the one that writes the best paragraph.

It may be the small model that decides which model should get the job.

On October 9, developer Jun Kim announced oMLX 0.7.1.dev1, an alleged development release of the Apple Silicon inference engine. The post says the release adds decision models, batched prefill for concurrent requests, a refreshed web UI, and faster support for several model families.

The same post claims a Qwen3.6 35B A3B decode improvement from 156 to 205 tokens per second on an M5 Max, and a GLM 5.3 Flash improvement from 31 to 40.6 tokens per second on an M3 Ultra. Those are claims from the release announcement, not an independently reproduced benchmark in this article.

The interesting part is not the speedup by itself. It is the decision-model feature sitting beside it.

A local runtime is becoming a control loop

A conventional local setup looks like this:

request -> one model -> response

A routed setup looks more like this:

request
   |
   v
decision model -> selected specialist -> response
       |                 |
       +-- cost, latency, context, capability

That extra step changes what the runtime is responsible for. It must classify the request, choose a worker, preserve the right context, and recover when the chosen model is a bad fit.

The decision model does not need to be the smartest model in the system. It needs to be predictable enough to make a useful choice at low cost.

That is a different optimization target from chat quality.

The model is not the product boundary

Local AI discussions still tend to compare models as if every request goes to the same endpoint. Bigger context, higher benchmark scores, and faster decoding dominate the comparison.

An agent runtime has more variables:

Runtime decision	Why it matters
Which model receives the task	A coding worker and a summarizer need different strengths
How much context is sent	Context costs memory and can slow every subsequent token
Whether requests are batched	Concurrency changes throughput and queueing behavior
What happens after a weak result	A router may retry, escalate, or ask for verification
Which model sees sensitive data	Local execution can still leak through the wrong worker or log

The alleged oMLX release is notable because it treats the first row as a runtime capability rather than an application feature bolted on later.

That distinction matters. If routing lives outside the inference engine, every application has to reinvent model selection, fallback policy, observability, and cost accounting. If routing becomes part of the local stack, those decisions can be measured in the same place as decoding and batching.

Decision models are not automatic reliability

There is a tempting story here: add a small decision model and the system becomes smarter for free.

That is not what follows.

A router can fail in ways a single model cannot. It can send a security-sensitive task to a worker with the wrong permissions. It can select a fast model that lacks the required tool-calling behavior. It can route a long context to a worker that truncates the evidence. It can oscillate between specialists and spend more time deciding than doing.

The routing choice therefore needs a verifier. A useful trace should record at least:

request class
selected model and version
context size and truncation
routing confidence or reason
worker result
verification result
fallback or escalation

Without that trace, a faster local stack can hide a worse control loop. The system may look efficient while quietly routing the hard cases into failure.

The benchmark claim is only the first measurement

The oMLX announcement reports large decode gains for specific model and hardware combinations. Those numbers may be useful, but they describe one segment of the loop.

For an agent operator, the important measurements are broader:

1. Decision latency. How long does routing take before the worker starts?
2. Selection quality. How often does the chosen worker solve the task without escalation?
3. Tail behavior. What happens to the slowest requests under concurrency?
4. Context integrity. Does batching or routing change what the worker actually sees?
5. Recovery cost. How much work is repeated after a bad selection?
6. Data locality. Can the policy guarantee that sensitive tasks stay on the approved worker?

A router that adds 20 milliseconds and prevents half of the unnecessary escalations could be valuable. A router that saves 20 milliseconds but sends the wrong task to the wrong model is just a new failure surface.

The useful comparison is not model A versus model B. It is one control loop versus another.

Apple Silicon makes this practical, not magical

The announcement is also a reminder that local inference is constrained by a physical machine. Memory bandwidth, unified memory pressure, prefill behavior, expert loading, and concurrent requests all shape the experience.

A runtime that can batch prefill and select among model variants is responding to those constraints directly. It is trying to use the machine as a system rather than treating inference as an isolated function call.

That does not make local execution automatically private or safe. A local router still needs explicit policy around model files, logs, tools, and network access. “Runs on my Mac” describes placement, not authorization.

What operators should require

If decision models become a normal part of local agent stacks, promotion checks should cover the router as well as the workers:

• Pin the decision model and every worker by exact artifact and runtime version.
• Keep a fixed routing corpus with task class, expected capability, and escalation policy.
• Record the selected worker and the reason for every production request.
• Test ambiguous tasks, prompt injection, long context, and tool errors as routing cases.
• Make sensitive-data policy a hard constraint, not a preference the router can override.
• Require a fallback when the router is uncertain or the worker cannot prove completion.
• Compare end-to-end success, latency, and resource use rather than decode speed alone.

The goal is not to make the router autonomous. It is to make the decision visible and bounded.

The local stack is becoming an agent substrate

The alleged oMLX release may be a small development step, and its benchmark figures still need independent reproduction. But the direction is clear enough to watch.

Local inference is moving from a single model process toward a substrate with batching, routing, specialist workers, and runtime policy. That is the same transition cloud agent platforms made earlier, now arriving on smaller machines.

The model you download will matter. The control loop that decides when and how to use it will matter more.

A local agent is not private because the weights are on disk. It is reliable only when the runtime can explain its choices, contain its workers, and verify the result.

Sources

• Jun Kim’s X announcement for oMLX 0.7.1.dev1 (release features and claimed hardware results)