Alice asks an assistant to rename a file. Bob asks it to prove a difficult theorem. Today both requests may travel to the same giant cloud model. Jon Saad-Falcon’s question is deceptively simple: why send Alice’s easy job across a continent if the laptop on her desk can answer it correctly?
Tokens per second measures motion. Intelligence per watt measures useful work: did this model-accelerator pair answer correctly, and how much power did it need?
Why the old efficiency metrics are incomplete
Throughput rewards systems that emit many tokens. Latency rewards systems that respond quickly. Energy per token rewards cheap generation. None asks whether the answer is good. A tiny model producing nonsense at enormous speed is not an efficient intelligence system.
The paper defines Intelligence per Watt (IPW) as task accuracy divided by power. It evaluates the whole pair: a model and the accelerator running it. This matters because the same model can behave very differently on an Apple SoC, a consumer GPU, and a cloud accelerator, while a stronger model may justify extra watts by solving many more queries.
The experiment
In the talk, Jon describes more than 20 models across Gemma, GPT-OSS, Qwen, and IBM Granite, spanning roughly 1–200 billion parameters and including dense and mixture-of-experts designs. The hardware sweep covers recent accelerators from Apple, NVIDIA, AMD, and SambaNova. Tasks include chat, reasoning, agentic, and coding work; measurements include accuracy, latency, energy, power, and compute.
Three results, carefully read
- Coverage: up to 88.7% of evaluated queries could be routed to local accelerators running local open models.
- Progress: intelligence per watt improved about 3× over two years through combined model and accelerator gains.
- Total energy: intelligence per joule improved about 18× in roughly 16 months, driven primarily by better accelerators.
- Routing: a perfect router could place roughly 80–90% of current queries locally; imperfect routers could save about 50–70% of energy, compute, and dollar cost in the study’s analysis.
A router, not a revolution
The practical architecture is hybrid: use both local and cloud resources and route according to capability. Jon does not prescribe a specific privacy classifier or list of tasks in this talk; his focus is the resource redistribution made possible by better models and local hardware.
| Question | Useful metric | What it misses |
|---|---|---|
| How fast? | tokens / second | answer quality and energy |
| How much energy? | joules / token | whether tokens are correct |
| How capable per power? | intelligence / watt | embodied hardware and total lifecycle cost |
Power is not energy
Watts are an instantaneous rate; joules are total energy. A 100-watt system finishing in one second may use less energy than a 20-watt system running for ten. IPW captures the power-constrained-device question, but deployments should inspect both IPW and intelligence per joule, plus latency and memory. No single scalar decides the fleet.
The caveats
Jon is explicit that local accelerators do not yet beat the strongest data-center systems. Batching, kernel work, and quantization let cloud hardware amortize its cost across users. In his comparison, an Apple M4 Max trails an NVIDIA B200 and falls further behind the inference-specialized SambaNova SN40L on intelligence per watt and per joule.
The takeaway
The paper replaces “local versus cloud” with a routing decision. The right question is not which side wins universally; it is what is the smallest model-hardware pair that can answer this query well, now, within the power budget? Once systems can measure that, millions of easy requests can stop taking the expensive road.
Why Jon calls this the mainframe era
Early mainframes concentrated scarce computing in centralized rooms. Users submitted work to a remote machine because they could not own equivalent capability. Jon sees a resemblance in modern AI: frontier models run in enormous data centers, and even ordinary requests travel to them over a network.
The analogy is not exact. Today’s cloud is globally distributed, and local computers already perform substantial work. Jon uses “mainframe era” to highlight a routing default: we often send every language-model task to centralized frontier infrastructure, even when a smaller model on a nearby device may be capable enough.
The demand problem behind the metric
Jon describes AI infrastructure investment at a scale that requires new power, land, water, accelerators, and transmission. Whether every macroeconomic estimate remains current is less important than the engineering fact: inference demand is growing fast enough that efficiency changes the number of data centers society must build.
If 80% of requests genuinely require frontier capability, central expansion is unavoidable. If many are file organization, formatting, unit-test generation, or routine chat, then routing them to existing consumer hardware could satisfy demand without proportionally expanding centralized capacity.
“Intelligence” is task capability, not a mystical quantity
The numerator in intelligence per watt must be operational. For a coding task, capability might be passing tests. For question answering, it might be a judged correct response. For an agent, it might be completing a workflow. Different tasks require different evaluation suites.
This means IPW is not one universal number printed on a model card. A model can be efficient at summarization and inefficient at difficult mathematics. A hardware pair can excel on short prompts but struggle with long contexts. The metric should be conditioned on the workload people actually send.
The denominator: power versus energy
A watt measures a rate of energy use. A joule measures total energy. A low-power device can still consume more total energy if it runs much longer. Conversely, a high-power accelerator can finish quickly enough to use fewer joules for the complete answer.
IPW is helpful when the deployment has a power envelope: a laptop, workstation circuit, or data-center rack can deliver only so many watts. Intelligence per joule answers the total-energy question. Jon reports both because optimizing one does not guarantee the other.
The model and accelerator must be evaluated as a pair
A parameter count alone does not reveal efficiency. Quantization changes memory use and arithmetic. Software kernels determine whether the accelerator’s units stay busy. Unified memory on an Apple system behaves differently from discrete GPU memory. A specialized inference engine can exploit a model differently from a general GPU.
Therefore the unit under evaluation is not “Qwen” or “M4 Max” independently. It is a particular model, precision, inference engine, configuration, and accelerator running a defined task distribution.
What the broad sweep includes
Jon describes more than twenty recent local model configurations across Gemma, GPT-OSS, Qwen, and IBM Granite, from small dense models to much larger and mixture-of-experts designs. The accelerator set spans consumer and specialized systems from Apple, NVIDIA, AMD, and SambaNova.
The study looks beyond one academic benchmark. It includes chat, reasoning, coding, and agentic work and records capability alongside latency, power, energy, and compute. That broad design is necessary because an efficient configuration that responds too slowly may not be serviceable, while a fast configuration that gives wrong answers is not intelligent.
How to understand the 88.7% result
Jon says that up to 88.7% of evaluated queries could be routed to local models and accelerators. “Up to” and “evaluated” are essential. This is not a guarantee that any laptop can answer 88.7% of every company’s production traffic. It describes the best coverage found under the study’s models, hardware, tasks, and routing assumptions.
Coverage also does not mean one tiny model answers everything. A local tier can contain several models and devices. The router chooses among them, escalating queries that exceed local capability or practical latency.
Why model and hardware progress compound
Jon reports roughly a threefold improvement in intelligence per watt over two years. Better post-training allows smaller models to solve more tasks. Better quantization fits stronger models into local memory. New accelerators add capacity and improve execution. Each advance amplifies the others.
This compounding is why local inference can improve faster than a hardware-only comparison suggests. If the same task moves from a large model to a better small model and that model runs on a more capable local accelerator, both numerator and denominator move favorably.
The larger intelligence-per-joule gain
The reported eighteenfold gain over roughly sixteen months concerns total energy per completed workload. Jon attributes much of it to accelerator improvements, including greater local memory capacity. More memory can eliminate slow offloading or make a previously impossible model fit entirely on device.
Quantization also affects total runtime. A compressed model moves fewer bytes and may generate faster, reducing both active power duration and energy. The exact gain varies by model quality, kernel support, and hardware precision paths.
Perfect and imperfect routers
A perfect router knows before execution whether a local configuration will answer acceptably. Real routers do not. They estimate difficulty from the prompt, task type, model confidence, historical performance, or a cheap preliminary model.
Jon reports that ideal routing could place roughly 80–90% of the studied workload locally. Even imperfect routers could save approximately 50–70% of energy, compute, and dollar cost. The difference between those ranges is the price of uncertainty.
A concrete hybrid-routing story
Suppose a developer asks an assistant to rename variables across one file. A local coding model attempts the edit and runs tests. If tests pass, the task ends locally. If the repository is large, the tests fail, or the user asks for an architectural redesign, the system escalates context to a cloud model.
This approach uses verification as part of routing. It also avoids making “local or cloud” a permanent user choice. The system can begin with the cheapest capable tier and climb only when evidence says it should.
Why cloud accelerators still lead
Jon explicitly rejects the idea that local hardware is already superior on efficiency. Data-center systems benefit from batching many users, mature kernels, high memory bandwidth, and optimized quantization. Their power draw is high, but utilization can be much higher too.
In his comparisons, Apple’s M4 Max trails NVIDIA’s B200 and trails the inference-specialized SambaNova SN40L further on the reported intelligence-efficiency measures. Local viability and local leadership are different claims.
What local hardware offers besides energy
Although the talk centers on efficiency, local execution can also reduce network dependence and provide predictable access when cloud capacity or connectivity is unavailable. Data can remain on the device for workflows designed that way. Latency avoids a network round trip, though generation itself may still be slower.
These benefits are workload-dependent and should not be used to hide the hardware gap. A privacy-sensitive local task is useful only if the local model is capable enough and the application handles data securely end to end.
The software stack required for hybrid inference
A practical system needs model packaging, memory management, local inference engines, hardware-specific kernels, a router, quality evaluation, cloud fallback, and telemetry. It must decide whether to download a model, keep it warm, or evict it when memory is needed by another application.
Open Jarvis, which Jon mentions near the end, explores operationalizing a personal AI and coding stack on device. The research direction is not merely benchmarking chips; it is building an engine that can use local and cloud resources as one hierarchy.
Breaking the metric down
An aggregate IPW score can conceal where a system succeeds. Jon proposes analysis by task, hardware, deployment, inference engine, kernel, compute, and memory. This decomposition helps builders act. If one configuration loses because of a poor kernel, replacing the model is unnecessary. If it loses on capability, hardware tuning cannot fix the answer quality.
| Question | Measurement | Decision enabled |
|---|---|---|
| Can it solve the task? | task-specific capability | local eligibility |
| Does it respond in time? | latency and throughput | interactive serviceability |
| Can the device sustain it? | intelligence per watt | power-envelope fit |
| What does completion consume? | intelligence per joule | total-energy comparison |
| What if local fails? | router recall and fallback quality | hybrid reliability |
From intelligence to useful work
Jon closes by asking how measured intelligence affects work people value. A model can score well yet produce little economic or social benefit. A useful next metric might count completed tasks, saved human time, or outcomes rather than judged answers alone.
This is difficult because human work is heterogeneous and quality matters. Automating a low-value step is not equivalent to improving a high-stakes decision. Still, the question prevents efficiency research from optimizing tokens that nobody needs.
A responsible interpretation
The paper’s routing result is a reason to build and test hybrid systems, not a reason to shut down cloud inference or claim that every workstation is a frontier model. The best local configuration, workload mix, and acceptable quality threshold will change quickly.
The durable method is to evaluate the complete model-hardware pair on real tasks, measure power and total energy, and route with a verified fallback. As local models and accelerators improve, the boundary can move without redesigning the application around one permanent winner.