On the Cheeky Pint podcast, Sam Altman pushed back on people who think open-source models mean they "are not going to need all the stuff in the data center." In his view, "it's quite terrible to have everybody have a bunch of the world's limited supply of DRAM sitting in their own computer. And be used 5% of the time." He later said on X that the comments were taken out of context and that he wants people to have "so much RAM."
I don't think local models replace data centers, and I don't think you have to pick one. Frontier models belong in the cloud. Batching thousands of requests onto expensive accelerators is hard to beat, and the hardest reasoning work needs that scale. Small models belong on the hardware you already own. The right model for the right job.
Altman's argument treats every AI task as a frontier task. Most enterprise work is not. You do not need a frontier model to classify an email.
Wrong Model, Wrong Math
In The Micro-Agent Assembly Line, I argued that enterprise AI should be built from small, discrete agents, each doing one narrow job: classify, validate, extract, route. A 9B model can do those jobs today, in 8GB of VRAM.
Run that math instead of Altman's. A small model needs a few gigabytes, not a rack. The memory is already in the machine, and it is already paid for. The "5% utilization" complaint applies to the DRAM. It says nothing about whether a small model fits in it, and it does.
The fleet is large. TrendForce put the average laptop shipped in 2024 at 11.8 GB of DRAM, and PassMark's survey shows roughly two thirds of Windows PCs with 16 GB or more. Most of those machines can host a small model. Few can host a frontier one. That is the point: pair the hardware you have with the size of model it can run.
Idle is also not the same as scarce. Your CPU and SSD sit idle too, but nobody argues they should live in a data center. The question is whether a task fits the silicon, not whether the silicon is busy.
You Are Carrying a Supercomputer
Most people have no feel for how much compute they own, so here is a comparison.
In 1985 the Cray-2 was the fastest computer on earth, at a peak of 1.9 GFLOPS. It stood nearly 4 feet tall, weighed 5,500 pounds, and drew 150 to 200 kilowatts. Sixteen years earlier, the Apollo 11 guidance computer peaked at 12,250 FLOPS, and it got three people to the moon and back.
Now look at your pocket. An iPhone 17 is estimated at about 2,000 GFLOPS, more than 1,000 times the Cray-2's headline number.
Some ways to picture it:
-
Your phone out-computes the machine that was the pride of NASA and the Department of Energy, and it charges from a wall outlet.
-
It would take roughly 160 million Apollo guidance computers to match it.
-
The Cray-2 ran on 150 kilowatts. The phone charges at about 40 watts.
-
Your work laptop spends its day on spreadsheets and Slack, a supercomputer waiting for a job.
A fair caveat: these are peak theoretical numbers, they compare different kinds of math, and a phone cannot sustain its peak on battery. The Cray-2 also did things a phone does not. But the order of magnitude is real, and most people have never thought about it. We keep treating the machine on the desk as a thin terminal for the cloud. It is a supercomputer that happens to run email.
Google Already Made the Bet
If you doubt that small local models are here, look at what Google did in your browser. In May, security
researcher Alexander Hanff reported that Chrome had written a roughly 4 GB
file called weights.bin to users' machines. It holds the weights for Gemini Nano, and he says
Chrome re-downloads it when users delete it. Snopes rated the claim
mostly true, and found the model on half the devices it checked.
The consent complaint is fair. A multi-gigabyte download with no prompt is bad manners, and Google only began rolling out a toggle under Settings, System in February, which Hanff's original post did not mention. But the story has a second half. As Ars Technica pointed out, Chrome has shipped this model since 2024. It powers scam detection and writing help, and Google says it does so without sending your data to the cloud.
Read that as a proof point. The company with the largest data centers on earth looked at the same DRAM trade-off Altman describes and decided a small model on your machine was worth 4 GB of your disk. Google picked local for narrow jobs: scam detection, rewriting, summarizing. It did not use a frontier model, and it did not send those jobs to a data center. That is the micro-agent argument, running on a very large fleet.
It also shows the limit. Chrome only installs the model on machines that meet its hardware requirements, and Snopes found it on half the devices it checked. Small local models are practical on the hardware people already own, and Google is betting on that.
Distributed Computing Already Worked
Folding@home passed exascale by harnessing spare cycles on consumer machines. Its work units were small, independent, and tolerant of delay. That is the profile of a micro-agent task.
It is not the profile of a giant model. Sharding one huge model across laptops over a network is slow, because inference is bound by memory bandwidth. So I am not proposing a grid that replaces the cloud. The cloud keeps the big models and the hard problems. Small, stateless tasks run where the hardware already sits.
The Enterprise Inference Pool
Here is the idea. A company has hundreds or thousands of workstations. Most are idle most of the day. Treat them as a pool of workers for small-agent tasks.
The architecture is simple:
-
A lightweight agent on each machine. It reports CPU/GPU load, free RAM, and readiness, and holds a small model.
-
An internal router. It takes requests over HTTP or gRPC and sends each one to a capable, idle machine. This is a load balancer, and the pattern is decades old.
-
Stateless tasks. Classification, extraction, validation, and tagging need no memory of past requests, so any machine in the pool can answer. If one drops off, the router retries on another.
This fits the assembly line. The queue, the classifier, and the deterministic router from my earlier post run as the same pipeline. The classifier step simply runs on a pooled workstation instead of a dedicated server. The email example is one representative case. Log triage, document ingestion, ticket sorting, and code review fit the same mold.
Three benefits:
-
Security. Data stays inside the firewall. A narrow worker with one permitted function also shrinks the prompt-injection blast radius, as covered in the earlier post.
-
Cost. The hardware is already bought. You skip per-token fees for steps that never needed a frontier model.
-
Scale. Capacity grows with headcount, within limits. Every new hire adds a node.
The Night Shift
The pool gets more useful after hours. Most workstations sit untouched from evening to morning, and the person who owns the memory is asleep. Nothing needs to yield to anyone at the keyboard, and the whole machine is free, not just the idle slice.
Every company already runs work this way. Somewhere there is a server where a cron job wakes up at 2 a.m. to rotate logs, build reports, or reconcile a ledger. Much of enterprise work is batch work that nobody needs before morning:
-
Classifying and tagging a day's worth of tickets, emails, or documents
-
Extracting fields from invoices and contracts
-
Summarizing call transcripts and chat logs
-
Triage on the day's logs and alerts
-
Validating and cleaning data before the next day's reports
These are the same narrow jobs as the assembly line, and none of them is latency sensitive. A scheduler queues the work during the day. At night it hands the queue to the pool, and by morning the results are waiting. A night shift of a thousand workstations, each with a small model loaded, gets through a large backlog without a new GPU purchase.
This also eases the memory argument. During business hours a workstation's RAM belongs to its user. Overnight, the same RAM belongs to the company's batch queue. The scarce resource gets used when it would otherwise sit idle, which is what Altman says he wants.
The Right Model for the Right Job
The tools exist. Ollama, llama.cpp, and LM Studio make running small models easy.
Altman is right that DRAM is scarce, and that is an argument for spending it well. Send the hard reasoning to the data center. Keep the classification, extraction, and routing on the machines already sitting on your desks. Both have a place, and the memory we already own should not sit idle while we pay frontier prices for small jobs.
P.S. AI Disclosure, Google's Gemini was used to put me into the Spaceballs 1987 movie
