Absolutely needed: to get high efficiency for this beast … as it gets better, we’ll become too dependent.
“all of this growth is for a new technology that’s still finding its footing, and in many applications—education, medical advice, legal analysis—might be the wrong tool for the job,”
The issue this time around is infrastructure. The current AI Summer depends on massive datacenters with equally massive electrical needs. If companies can’t monetize that enough, they’ll pull the plug and none of this will be available to general public anymore.
This system can go backwards. Yes, the R&D will still be there after the AI Winter cycle hits, but none of the infrastructure.
We’ll still have models like deepseek, and (hopefully) discount used server hardware
Local models aren’t going anywhere. You can hoard them if you’re worried. People freely tweak them to do some weird shit. There’s whole independent websites for sharing these tweaks.
The only infrastructure that’s at risk is the part where corporations burn billions of dollars hoping the really really big models will turn a profit somehow. The power and capability of small models is basically whatever the big boys did a year ago - and a ton of money has been spent on improving the efficiency of training. Tightening the belt is not going to make things worse. It’s certainly not going to make this whole new kind of software un-happen.
Local models aren’t what’s driving the current AI Summer. No billionaire will be quoted in the New York Times saying how great local models are.
Why on Earth is that your metric?
The money addicts desperately want big models to win because that means less competition and more subscription revenue. But that’s three delusions in a row. Mainframes are basically never a sensible business model. Models costing mere tens of millions already challenge billion-dollar projects. And as you say, infrastructure is expensive, so most people want software that runs within arm’s reach. There’s every reason these cloud services bleed money.
Local models are what’s making all the weird shit you see online. Certainly the porn is not made by typing stuff into Facebook. And again: whatever cutting-edge advances happen in moonshot-priced, datacenter-sized models, they’re soon replicated on any recent desktop. If that same blank-check experimentation had only bothered with small models, do you think they’d be further behind or further ahead?
What’s driving this AI Summer is a horde of ultranerds, funded by these rich idiots lying to each other. A lot of them will still be doing this if the money dries up. A lot of them already do it for free. See aforementioned tweaking and sharing. If that has to expand to training new models from scratch, expect them to get even smaller, and rely on experimentation more than brute force. Like BitNet reducing all weights to trinary. If it still takes a mountain of GPUs… well, SETI@home couldn’t promise people interactive anime waifus.