Absolutely needed: to get high efficiency for this beast … as it gets better, we’ll become too dependent.
“all of this growth is for a new technology that’s still finding its footing, and in many applications—education, medical advice, legal analysis—might be the wrong tool for the job,”
Local models aren’t what’s driving the current AI Summer. No billionaire will be quoted in the New York Times saying how great local models are.
Why on Earth is that your metric?
The money addicts desperately want big models to win because that means less competition and more subscription revenue. But that’s three delusions in a row. Mainframes are basically never a sensible business model. Models costing mere tens of millions already challenge billion-dollar projects. And as you say, infrastructure is expensive, so most people want software that runs within arm’s reach. There’s every reason these cloud services bleed money.
Local models are what’s making all the weird shit you see online. Certainly the porn is not made by typing stuff into Facebook. And again: whatever cutting-edge advances happen in moonshot-priced, datacenter-sized models, they’re soon replicated on any recent desktop. If that same blank-check experimentation had only bothered with small models, do you think they’d be further behind or further ahead?
What’s driving this AI Summer is a horde of ultranerds, funded by these rich idiots lying to each other. A lot of them will still be doing this if the money dries up. A lot of them already do it for free. See aforementioned tweaking and sharing. If that has to expand to training new models from scratch, expect them to get even smaller, and rely on experimentation more than brute force. Like BitNet reducing all weights to trinary. If it still takes a mountain of GPUs… well, SETI@home couldn’t promise people interactive anime waifus.