One prominent author responds to the revelation that his writing is being used to coach artificial intelligence.
By Stephen King
Non-paywalled link: https://archive.li/8QMmu
One prominent author responds to the revelation that his writing is being used to coach artificial intelligence.
By Stephen King
Non-paywalled link: https://archive.li/8QMmu
LLMs regurgitate their training set. This has been proven many times. In fact from what I’ve seen LLMs are either regurgitating or hallucinating.
Early days.
With great respect I believe that to be a gross simplification of what an LLMs does. There is no training set stored in the LLM, only statistics about what word set is likely to follow what word set. There is not regurgitation of the date - if that was the case, they temperature parameter wouldn’t matter when it very much does.
A slightly compressed JPG of an oil painting is still, at least for purposes of intellectual property rights, not distinct from the original work on canvas. Sufficiently complex and advanced statistics on a work are not substantially different from the work itself. It’s just a different way of storing a meaningful representation.
These LLMs are all more or less black boxes. We really cannot conclusively say one way or another whether they are storing and using the full original work in some form or another. We do know that they can be coaxed into spitting out the original work, though, which sure implies it is in there.
And if the work of a human that needs to be fed is being used by one of these bots – which is pretty much by definition a commercial purpose given that all the relevant bots are operated as such – then that human should be getting paid.
Only very rarely, under extreme cases of overfitting. Overfitting is a failure state that LLM trainers want to avoid anyway, for reasons unrelated to copyright.
There simply isn’t enough space in a LLM’s neural network to be storing actual copies of the training data. It’s impossible, from a data compression perspective, to fit it in there.