The world’s top two AI startups are ignoring requests by media publishers to stop scraping their web content for free model training data, Business Insider has learned.
OpenAI and Anthropic have been found to be either ignoring or circumventing an established web rule, called robots.txt, that prevents automated scraping of websites.
TollBit, a startup aiming to broker paid licensing deals between publishers and AI companies, found several AI companies are acting in this way and informed certain large publishers in a Friday letter, which was reported earlier by Reuters. The letter did not include the names of any of the AI companies accused of skirting the rule.
OpenAI and Anthropic have stated publicly that they respect robots.txt and blocks to their specific web crawlers, GPTBot and ClaudeBot.
However, according to TollBit’s findings, such blocks are not being respected, as claimed. AI companies, including OpenAI and Anthropic, are simply choosing to “bypass” robots.txt in order to retrieve or scrape all of the content from a given website or page.
A spokeswoman for OpenAI declined to comment beyond pointing BI to a corporate blogpost from May, in which the company says it takes web crawler permissions “into account each time we train a new model.” A spokesperson for Anthropic did not respond to emails seeking comment.
Robots.txt is a single bit of code that’s been used since the late 1990s as a way for websites to tell bot crawlers they don’t want their data scraped and collected. It was widely accepted as one of the unofficial rules supporting the web.
The game plan is to scrape, store and utilise as much data as possible regardless of conventions, best practice, license agreements etc until specifically regulated to stop.
At that point, a few early companies will have used vast swathes of data that any newly established company is banned from also using
And they will be “unable” to purge it.
Hoping the EU drops GDPR 2 requiring them to delete the entire model if it infringes or something.
Expecting the US to meaningfully regulate US companies is like expecting…
You know what, even including physical impossibilities, I’m struggling to think of anything less likely
I’m in the US so yeah…. Even if the current of future GDPR requires deletion I guarantee it’ll still be used in the US. I have no faith that any US company will follow rules like that. Any fines are just looked at as the cost of doing business.
Or they’ll “purge” it and somehow the canaries will end up in the model anyway
It’s like weapons testing. You only move to ban testing after you’ve developed it yourself.
Whatever happened to those “nightshade” images that poison the model?
You mean that work that took open source software, closed sourced it and refused to release the source code and the poisoning only worked against one specific open source model (stable diffusion)? I don’t think that’s going to come riding to anyone’s rescue.
They only kinda work but more importantly they need mass adoption to actually poison training data. Most people aren’t going to add another step to their posts so probably the only way to mass adopt it is to have platforms automatically poison uploaded images. I wonder if reposts on a platform like that would start to have noticable artifacts in the images like jpeg but different
Same approach as all the other ‘disruptive’ new companies that ignore industry standards, rules, and laws.
I’d say they are pushing for regulations behind the scene because they know it gives them an instant monopoly.
They are already pass the door, they can afford to shut it behind them to own the room. Having to send checks to websites like Reddit and Getty in the future is a small price to pay.