tekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 6 months agorobots.txt is a suggestionusenet.lolexternal-linkmessage-square25fedilinkarrow-up1617arrow-down13
arrow-up1614arrow-down1external-linkrobots.txt is a suggestionusenet.loltekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 6 months agomessage-square25fedilink
minus-squareAlbum@lemmy.calinkfedilinkarrow-up13·6 months agoSearch engine scrapers index. But that’s a subset of scrapers. There are data scrapers and content scrapers, and these are becoming more prolific as AI takes off and ppl need to feed it data. This post is specifically about AI scrapers.
Search engine scrapers index. But that’s a subset of scrapers.
There are data scrapers and content scrapers, and these are becoming more prolific as AI takes off and ppl need to feed it data.
This post is specifically about AI scrapers.