tekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 5 months agorobots.txt is a suggestionusenet.lolexternal-linkmessage-square25fedilinkarrow-up1617arrow-down13
arrow-up1614arrow-down1external-linkrobots.txt is a suggestionusenet.loltekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 5 months agomessage-square25fedilink
minus-squareekZepp@lemmy.worldlinkfedilinkEnglisharrow-up17·5 months agoAll this robot.txt stuff “perplex” me.
All this robot.txt stuff “perplex” me.