Nuking the Corporate Web's Web-Scraping Robots
cheapskatesguide.org
Many small website owners can defend their sites from web-crawling robots without big tech's help. Here is how.

Lemmy newb here, not sure if this is right for this /c.

An article I found from someone who hosts their own website and micro-social network, and their experience with web-scraping robots who refuse to respect robots.txt, and how they deal with them.

  • splendoruranium@infosec.pubEnglish
    41·
    15 days ago

    They block VPN exit nodes. Why bother hosting a web site if you don’t want anyone to read your content?

    Fuck that noise. My privacy is more important to me than your blog.

    It’s a minimalist private blog that sets no 3rd party cookies and loads no 3rd party resources. I presume that alleviates your concerns? 😜

    • 𝕽𝖚𝖆𝖎𝖉𝖍𝖗𝖎𝖌𝖍English
      6·
      15 days ago

      That’s not what I’m complaining about. I’m unable to access the site because they’re blocking anyone coming through a VPN. I would need to lower my security and turn off my VPN to read their blog. That’s my issue.

      • klu9@lemmy.caOPEnglish
        4·
        14 days ago

        I believe using a CDN would defeat the author’s goal of not being reliant on third-party service providers.