Conversation

Notices

1. Foone🏳️‍⚧️ (foone@digipres.club)'s status on Tuesday, 30-Jul-2024 05:57:53 JST Foone🏳️‍⚧️

   One of the downstream effects of the AI boom that I hadn't really thought about is that sites are starting to introduce anti-crawling tech as standard, because they don't want all their bandwidth going to training chatgpt7.

   This is going to make life harder for people who run benevolent scrapers. How many sites are going to be blocking the wayback machine now, just because it looks too much like an AI scraperbot?

   • Foone🏳️‍⚧️ (foone@digipres.club)'s status on Tuesday, 30-Jul-2024 05:59:25 JST Foone🏳️‍⚧️

     in reply to

     And there's plenty of sites I've written my own scrapers for, in other to ensure the site doesn't just die. Doing that is going to become a lot harder if they're already worried about scrapers

   • vxo (vxo@digipres.club)'s status on Wednesday, 31-Jul-2024 00:09:59 JST vxo

     in reply to

     @foone yeah I gave this some thought-- like there's a Wordpress plugin I use on my site that adds a bunch of specific "User-agent: ai-hooverbot-v69
     Disallow: /"
     to robots.txt, which at least keeps the well behaved training crawlers from beating it to death.

     Of course I have seen evidence of non well behaving ones still coming through as well as desirable things like search spiders and archive bots. I don't want to block those inadvertently or create any user annoyance