Conversation

Notices

1. nixCraft 🐧 (nixcraft@mastodon.social)'s status on Thursday, 02-May-2024 11:32:17 JST nixCraft 🐧

   In a deliciously ironic twist, OpenAI's website forbids scraping... lol.

   • nixCraft 🐧 (nixcraft@mastodon.social)'s status on Thursday, 02-May-2024 11:34:08 JST nixCraft 🐧

     in reply to

     FYI, you can block OpenAI, Google AI and others with robots.txt now https://www.cyberciti.biz/web-developer/block-openai-bard-bing-ai-crawler-bots-using-robots-txt-file/

   • nixCraft 🐧 (nixcraft@mastodon.social)'s status on Thursday, 02-May-2024 11:37:00 JST nixCraft 🐧

     in reply to

     this was original but they recently updated it

   • Jean François Kennedy (jeanfrancoiskennedy@mastodon.social)'s status on Thursday, 02-May-2024 11:44:41 JST Jean François Kennedy

     in reply to

     @nixCraft doesn't this remove search engine referencing all together?

   • Fahri Reza (dozymoe@mastodon.social)'s status on Thursday, 02-May-2024 13:35:02 JST Fahri Reza

     in reply to

     @nixCraft they don't want to accidentally incest other AI scraper

   • Fahri Reza (dozymoe@mastodon.social)'s status on Thursday, 02-May-2024 13:36:30 JST Fahri Reza

     in reply to

     @nixCraft how about X's AI?

   • Erik Jonker (erikjonker@mastodon.social)'s status on Thursday, 02-May-2024 15:12:43 JST Erik Jonker

     in reply to

     @nixCraft ..but good to remember it won't help against scrapers, crawlers who just ignore robots.txt , i don't think a chinese crawler bot will care for example.

   • Frederic Jacobs (fj@mastodon.social)'s status on Thursday, 02-May-2024 17:53:34 JST Frederic Jacobs

     in reply to

     @nixCraft I wish there was the possibility to specify the usage of the scraping rather than the agent.

     https://mastodon.social/@fj/112280775190792281