Reddit.com robots.txt file. Reddit blocked all search engines including bing, DDG, and others, but because of the deal with Google, it is still making money and is the top destination for many Google searches. Here is robots.txt. They are blocking it because AI bots are scrapping and profiting from user-generated content.

Download link

Notices where this attachment appears

nixCraft 🐧 (nixcraft@mastodon.social)'s status on Friday, 26-Jul-2024 19:56:06 JST nixCraft 🐧

Reddit blocked all search engines including bing, DDG, and others, but because of the deal with Google, it is still making money and is the top destination for many Google searches. Here is robots.txt. They are blocking it because AI bots are scrapping and profiting from user-generated content. If you do this you will be dropped from Google immediately but now they have a deal with Google

In conversation about 4 months ago from mastodon.social permalink