Public
- Public
- Network
- Groups
- Popular
- People

This picture displays the contents of openai.com's robots.txt file: User-agent: * Allow: / Disallow: /microsoft-for-startups/ Sitemap: https://openai.com/sitemap.xml

Download link

This picture displays the contents of openai.com's robots.txt file: User-agent: * Allow: / Disallow: /microsoft-for-startups/ Sitemap: https://openai.com/sitemap.xml
https://files.mastodon.social/media_attachments/files/112/369/143/942/873/276/original/417a8f675d0971eb.png

Notices where this attachment appears

nixCraft 🐧 (nixcraft@mastodon.social)'s status on Thursday, 02-May-2024 11:32:17 JST nixCraft 🐧

In a deliciously ironic twist, OpenAI's website forbids scraping... lol.

In conversation about 7 months ago from mastodon.social permalink