@nixCraft I tried to tell everyone that the robots.txt thing wouldn't be respected back when they first started trying to block LLM data crawlers and I got ripped into.
I don't know if there is any truly great solution, but yeah, companies like those are going to just ignore them.