@mischievoustomato @Cyrillic @amalthea It's only there to control crawling (recursively following links), FBA simply looks up the predefined API endpoints. Same reason why web.archive.org respects robots.txt but archive.fo (which only saves single pages upon user request) doesn't.