Anubis, which block AI scrapers from scraping websites to death, has been downloaded almost 200,000 times.
  • kcweller@feddit.nl
    88·
    2 months ago

    Robots.txt expects that the client is respecting the rules, for instance, marking that they are a scraper.

    AI scrapers don’t respect this trust, and thus robots.txt is meaningless.