How to use Really Simple Licensing (RSL) to block all AI crawlers - PieFed
join.piefed.social
RSL is a new initiative by a group of big internet publishers that seeks to define the conditions under which AI crawlers can harvest their content. Their guide describes the various ways the content can be made available, including for free or a paid royalty but only by digging deeper into their reference material was […]
  • slate@sh.itjust.works
    2·
    6 days ago

    You could point fail2ban at the access logs and automatically block any ips that are sending a crazy number of requests. Or that are sending bad requests or really however you want to configure it.

    It’s a little trickier for public servers, but I run some private web server stuff and use fail2ban to automatically ban anyone that attempts to access the server through the raw ip or non-recognized hostname. I get like 15-25 hits per day doing that.

    • hendrik@palaver.p3x.deEnglish
      2·
      6 days ago

      Thanks. But I’m not sure if that’s going to help me. What I see in my logs are many different IPs from several /18 networks. It’d take a while to let fail2ban fight such a crawler on an individual address level. Or I go for some nuclear approach, but I’d really like to avoid restricting the open internet even more than it already is. And it’d be hard to come up with a number of allowed requests so my services still work for humans. Me scrolling through PieFed definitely does more requests for a while than one individual crawler IP from Tencent does. Maybe if I find a good replacement for fail2ban which makes tasks like that a bit easier. And it’d better be efficient because fail2ban already consumes hours of CPU time sifting through my logs.

      Calling my server with the IP is handled. I think that just returns a 301 forward to my domain name. I get a lot of exploit scanners via that route, looking for some vulnerable wordpress plugins, phpMyAdmin etc. But they end up on my static website and that’s it.