I cut bot traffic to my site by over 80% with a simple trap. Last month, my hobby website OpenBenches.org transferred more than 2.5TB of data to visitors. As much as I’d like to think the site is enormously popular, it looks like it is just AI bots endlessly crawling the data and slurping up the images. So I’ve built B to be an indiscriminate tool to eradicate these unwanted parasites. It is…
Alliteration always annoys
Here’s the high level description of how it works.
- Create a directory like /honeypot/ on your site
- Put some code in index.php within the directory
- Add the honeypot as an explicit “Disallow” in robots.txt
- Include a hidden link to to the honeypot on every page of your site
- If something accesses that page, write its IP address to your .htaccess
- The .htaccess bans the IP address from ever accessing your site again.
But, I thought that scrapers often used proxy networks so they retrieve through huge amounts of IP’s.
Bots using residential proxies would ban bunch of users forever.
Well, then these proxy networks will be banned. Maybe he should parse the ban lists occasionally if some entries can be compacted down to something/24 or so.
The proxies are implemented by malware running inside TV sets, so the addresses are whatever Comcast or the like hands out. They’re not organized in blocks, and there are millions of them. Typically you’d get just one or a few requests from each address.
Sounds simple and effective, interesting! Is there also an option to send a missile strike to the originating location ?
No you need to compile your own server side missiles, still, but I’m sure somewhere you can find a Linux guide for it.
That really does sound like a great solution - does anyone here know if it could cause any problems?
Says so in the article
