AYO provides customized AI solution for comapnies to improve their process. We take inspiration in the latest research in machine learning : deep learning, probabilistic models, neuroscience and physics.
This is a little bit of a wall of text but I hope it will be a nice sample for such a setup.
For me, every single request is coming from a different random IP address (different ASN, different country, etc.) using a random plausible User-Agent. There’s no way to discern them as bot requests. And since every request is coming from a random source, things like fail2ban or CrowdSec don’t really work. You can block a few major ASNs but there will still be lots of requests making it through.
And these things work by going through a list of URLs previously collected. About a week ago I’ve configured a redirect for ALL requests to my personal Gitea instance to a zip bomb. It’s been 7 days now where the website was inaccessible and I still see requests for deep links, e.g. specific commits or lists of pull requests. These things are relentless. That’s also one of the arguments against Anubis - those scrapers just don’t care.
I’ve found some comments that at least a few ISPs check AbuseIPDB.com and cancel their customer’s contracts if their IP appears in that database. So, I’ve now resorted to occasionally collect those IPs hammering my poor Raspberry Pi and reporting them to the AbuseIPDB. If people are unknowingly running a “residential proxy” (why don’t we call them “open proxies” anymore?) somebody needs to let them know, I guess.
Isn’t the use of different random IP per request by bots a stronger arguments in favor of the default Anubis behavior? This would mean a lot of POW to be done by the bots.
The deep link accesses is worrying indeed and Anubis is not a silver bullet.
I also agree with the use of a known abusive IP database.
The bots don’t care about Anubis, though. Yes, their request will take a bit longer, but since each request is coming from a different device, they don’t care. They’ll just do the pow in parallel. And in the meantime you’re annoying every actual human visiting your site.
For me, every single request is coming from a different random IP address (different ASN, different country, etc.) using a random plausible User-Agent. There’s no way to discern them as bot requests. And since every request is coming from a random source, things like fail2ban or CrowdSec don’t really work. You can block a few major ASNs but there will still be lots of requests making it through.
And these things work by going through a list of URLs previously collected. About a week ago I’ve configured a redirect for ALL requests to my personal Gitea instance to a zip bomb. It’s been 7 days now where the website was inaccessible and I still see requests for deep links, e.g. specific commits or lists of pull requests. These things are relentless. That’s also one of the arguments against Anubis - those scrapers just don’t care.
I’ve found some comments that at least a few ISPs check AbuseIPDB.com and cancel their customer’s contracts if their IP appears in that database. So, I’ve now resorted to occasionally collect those IPs hammering my poor Raspberry Pi and reporting them to the AbuseIPDB. If people are unknowingly running a “residential proxy” (why don’t we call them “open proxies” anymore?) somebody needs to let them know, I guess.
Isn’t the use of different random IP per request by bots a stronger arguments in favor of the default Anubis behavior? This would mean a lot of POW to be done by the bots.
The deep link accesses is worrying indeed and Anubis is not a silver bullet.
I also agree with the use of a known abusive IP database.
The bots don’t care about Anubis, though. Yes, their request will take a bit longer, but since each request is coming from a different device, they don’t care. They’ll just do the pow in parallel. And in the meantime you’re annoying every actual human visiting your site.