Luu Tuyen@lemmy.world to Technology@lemmy.worldEnglish · 1 year agoTikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAIfortune.comexternal-linkmessage-square86linkfedilinkarrow-up155arrow-down10
arrow-up155arrow-down1external-linkTikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAIfortune.comLuu Tuyen@lemmy.world to Technology@lemmy.worldEnglish · 1 year agomessage-square86linkfedilink
minus-squarejagged_circle@feddit.nllinkfedilinkEnglisharrow-up2·1 year agoI think a common nginx config is to just redirect malicious bots to some well-cached terrabyte file. I think hetzner hosts one iirc
minus-squareSomething Burger 🍔@jlai.lulinkfedilinkEnglisharrow-up3·1 year agohttps://github.com/iamtraction/ZOD 42kB ZIP file which decompresses into 4.5 PB.
I think a common nginx config is to just redirect malicious bots to some well-cached terrabyte file. I think hetzner hosts one iirc
https://github.com/iamtraction/ZOD
42kB ZIP file which decompresses into 4.5 PB.