shish_mish@lemmy.world to Technology@lemmy.worldEnglish · 1 year agoGoogle Is the Only Search Engine That Works on Reddit Now Thanks to AI Dealwww.404media.coexternal-linkmessage-square30linkfedilinkarrow-up177arrow-down12cross-posted to: technology@beehaw.orgtechnology@lemmy.world
arrow-up175arrow-down1external-linkGoogle Is the Only Search Engine That Works on Reddit Now Thanks to AI Dealwww.404media.coshish_mish@lemmy.world to Technology@lemmy.worldEnglish · 1 year agomessage-square30linkfedilinkcross-posted to: technology@beehaw.orgtechnology@lemmy.world
minus-squareBrewchin@lemmy.worldlinkfedilinkEnglisharrow-up30·1 year agoParts of the Internet now only searchable on specific sites now? What next - charging a monthly subscription to use Google? This needs to be regulated before the Internet becomes like streaming TV.
minus-squaretal@lemmy.todaylinkfedilinkEnglisharrow-up16arrow-down1·1 year agoRobots.txt has been around for a long time, and all the major search engines will honor it. Not having a full index of the Web is the norm. That isn’t to say that the practice of signing agreements isn’t potentially a concern.
minus-squarereddig33@lemmy.worldlinkfedilinkEnglisharrow-up2·1 year agoWhat isn’t the norm is to serve one robots.txt to one company, and a different robots.txt to everyone else. Which is what Reddit is doing here.
minus-squareDominusOfMegadeus@sh.itjust.workslinkfedilinkEnglisharrow-up1·1 year agoWhat is robots.txt and how is one supposed to utilize it?
Parts of the Internet now only searchable on specific sites now? What next - charging a monthly subscription to use Google?
This needs to be regulated before the Internet becomes like streaming TV.
Robots.txt has been around for a long time, and all the major search engines will honor it. Not having a full index of the Web is the norm.
That isn’t to say that the practice of signing agreements isn’t potentially a concern.
What isn’t the norm is to serve one robots.txt to one company, and a different robots.txt to everyone else. Which is what Reddit is doing here.
What is robots.txt and how is one supposed to utilize it?