tekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 8 days agorobots.txt is a suggestionusenet.lolimagemessage-square25fedilinkarrow-up1619
arrow-up1619imagerobots.txt is a suggestionusenet.loltekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 8 days agomessage-square25fedilink
minus-squareekZepp@lemmy.worldlinkfedilinkEnglisharrow-up18·8 days agoAll this robot.txt stuff “perplex” me.
All this robot.txt stuff “perplex” me.