That makes sense. I wonder why Amazonbot specifically as a target to spoof. I ho... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		arjie 56 days ago \| parent \| context \| favorite \| on: How I protect my Forgejo instance from AI web craw... That makes sense. I wonder why Amazonbot specifically as a target to spoof. I hoped to get them not stuck using a robots.txt but they refuse to obey it and keep hitting that page with various params. No problem for me, but they are going nowhere.

reconnecting 56 days ago [–]

Perhaps result of A/B testing, because web masters less frequently block them.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact