Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Google's web scrapers obey robots.txt, you can stop Google from crawling your website if you want. Google doesn't want you crawling their website.

That word, I don't think it means what you think it means.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: