Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

In the specific case of Google would there be any additional traffic that isn't just the normal googlebot? I can't imagine they would bother crawling twice for every site on the internet.


There are about a dozen Google crawlers that can hit your website for different reasons:

https://developers.google.com/search/docs/crawling-indexing/...

Google-Extended is what is associated with AI crawling, but GoogleBot also crawls to produce AI overviews in addition to indexing your website in Google search.

While the number of crawlers and their overlapping responsibilities makes it difficult to know which ones you can safely block, I should also say that pure AI company bots behave 1000x worse than Google crawlers when it comes to not flooding your site with scraping requests.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: