Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
vinceguidry
on June 1, 2018
|
parent
|
context
|
favorite
| on:
Bypassing website anti-scraping protections
Google's web scrapers obey robots.txt, you can stop Google from crawling your website if you want. Google doesn't want you crawling their website.
That word, I don't think it means what you think it means.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
That word, I don't think it means what you think it means.