Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Is there a reason for any "/x?" URLs to be indexed?

If not, a simple robots.txt would fix the crawler-traffic problem without prejudicially banning any upstart search engines:

  User-Agent: *
  Disallow: /x?


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: