Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Is there a way to block only the useless calls? The x that fail? Or to robots.txt anything that isn't main story pages so it wouldn't follow every individual comment link?

It might be good to still be indexed on the primary single page for each discussion... Then again I only use google, so it doesn't bother me any.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: