If not, a simple robots.txt would fix the crawler-traffic problem without prejudicially banning any upstart search engines:
User-Agent: * Disallow: /x?
If not, a simple robots.txt would fix the crawler-traffic problem without prejudicially banning any upstart search engines: