I agree to the site being invisible to yahoo, msn, ask, etc.
At the same time, please let the newer/upcoming search engines crawl the site, else, google would become IE circa 2000-2004 and there would be very little innovation in search.
This patchwork deny/allow is nearly as bad as banning everyone but Google. Every small site will make the decision of who's young enough to deserve a chance differently -- making only Google the safe choice for searching.
A principle of equal access by any well-behaved search engine is the only policy which supports startups and diversity. The market already has a winner-take-all quality; why make it even worse for competition?
I don't mean it to be a patchwork system, I should clarify, a simple deny Yahoo, MSN works fine with me since Paul is making a business decision. What is not fine with me is allowing only google, and denying everyone else.
Like you, I support startups and want diversity and innovation, which is why I would like search startups to crawl this website.
At the same time, please let the newer/upcoming search engines crawl the site, else, google would become IE circa 2000-2004 and there would be very little innovation in search.
my 2 cents.