Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Faroo Employs P2P Technology to Search and Index the Web (startupmeme.com)
4 points by usablecontent on May 14, 2007 | hide | past | favorite | 1 comment


I wish 'em luck but distributed searching is a tarpit with four fundamental problems.

One is speed: a search may touch on many 'shards' that then have to be combined and ranked somewhere. That's hard enough to do when you have 100K machines linked with 100Mbps up and down.

The second is honesty: search ranking is a high-value target. It's ripe for a takeover by zombies, even if each node's keyspace is assigned by some trusted authority.

The third is coordinating a web-wide crawler: how to avoid floods, duplicate work, spammy submissions, etc.

The fourth is most important: the system has to work much better than Google for people to care enough to use it. Simply being P2P (i.e. cheaper for the operator) is not enough.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: