I always thought that a user's votes should be normalised by something sensible, like total number of votes or time spent online (with sensible scaling). That way opinionated clickers don't have outsize influence.