The system should account for this. This is the internet, people are always awake and using the site somewhere in the world. Maybe measuring page views and adapting the algorithm to give more weight to a vote at times when there are few people on the site would help.