That seems like a decent idea but a bit tedious for the voter. I also like the Stack Overflow model where each downvote costs 1 of your own karma points.