I think it would be interesting to think about a correlation between the vote and the comments. For example the vote should systematically be followed by a commentare to be validated. In this case, it would correlate the power of the vote with the quality of the comment.
That is certainly a well made point. I agree, that there probably should be some sort of requirement for vote validation to counter excessive downvoting especially since it affects both visibility and reputation.