I don't think someone simply voting against the trend should be penalized any more than the cost to their voting power to vote in the first place.
The advantage of hidden votes is that during a round, there is no trend established against which a voter would be contrarian.