Downvoting definitely stifles free speech since it robs you of the money the upvoters want you to have. Personally, I have decided to stay away from topics I feel may be controversial and just post my photography and the occasional article. I'm trying to build my reputation and account. I don't need the headaches that these flaggots bring. I have said early on that the downvote has too much power.
While I think you should be able to downvote, I don't think it should take away from the money the upvoters want you to have. I have seen it used just because they feel that you made too much money on the post.
Best of luck to you.
Yeah, you're right it really is punishment and abuse and there should be a way to be able to stop someone from abusing you on your posts. They do this so that people will be disheartened and stop posting.