The issue with this is that not even a majority must misuse the downvote instrument to cause frustration. If you're targeted because of affiliation with another project, token or initiative it won't help you that the majority isn't misusing the instrument.
I think a fair solution would be to make mandatory that only accounts that actively post content or comment on content can downvote.
The style in which the downvotes are implemented right now is a standing invitation to misuse imo.
Cheers!
Lucky