... we have things called downvote trails. People trust a single entity with their downvoting power, and that entity uses that DV power as they see fit.
Reworking the delegation of downvotes seems a good area for discussion and a great place to start to address the issue.
Indeed, I remember one time I was getting notifications that I was being downvoted, hundreds of accounts and it really felt horrible.
The only thing worse than getting no votes. 😎