I've downvoted probably more than almost anyone else on the entire platform so don't look at me when it comes to people not downvoting.
Probably a good part of the problem is that downvotes are so extraordinarily rare (I had previously heard 0.01% to 0.1% but here we are seeing 0.005% which is really negligible) which turns every single downvote into some sort of man-bites-dog type of story.
If we're able to get downvotes used more frequently by more people then more of them will be about pushing back against reward milking, as is intended.