The best solution I can think of is a very sophisticated rep system that has less influence over upvotes but a big influence on the weight of downvote
I would say, it should have the same influence on the downvotes as well as the upvotes. So people just ciclejerk themselves or their alt account upvote would be worth less, if thats how the downvote is monitored