Can't some "vote swapping" just be people voting for content of their friends?
It's not all bad. I only pass out votes to stuff I like, from people I trust.
I like posts about politics, not kittens. So I'll mostly only ever be voting for a small number of users.