I think the down-votes would be free to a point and not unlimited so down-voting would start to affect mana after a yet to be decided amount of weight. They would still be attached to SP and therefore not equal - same as is the case now. I've not seen any discussion on how rep would be affected - same as now I assume.
There will likely be pockets of abuse.