I really don't mind too much if you downvote those who are milking Steem and trying to benefit from Hive too. That is your right as a stakeholder. There are some special cases you really deserve this, but then some others are still sitting on the fence. Any downvotes could be weighted according to some algorithm. If more rewards go to those who dedicate their efforts to Hive then we probably benefit overall.
In this fairly small community it is easy to have a lot of influence, so you may get some flack, but life will go on.