I see your point. Yes, sometimes this may be a problem.
But in my case I am not concerned at all. And here is why:
My POB curation downvotes comes from this modest 52.337 reputation account. With only 128HP.
So only those with <52 rep may suffer , and suffer in an EXTREMELY low amounts.
Reputation decreases depending on the power ( vests) downvoter has. My hive power
128 HP even at a full swing brings ( or takes away ) only $0.00172 .
O doubt that even the most vulnerable REP levels of 25-40 would go down more than -0.01 (30>29.99) from my DV.