Not having a SP at stake for downvoting is a mistake. It should absolutely affect your VP.
I was advocating a SP stake... so there would be stake risk, not just a drop in Voting Power... but you would also have to back with some actual SP/Steem. This is in keeping with other consensus protocols that employ a loss of stake for bad behaviour (bad or rogue block production).
The one thing I have noticed and I don't know if it is related or not, but the value of Steem has dropped significantly during this time zero tolerance of bidbots.
I'm not sure that it was only related to that... but if it was, there would be only certain types of accounts that would have been selling... and those are ones that we could do without!