If we had to do everything manually, we would lose or require many more resources.
For sure, this is why it has to be automated to some degree at the very least - but as you said, it has to be continually tweaked to both keep up with the abuse as well as make sure it isn't doing more harm than good, the last one is hard to know.
My thought is we can look at things like HP, account age, and rep (both the number and vouching) and require manual reviews of well knwon accounts.
THis is not a bad idea as I know that there are a "few" large accounts that have already been caught being pretty dodgy (to say the least) and there are quite a few of the "returned" who have immediately picked up their autovoter trails again...
No one would be immune to punishment or moderation, but it would prevent an automated mistake.
No one should be immune, no matter how much stake they have :D