When I click downvote, I get a popup, it's really difficult to do it by accident.
I suspect a lot of those people either never downvoted anyone or want to make downvoting as inconvenient as possible for whatever reason.
I don't think you have anything to worry about, maybe a onetime downvote from some jerk, but that is about it. Generally, if your conent is good and you complain about the downvote, someone will give you an upvote to negatate it and warn the flagger. It's only gray-zone activity you need to worry about. In my case, a few of the people I downvoted won't get caught by the cleaners or flaggers because their conent is like a 2/10 and not the 1/10 that the cleaners are after.
I thought there was a bit of a "safety" with the downvote button, but didn't want to accidentally hit it to check. Thanks for confirming that! I'm sure like everything else people will get used to it. I don't imagine it's going to change my user experience much if at all. ;)