I just tend to ignore these people. IMHO most of the comment bots are no better either. Many newbies are suckered into responding and upvoting.
I have been tempted to downvote, but generally don't want to give precious time to it. Perhaps if they are pesistent, yes. I have however found, that after once or twice of me ignoring them, they never come back.
I do upvote comments of value on my posts, so these spam comments end up buried on the bottom looking the rather sad pointless things that they are.
Yes, I ignored them too, but yesterday I just got too annoyed. It does take away voting power, but not that much time.
I upvote and answer all comments on my posts, so I have to read them anyway. I hope that a downvote can scare them off.
I look at upvoting the comments on my post as bringing out the drinks and snacks when visitors come over. If you like your guests, then of course you'd want them to hang around a bit longer. ;-)