Yes, that's a good point - if you reply on one of those comments you don't often get the engagement back. A bit of sleuthing usually turns up the evidence!
I think I prefer talking to people about it rather than flagging it or down voting as I can see WHY they might do it - it's just the ones who are intentionally nefarious that bother me. I like to chat to them about it as I don't want them to end up leaving because they've blown their rep early.
What pisses me off is the ones that get their knickers in a knot over it and start screaming at you and down voting you just because you've politely raised the issue. Thankfully there definitely are more good players here.