Everything you’ve said are true but then I feel if you let everything slide and you act you don’t care, people will tend to take advantage of that and do you dirty. That could end up affecting you in a very bad way during the long run.
So my take is, while it’s good to not pay attention to most people sometimes, it can also be dangerous not to act when you’re suppose to.
Well, I am of the opinion that the only thing or people who can take advantage of you are outside the confines of your knowledge. Well, except you willfully consent to be taking advantage of...
Also, like you said, it is not everytime you let things slide. Pick your battles carefully. Sometimes it's best to do and say nothing. That can be the best response in certain situations.