Very true. I'm the same way. When I was using SteemVoter I found that there were times I didn't want to upvote a post, even though I liked the author. For example, I may love a person's posts about blockchain and crypto, but if they post a video of them playing a video game then I'm not sure I want to upvote that.
One reason many people use sites like SteemVoter is to automatically upvote authors who consistently get high rewards, which in turn gives them big curation rewards. Nothing wrong with that and I can see why many people do it. I'm sure it pays nicely.