I used to do a lot of following back just out of courtesy or if asked. It kind of makes it tough though if your feed gets full of things that you don't care about too much. Dropping someone that you have followed doesn't feel too good either so I generally avoid that.
Some of the folks that try to get you to follow them spam out a lot of smaller posts quickly that I'm not too interested in. It makes it difficult to find and vote up content of the people that you care more deeply. A lot of times I check which people I comment on or have commented on my stuff recently and pay less attention to my home feed.
I generally don't vote for my own comment unless someone else votes for it first but that is just my own thing. I feel like it keeps me from going hog wild and spamming out comments so I can vote for them myself, lol. ;)
It's nice to hear a few people talking about what they experience with all of this. Helpful! Thanks so much for your thoughtful comment. I feel the same way.. I will unfollow if it's an authentic unfollow, but not very often at all. And so I don't follow unless I'm interested in hearing from people. lol about the self-voting :)