you're probably best keeping those that at least upvote some of your posts, no point shooting yourself in the foot over it! I think it's also a bit funny how all your "more active" followers may be trying extra hard to add a thoughtful comment to make sure they stand out from that type of crowd (not necessarily consciously either)...
Oh wait a a sec, *did I do that" just now too?! lol :D
As long as I get something more than "follow me" everybody's OK!