Yes, I remember the good days when my wallet was a lot fatter with much less stake. No worries, those days will be back.
I always make sure I have a valid reason to downvote because I believe we need to keep Hive healthy and clean, to maintain an appealing and interesting feed. That's the first thing investors will see. As to the stress, I just ignore it.