I don't think your commenters/followers would mind, man. At least the loyal ones. I'm happy to receive an upvote from you when I leave them, but that's not why I'm leaving them. I just like to interact with people on here and enjoy your works and wanna let you know.
I don't think it would be a big deal to not receive upvotes for a couple of days so you could let your voting power go back up, bro. But, that's just me.