Now I am no expert on blockchain technology nor a Hive programmer, but somehow I do not feel that anytime someone else's post is downvoted, those removed rewards somehow increase the payout of my posts, but maybe I just don't fully understand how Hive works.
It's very simple: The blockchain is continuously distributing out a fixed amount of new tokens to users through the reward pool (the part of Hive's inflation that goes to author rewards and curation rewards). Whenever payouts happen, the blockchain distributes this fixed amount out to users who made posts whose payout time has reached the same window, and the fixed amount is distributed among them based on stake-weighted votes. So since the amount paid out is fixed, if you get less, the others get more.
Of course, you will not notice your own payouts going up just because you give someone a downvote simply because the value of one downvote is so small compared to the size of the total payouts of the rest. It's like how you can't directly notice the effects of one car driving on the other side of the world on the climate in your country, yet all emissions taking place all over the world still contributes to climate change.