Your remarks are valid, I think it's because the culture is changing rapidly, people will learn pretty quickly the overall culture and how to behave. I've seen quite a few people who came to me to ask why I've downvoted them and I replied. It would be better to be able to leave a comment, but I think there are enough automated comments everywhere already.
" only content that is accepting rewards should be downvoted"
Great idea !