In my own opinion, downvotes are very much different from censorship. The downvotes are actually the voice of someone or a group speaking against another. The problem though is how it works. That is the problem that the democracy of the designated proof of stake faces.
Technically it is fair as a larger stake warrants a larger voice. In most cases though, it turns out to be unfair as this larger stake is controlled by a person or a group with their own interests.
If only it were possible for downvotes to be more equal amongst members. But then again, that would generate a different kind of abuse.