It is good to have accountability when it comes to communication and performance. The difference with downvotes is that a lot f them are just made for nuiance purposes, especially the smaller ones. There used to be trails of downvotes from some users who were just doing it to be annoying and get a reaction. For the most part, big or small, best to ignore. Often, the large downvoters will give a reason for the downvote - but not always.
