I made a comment elsewhere on a cross-post but I will come over here. I only know one person who got significantly downvoted, and they at least commented on why they did it. When my friend started posting differently he got rewarded again...
You seem to know why they started to downvote you - did they leave any comments that you can show us at the beginning?