You are viewing a single comment's thread from:

RE: Do Users Leave Hive Because of Downvotes?

in Hive Statistics9 months ago

I could be wrong, but I think to potentially see the impacts you'd need to establish a clearer test on the impact of downvotes on a user by user basis.

For instance, a control might be to look at a users Posting rate before receiving a downvote, then find another sampling of users with a similar posting frequency.

Then, look at the posting rate after the downvotes - this would potentially tell you if the downvote had an effect on those users.

The control would be to see other users with similar posting rates but without receiving the downvote to see if the downvote might have had a correlation to a change in posting rate.

Outcomes could be a decrease in posting rate in the period after the downvote, no change, or even an increase.

I think that kind of comparison might be important to be able to measure for potential opportunity costs that downvotes may be causing.

I could be wrong, but that should let us see "does downvoting cause people to stop participating"

Sort:  

The problem with this test design is that it's very hard for the samples to be genuinely independent. Without performing an actual controlled experiment, ie. setting the population and introducing the variable yourself, attempts to sample users who received the treatment (downvotes in this case) basically always involve sampling bias - there is some reason this user was downvoted and one other was not, and this is difficult to control for.

The advantage of the method above with granger analysis is that it works reasonably well retrospectively, by not making such attempts to sample - looking at the whole population.

That makes sense.

There would still be some level of a correlation you could detect though right?

The sampling bias would mainly cause a problem for being sure if the correlation indicated causation right?

But it could at least be seen - does downvoting correlate to users not participating anymore. There could still be a question as to whether that's a good thing, a bad thing, or if the downvote was likely the cause or not.

Keep in mind... this is all with only the vaguest of understandings of proper controls and testing on what you're describing - fully aware that my thoughts can be full of blindspots :)

You could see such things as those users who receive downvotes being more likely to leave, but there are many forms of sampling bias that could end up giving you that same conclusion mistakenly. It would take a greater degree of rigor to control for such things. Possible to do but takes more work and expertise.

It might be worth doing these kinds of tests anyway, because it can be interesting to examine even while being aware of the limitations.