Feedback and clarification on this paragraph:
“We can therefore assume that any given measurement has a wide margin of error and that many measurements are required in order to get a more accurate measure of community consensus.”
This assumption requires clarification since repeated measurements only reduce statistical error.
There are two types of measurement error: 1) statistical error 2) systematic error
All measurement error is a combination of these two types of error and needs to be characterized and quoted separately in order to judge the merits of a proposed change to an experiment and its experimental apparatus.
Statistical error can only be reduced by adding additional measurements, while systematic error is always constant regardless of the number of measurements and is inherent to the experimental apparatus.
Systematic error can only be reduced by improving the experimental apparatus, which is part of what “Fractally White Paper Addendum 1” is proposing.
The proposed changes in this blog post hypothesize that reducing measurement error (a good thing) will be accomplished by reducing the systematic error (another good thing) by improving the experimental apparatus through the addition of a “high-pass filter” (aka a “high-value contributor filter”) to reduce low frequency noice (another good thing) in the form of a sliding/rolling window with the addition of a second consensus round populated with an improved source of signal.
The addition of a second consensus round improves the statistical error since it adds additional measurements to the experiment.
In conclusion, the proposed changes hypothesize that changing the experimental apparatus improve both the systemic error (through structural changes) and statistical error (through the addition of additional measurements).
I support the principle behind these changes when looked at through this perspective.
Someone (maybe me) could create a simple “measurement error calculator” for the fractally team to help them understand the pros and cons of this and future proposals. I would value this contribution. I may build this calculator unless someone beats me to it (which I encourage).