HF27 and the scheduling problem.

in HiveDevs2 years ago

As you already know we are having HF27 tomorrow. It contains only two fixes. First one addresses old rare bug in snapshot mechanism that obviously had to actually happen. Second one fixes problem with backup witness schedule, described f.e. here, an unforeseen side effect of changes related to new HF26 OBI mechanism.


Snapshot problem

Just quickly about first one. When snapshot is generated, it basically stores state on disk, so it can be restored later, making setting up nodes much faster than full replay. The bug was caused by the fact that next_id - an internal counter of objects of particular type - was not stored, so when snapshot was loaded, it was set to value of id of last restored object plus one. Object ids are internal thing and while they are returned by some APIs, database_api in particular, they should not matter... Except proposals use that internal id to set proposal id that is then used in transactions to target the proposal f.e. during update. So when snapshot was prepared when last proposal ever created was also already removed, the value of next_id in original state was different (bigger) than its value after snapshot was loaded. That was a setup necessary for disaster. As a result new proposal being created had different identifier on original node and on node recreated from snapshot. That meant that transaction containing proposal update that was correct on one node was guaranteed to fail on the other. And that's what happened killing some API nodes set with use of snapshot.


The scheduling problem is more interesting, because the scheduling itself is more complicated. I'll try to express it in "human language" :o)

The schedule

Imagine a bunch of runners (witnesses) competing who can run to the top of the hill first. There are also crowds of people backing those runners (voters). The reward is a place in history for all eternity (chance to produce a block as part of immutable blockchain). Every 21 blocks the winners are elected in the following way by impartial judge (scheduling algorithm):

  • first 20 winners (witness_schedule_object.max_voted_witnesses) are chosen based on their backing (total power of votes).

As the winners are selected, they are teleported back to the starting line, so they are not chosen again in further selection steps.

  • next 0 winners (witness_schedule_object.max_miner_witnesses) are chosen for their effort
  • finally remaining number to fill the quota of 21 (HIVE_MAX_WITNESSES), that is 1 (witness_schedule_object.max_runner_witnesses), is selected based on extrapolation of their current position and pace.

Since blocks need to be produced, the scheduling can't wait until witnesses actually reach the finish line.

So far so good. The problem of the bug was not in the scheduling but in handling of special events related to voting.

For most of the time people in the audience just sit and wave their flags. However once in a while a person will stand up and either start cheering, which increases the pace of the runner they are backing (increased vote), or they will start booing, sometimes throwing their flag completely, which decreases the pace of runner (decreased vote). When such event happens, the judge has to recalculate their estimate on the completion of the race, using old power of the backing and the time of last estimate up to current time, and then use new value of that power for further run. The problem arose not from what changed, but from what did not change but should've.

The change

One of the main courses of HF26 was OBI. It works by allowing witnesses to exchange their confirmations of block validity outside of blockchain, without waiting for actual vote in form of signed block. But the confirmations can't be sent by just any witness - they have to be signed by the witnesses that are going to produce blocks in near future. For that reason we have to know in advance who they are. With just current witness schedule, depending on when we are within it, we might know only couple of witnesses, not enough for full irreversibility. That's why new future witness schedule was introduced, so we always know enough witnesses in advance. It looked like that new object needed to be used only for OBI and as a target for scheduling mechanism, but the bug showed that such assumption was wrong. As a result when change in vote happened, witnesses were recalculated using wrong data (from previous schedule).

The bug

Imagine you are the runner and that every time one of your backers stands up, even if they are to cheer you on, they also throw a bottle of water at you. Of course that would slow you down, and in some cases it might even make you roll downhill. Moreover, the more individual active backers you have, the more frequently they stand up and therefore the more bottles are thrown your way. Even worse, you are pushed back proportionally to the power of your support. In other words, you are hit by a bug more frequently if you have more active voters and the bug hits you harder if your overall votes are big. There is one more aspect to it. If at some point, despite the bug, you are the winner - which puts you back to the starting line - and then you are hit by the bug, you might be pushed back below starting line (underflow), which is teleporting you near the top of the hill, which makes you likely to be winner again.

"But is it really that common? I mean, the bug only happens when voters change their vote."
It might look weird at first, but it is frequent due to claim_reward_balance_operation. Be it curation, authoring or beneficiary, the reward always contains part that is expressed in VESTs and claiming it influences voting power.


Positives

There are two positives of the bug. First, the backup witnesses that are normally very rarely selected, are now battle tested. They have a proof that their nodes are not just mock-ups and are up to the task. Second, it showed the importance of testing on mirror-net. It is a new tool, so not everyone knew / was convinced enough to join, but if enough real people took part of the testing, the bug was trivial to find and fix. All it took was to spot that it was there. We can't expect @gtg to do that when he is substituting for several witnesses all at once :o)

Sort:  

I appreciate the exposition of the issues necessitating the HF tomorrow. Just to make things more difficult, and to reveal my lack of understanding, I would ask if you can explain how the 30 witness votes weight.

My understanding at present is that each witness vote we cast fully represents all our HP, and so our witness votes total 30x our HP. This, as I understand it's mathematical effect, thus weights stake 30x in governance, such that a voter with 1000 HP throws votes worth 30k HP for witnesses, while a voter with 100 HP, rather than having 900 VP less than the VP the 1000 HP voter has, only casts votes worth 3k HP, which is a 27k VP difference.

Further, I reckon that I, and likely many others, would prefer to be able to deliver all my governance VP to fewer than 30 witnesses, but the previous paragraph reveals that voting for less than 30 witnesses just reduces the total VP I can deploy, rather than increasing the weight I can throw behind a lesser number of witnesses.

Accordingly, it seems to me that instead of delivering 30x times our HP as influence on governance, it would be better for my purposes, and equalize the influence of stake on governance at all HP levels, were our VP simply split across our witness votes, and better yet if we could set the weight of our VP as a percentage of our HP witness by witness.

Please correct my lack of understanding as necessary, and address whether or how changing how our witness VP is weighted would complicate voting for witnesses. The 30x multiplication of influence of stake on governance strikes me as not only unnecessarily plutocratic, but unwise, given what happened to Steem.

Thanks!

Ugh, no.

voter with 1000 HP throws votes worth 30k HP for witnesses

No, he throws (up to) 30 votes of 1000 HP for different witnesses. His influence with each cast vote is still 1000 HP.

while a voter with 100 HP, rather than having 900 VP less than the VP the 1000 HP voter has, only casts votes worth 3k HP, which is a 27k VP difference

The sum of absolute values of votes is a meaningless number. The smaller voter has 10 times less HP than the bigger voter and therefore has 10 times smaller influence in the election. He still needs to only increase his stake by 900 HP to reach the bigger guy, not 27k HP.

First of all, top 20 are selected based on being within first 20 positions when ordered by power of their backing. The absolute vote values have zero meaning. Even if the last one of those 20 had a 0.000001 VESTS more backing than the 21st guy, he'd still win. Backup witnesses are running at the speed proportional to their backing (so bigger absolute vote power means more speed), but they win the race at the frequency proportional to the relative weight of their backing (so a guy with 1000 HP backing will win 10 times more frequently than 100 HP backed guy and the guy with 30k HP backing will win 10 times more frequently than the one with 3k backing).

I could go on how the system you're proposing would make it possible for someone with relatively low stake to block any fixes and changes that require hardfork, or how the only logical way (at least for whales and orcas that have most to lose on bad governance) would be to split the vote for 15 witnesses (17 around hardfork), how there would be need to constantly keep an eye on any changes in the votes to shift vote to next acceptable witness in case one of those you've chosen fell from top spot, how the notion of "wasted vote" and "voting for lesser evil" known from normal elections would be introduced to the system. But that's not necessary. The 30x multiplication of influence does not exist, therefore the conclusions derived from it are unfounded.

From the election standpoint it would be best if votes were not limited at all, however that would slow down updates of voting power and would pose potential attack vector. 30 votes represent good middle ground. They let you chose two sets of 15 witnesses (each poses consensus majority if elected) - the ones you actually want to represent you, and the others that are still somewhat acceptable but are more likely to be selected to the top positions. This way you vote for your representation and your choice is wide enough to be stable - not require constant updates due to votes of other people.

Loading...

The explanation was good, but the art is amazing.

Still I'm dissapointed that no community member picked it up and created real time scheduling visualization based on this ELI5 scenario ;-)

Let's give them some time for development.

Oh damn. And here I thought throwing bottles of water at these people was a fun new feature...

Is 35 still considered "several"? ;-)

I mean this just cements @gtg's position as a wizard, now that he is able to magically impersonate over 30 witnesses on testnet :)

Thanks for the beautiful artwork and awesome explanation ABW.

wow I understood!!! I feel smart thank you also loved the art.

I still wonder what's the best part of your post: the art, the runner and bottles story or the detailed technical explanation? 🤔

Congratulations @andablackwidow! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s):

You received more than 2000 upvotes.
Your next target is to reach 2250 upvotes.

You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP

Support the HiveBuzz project. Vote for our proposal!

🍕 PIZZA !
@andablackwidow! The Hive.Pizza team manually upvoted your post.

Please vote for pizza.witness!

$WINE


Congratulations, @theguruasia You Successfully Shared 0.800 WINEX With @andablackwidow.
You Earned 0.800 WINEX As Curation Reward.
You Utilized 4/4 Successful Calls.

wine_logo


Contact Us : WINEX Token Discord Channel
WINEX Current Market Price : 0.213


Swap Your Hive <=> Swap.Hive With Industry Lowest Fee (0.1%) : Click This Link
Read Latest Updates Or Contact Us

i'm quite confused😕

They have a proof that their nodes are not just mock-ups and are up to the task.

That's an interesting way to put it. I was more like 💩 in my pants literally, not knowing the unknowns of the experience of producing so frequently. Was really valuable to see how well the code performs and since I am not yet open to the public, to see that my side-chain node(s) requests did not impact any of that.

I think slowly learning all of this has also been quite a positive. There is so much and it's very easy to be overwhelmed with roadblocks. But luckily, being around many who already know much has been very fruitful. Either by just observing or reading or when asking a question. Everyone's super helpful in their own ways.

💪

Is this the reason that the Binance hardfork has taken so long due to the issues in 26 and subsequently, HF27? I have had HIVE stuck in Binance for a while now and want to use it to power up. Do we know when things will be back to normal?

That would be the question to their Customer Service.

The issue that was addressed in HF27 didn't affect anyone other than backup witnesses so it is hardly an excuse. Moreover that hardfork didn't even require replay. Just stop the node, replace executable and start again. Not even five minutes of downtime once you build the exec.

So, Binance either messed something up quite badly or they're just taking a while because it's not a priority for them.