Most of our Steem witnesses are now back online, the majority of us have reverted to a patched version 0.19.12 of the blockchain software.
What happened exactly?
A lot of witnesses have installed HF20 (v0.20.0 of the blockchain software) and it was going well for several days. Tests on Steemit Inc side were going good too. But suddenly, more and more witnesses missed their blocks. After an hour or two, if memory serves, it was spotted that the transaction that triggered the issue was an upvote on a post that was less than 12 hours from expiration.
What usually happens in HF19 is you cannot upvote a post that is less than 12 hours old, there is an upvote lockout. However, a bug was introduced in HF20 which has dropped that 12 hours window to just 12 seconds. You can see the fix for this here:
https://github.com/steemit/steem/commit/86df97f322c86f767f01b80ea64e193c73ef6d89
I made a screenshot for you, check the green and red lines. Green is the original code, red is the modified one.
That piece of code seems to be from HF17 so it's rather old, why had it been changed from 12 hours to 12 seconds? They were testing something and forgot to put it back?
Anyway, I think all of the witnesses (including me) have failed here. We should have reviewed the code change more thoroughly and spotted this more or less obvious bug in the code. There is no expectation for backup witnesses (under rank 20) to do a lot of that kind of code review but those who can understand C code should start looking into code changes before applying any upgrade. I'm not a C developer and was relying on the witnesses above me to help with code review. This has obviously revealed one of my weakness as a witness, one that I will have to strengthen as I climb up the ladder.
What's the plan with HF20?
I guess, we will wait for Steemit Inc to fix, test and release a new version then witnesses will have to do a proper code review and apply the new code.
I'm back in business!
I'm currently shown at global rank #121 and active (global rank minus dead and inactive witnesses) rank #83. As more users re-enable their witness, my active rank will progressively go back down to its normal rank of around 100 I think.
After hours of down time due to patching and replaying (reprocessing) the blockchain, my server is finally back up and few minutes after re-enabling my witness, it has produced its first post-patch block.
Extent of the damages
I'm unsure about the status of the damages on the blockchain itself. Has the fix to remove the multiple forks on the blockchain has lead to a loss of transactions? Maybe. Your STEEM, SBD and SP should be untouched and safe, but some of the transactions you made might not have gone through, things such as doing an upvote but it wasn't recorded. I'm just guessing here...
On a personal side, I spent stressful hours on the night it happened, staying up late when I had planned to go to sleep early (that's the life of developer lol). The server wasn't operating for almost a full day after that. I spent some times helping fellow witnesses on public and private chat on Steem Chat and Discord.
Because my current rank is low, I don't receive many blocks to sign so I didn't expect to miss many blocks. But as witnesses above me went on to temporarily disable their operation, I got boosted upwards and as others above me were missing blocks, they naturally came to me too and the bug also forced my witness to miss two blocks in a very short time. To help with debugging, I left my witness on until a fix has been found. Luckily, the top witnesses have soon decided to revert to HF19 and stopped their server to do so, this put the blockchain to a halt so I didn't miss any more blocks until the fix was provided the next day and I disabled my witness to apply the patch.
Many backup witnesses are discovering their weakness and technical limitations during these kinds of event. It can be stressful and desperate for some but if you don't give up and be patient, then it's also a good way to learn more about the technical side of the platform. The witness community is also very helpful.
Related content
- ULOG 087 - My Witness has been stopped to apply a patch
- Witness Earnings Report - 2018-09-17
- ULOG 085 - Garden Spring Video Tour
- ULOG 082 - Propagating some veggies
Summary of some of my other contributions
| Project | Description | Link |
|---|---|---|
| Steemed Phish | Chrome browser extension to make external links more visible and alert you on Phishing/Scamming links | Steemed Phish |
| Dtube Community Support | This project by @evildido allows users to run their own IPFS node and pin videos in order to cache them for longer. Dtube has the tendency of deleting non-popular video after few days only. I helped Dockerizing the project for an easier deployment on almost any platform. I've also contributed by running two nodes, one in Australia and one in Canada to cache #teamaustralia and #teamnz videos. | Dockerizing Dtube Community Support |
| DiscoSteem Bot | This is a Discord/Steem bot project by @planetenamek. When I started Steemulant Bot for the Vietnamese community, I didn't know the existence of DiscoSteem Bot and wanted something written in NodeJS. I've ended up continuing my own project as it contains code and logic for personal use and for the Vietnamese community. But I've decided to share some useful features that can benefit the users of DiscoSteem Bot and will continue to add more functionalities as I go. | Dockerizing DiscoSteem Bot |
| A Dollar A Day | This is a project run by @pennsif and will donate 15 SBD every day to a charity | "A dollar a day" fundraiser - Karaoke Singing: Le chanteur malheureux |
| Team Vietnam | I started a new Vietnamese community on Discord and developed @teamvn, a community bot to help grow the Vietnamese community and help them create higher quality content. | |
| POCKET-JS | This is my Javascript (NodeJS) port of the POCKET confirmer by @biophil. POCKET is an experimental crypto token on top of STEEM. | POCKET-JS Confirmer |
| Skippy Bot | I've written this Discord bot for @teamaustralia, it has three main features: a level bot (like Tatsumaki but with Australian animal images), a chatter bot and a footer banner generator for both @teamaustralia, @teamnz and @steemph | Witness Update 2018-07-30 - Team Australia's Skippy Bot |
Credits
- The image at the top has been generated with the Canva app using my own photo.
Nice work mate. Glad everything is back up and running smoothly.
Let’s hope next HF20 attempt goes smoothly too lol
Posted using Partiko iOS
Level heads prevail. Keep up the good work.
Posted using Partiko iOS
Thanks :-)
you remind me of highoncoins.
Chúc mừng @quochuy, bạn đã nhận được một upvote 35%. Tôi là con bot của cộng đồng Việt Nam trên Steemit. Tôi được tạo ra bởi nhân chứng @quochuy và được uỷ quyền, tín nhiệm bởi các thành viên Việt Nam, hôm nay tôi vote cho bài của bạn để ủng hộ bạn. Tôi hi vọng sẽ được thấy nhiều bài viết hay từ bạn.
Chúc bạn vui vẻ, và hẹn gặp lại một ngày gần đây.
Congratulations! This post has been upvoted from the communal account, @minnowsupport, by QuocHuy [witness] from the Minnow Support Project. It's a witness project run by aggroed, ausbitbank, teamsteem, someguy123, neoxian, followbtcnews, and netuoso. The goal is to help Steemit grow by supporting Minnows. Please find us at the Peace, Abundance, and Liberty Network (PALnet) Discord Channel. It's a completely public and open space to all members of the Steemit community who voluntarily choose to be there.
If you would like to delegate to the Minnow Support Project you can do so by clicking on the following links: 50SP, 100SP, 250SP, 500SP, 1000SP, 5000SP.
Be sure to leave at least 50SP undelegated on your account.
Thanks for the update and all the effort you put in to getting us back up and running. :)
Hi @quochuy!
Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your UA account score is currently 5.508 which ranks you at #527 across all Steem accounts.
Your rank has improved 28 places in the last three days (old rank 555).
In our last Algorithmic Curation Round, consisting of 516 contributions, your post is ranked at #120.
Evaluation of your UA score:
Feel free to join our @steem-ua Discord server