Yesterday, the blockchain - Steem - stood still and in this post, I'm going to take a retrospective look at the events.
What happened?
Now, due to a bug introduced in the new version v0.20.0 and activated after an edge case scenario, the blockchain froze. (Steem works in that way, that if there is a critical problem, the blockchain just halts)
After a few minutes of being in the black, we got the news inside the #witness channel on steem.chat by @timcliff: the TOP 20 witnesses came to the consensus to rollback to 19.x.
I believe that this was the correct decision.
Problems: Low Communication from Steemit Inc. & No ETA
But what happened afterwards, was a very long period of waiting of over 10 hours without any ETA and not enough communication.
Now, please don't get me wrong. I can fully understand that fixing important bugs takes time and sometimes, giving ETAs can be difficult.
But as a witness and as someone who is responsible for a business running on the blockchain, I need to know what is going on. (And I'm sure that I wasn't the only one. Steem Monsters is running a kickstarter right now in addition with being on Token Fest. Not being able to tell the customers when they can buy packs again etc. - that's lost money and unhappy customers)
Getting the info: a difficult path
And the flow of communications was very supoptimal.
I had to use a combination of the #witness channel on steem.chat, steemspeak on discord and the official SteemNetwork twitter account to be in the loop of what was happening.
Finally, at 2am in the morning for me, (nearly 10 hours after tim wrote the message on steem.chat) I was made aware of a solution. But I only knew of it by being in the steemspeak discord and hearing @themarkymark say that a specific version was released on github.
4 hours later and my nodes were replayed and first block was produced.
My Lessons out of this
I'm a big believer of taking a step back and using experiences as a way to grow and learn.
And in retroperspective, I think I wouldn't have needed to replay one of my nodes, as it was already on the right version, which would have made my node one of the first 20 running on the chain.
But in this case, I was active rank #24. (up from the real rank #32). Still very good, as I'm estimating that the TOP 20 got a heads up with the new fix, to not undermine the security of the blockchain. Which I understand.
Besides that, I also learned how important it is to always have a different version on your backup node. Especially if it's such a huge upgrade as v0.20.0.
And regarding the problems I've explained above:
Steem has a market capitalization of over 230 Million USD and this is on the low end. We were easily over 2 Billion. And a company (or however you might want to call it) of that size, has to make sure there is an open communications flow, especially with witnesses or business owners who are relying on the blockchain.
So please, @ned, take this feedback seriously. I love Steem, I want it to succeed and that's why I'm writing these lines.
Didn't witnesses review the code?
Now, regarding the accusations that witnesses didn't review the code enough, otherwise the problem would have been known.
Well, yes and no. First of, reviewing Steem code is actually very difficult as it requires a very high level of programming knowledge of c++, in addition with deep knowledge about blockchains & Steem.
On top of that, the error was only visible in an edgecase scenario and would have needed extensive tests with specific circumstances (at least that's my understanding).
While I am totally in favour of witnesses reviewing the code, most of us don't have the necessary skills. Now, achieving these skills is on my to-do list, but it will require quite a lot of time and for me, Steem has also multiple other needs. For example: we need more dApps like Steem Monsters & Co.
And currently, I rather focus on my strengths and the skills I have already developed to bring value to Steem by being as effectiv as possible.
Is Steem not decentralised?
Last but not least, I also want to get into the accusation based on yesterdays freeze: Steem is not decentralised.
In my opinion, this is not correct.
Steem isn't being run on one server or by one organization, which would be centralisation, but on many, many different servers all around the world.
Now, you could argue that some servers are being hosted in the same data-centers and from the same providers, or that some witnesses outsource their hosting to other witnesses and yes - those are security risks everybody should take seriously! (!)
Which means: the fact that Steem has hundreds of witnesses running multiple servers shows that it is indeed decentralised.
However, there is a bottleneck/weak point in Steem, which caused yesterdays problem, but all other blockchains have this in common: the code.
And I don't mean some holy code or law code, I mean the written code by the developers.
If there is a bug or problem inside the code, we could have 1000% more witnesses, it wouldn't change anything.
Maybe far in the future, we will have an alternative, but for now - we need to be very careful about the code and make sure it's well audited.
With that said:
Have a great day!
therealwolf
Witness Infrastructure:
Primary Node: 128GB - v0.19.12 | Backup Node: 64GB - v0.19.12 | Seed Node: 64GB - v0.19.12
Projects I've developed on Steem:
Smartsteem.com > Investment & Promotion Service on Steem
Steem Chat-Wallet & Witness Essentials > Github: https://github.com/therealwolf42
If you believe that I'm of value for Steem, then please vote for me as witness. You can also set me as a proxy and I'll vote on great witnesses for you.
Danke, dass du uns wieder mal aufklärst. Zum ersten mal in meinem Leben hatte ich Entzugserscheinungen als Steemit nicht erreichbar war.
Mir ging es ähnlich :D
I'm not a witness but I agree with what you just said. Yesterday, I was on Steem.chat thinking that it was the best place to get updates on the issue and what we got as answers was: "relax and wait for further instructions", just imagine one of the major centralized service saying that kind of stuff to their users... I don't think that anyone at Stinc realize that they have a huge community behind them and I don't think that a few messages posted here or there with a few words giving the status of the issue would have hurt anyone...
Also, still, we are waiting for the real reason of this halt... (unless you know c++ and read the commits from @abit, good luck to figure out what was the problem)
I believe in this blockchain, even though I'm not a big stakeholder, I work everyday for Steem, to bring new features and ideas, hopefully the new comers won't look too much at Steem's history...
Exactly. There are much better updates for football games, but when a 230 Million dollar blockchain is frozen and businesses can't operate, then we get an update every x hours.
Now with that said, I'm sure everyone learned out of this experience and the next time we may have better updates :)
Thanks for this well written article. I think it helped some of us to learn something more about Steemit. Also for me some things are new and i am glad that you wrote this text. I also share your opinion that the flow of information in a blockchain like Steemit, definitely needs to be better organized. it shouldn't be that you don't know what's going on for so long.That's why I find your input more than just appropriate and appreciate your constructive choice of words.
Best regards out of the van
Sarah
Thank you for the great comment, much appreciated!
A very well written article. Though I thought I was pretty well-versed, I actually learned a quite a lot from this including the intricate parts of the inner workings of Steem. Thanks for the writeup, and completely agree with your stance on how this was handled. Steem on!
Posted using Partiko Android
Glad that you found it useful! Steem on!
I was having trouble gathering information on the outage myself as most media tends to correlate to steemit and a quick google search turned nothing up, that said I knew the witnesses were probably on the ball and was happy to see it back online right away, there will always be a few kinks with new forks i'm just glad someone is doing the work to ensure we operate smoothly.
Steem of-course is centralized .The essence of decentralization is that service shouldnt be broken if anything goes wrong on the network...with 20 node algorithm , there is no argument to be made here
And regarding 20 nodes - you cant have extreme decentralisation and extreme scalability. At one point you have to decide and Steem is decentralised enough.
More on this point, if one of the 20 nodes trys to cheat or isn't playing by the rules they will get voted out immediately and lose a lucrative job so the takeaway is... consensus is still achieved in a decentralized manner.
What if an individual or individuals on behalf of a company becomes the majority witness, we saw that with EOS
Nice article! Very clear!
Hi sir
Posted using Partiko Android
Sir can u tell me about steemit how to increase my income plz tell me
Posted using Partiko Android
Keep working on your content and building relationships with the broader Steem community.
Very good article, thanks for sharing congratulations @therealwolf This article really helped some of us to learn something more about Steemit. I Believe definitely we needs to be better organized.
Thanks for this article. It’s really useful especially for me I’m just starting my journey with Steem. 👌🏻
Excellent post and some very clearly expressed points. I think the Steemit development teem are communicating more than they ever have; and hopefully take on board your feedback here. As the number of stakeholders increase (particularly with SMT's ) the points you raise will become even more critical I we want to keep all the DApps from shifting to other blockchains.
Hi @therealwolf. Communication is the key. The downfall of many things is poor communication.
With the quick resolution of the issues that held the block chain for those hours, i am fully assured that those at the top of the block chain which includes witnesses and others i may not know will always run to our rescue whenever there are any treats to the block chain.
The other thought I had during the downtime was that there should somehow be at least a read-only version of the blockchain. So links and such would still be things they could read. Is that more impossible than I think?
Posted using Partiko Android
wow! once again very informative and interesting! :3
By the way awesome new logo!*-* i realy like it :3
So top 20 marketers should hand it over to people who actually reads c++, ay?
That's why tests exists, you don't just login, comment, vote and expect everything will be fine. :face_palm:
missed the sarcasm here :( https://en.wiktionary.org/wiki/(!)
LOL that was pathetic mr.businessman. A flag without a reasonable response?
And of course, non-top 20 marketers shouldn't use this as an opportunity :)
Thanks for the update @therealwolf. We need to get you further up the witness list!
I am not so keen on vote bots, but I like your professionalism and values. The steem blockchain is lucky to have someone of your talents contributing to it.
@kabir88