Someone tried to power down negatively (withdraw negative VESTS) last week. The account @nijeah was responsible for that:
As the transactions kicked in last night, the system bugged and caused everything to halt. The servers were running, but without any block production.
The official statement is here:
https://steemit.com/steem/@steemitblog/steem-blockchain-patch-issued
Steemit's developers and engineers quickly addressed the issue in a concerted effort with the witnesses. After a few hours of investigating and testing, the developers came up with a fix that was slowly implemented until the system recovered this morning to almost full participation (98.4375%). It's imperative that all witnesses update to the latest 0.19.5 version, or 0.19.10 (AppBase), since the update was a soft fork.
The update is at https://github.com/steemit/steem/releases/tag/v0.19.5
Bug Fixes
A field in withdraw_vesting_operation was not properly checked. This invalid op could halt the blockchain when the vesting withdraw was processed well after the op was made irreversible. Included ops are now no-ops to prevent the halt and new ops have the proper check to prevent this behavior in the future.
During the crisis, I manually rebuilt steemd on all my servers. The main and backup servers were updated to 0.19.5, and my AppBase test server to 0.19.10. Unfortunately, some witnesses experienced a random forced replay (which takes hours) instead of a restart that takes a minute; I was one of unlucky ones. It was a very long night for everyone, while some went to bed, some were waking up; the witnesses are spread across the globe. I tried to sleep a 4:00 am, but couldn't sleep at all. Got up again to monitor the progress and keep in touch with the others. Finally, my first replay finished around 7:00 am!
I enabled my signing key for it and went to sleep. It produced fine. Since many witnesses are disabled or missing blocks for not running 0.19.5, I became a top 20 witness by interim. I finally made it up there, even if it's going to be for a short period (whales, you're always welcome to vote for me). It's a good feeling.
Today, I switched my production to the 0.19.10 server, since I've been running AppBase for a few weeks already without any problems.
Applying The Fix
If you're a witness using the dockerized setup and haven't patched your server(s) yet, check this post https://steemit.com/witness-category/@someguy123/emergency-update-for-steem-in-a-box-witnesses-seeds-and-rpcs
If you're building manually, do:
git pull ; git checkout stable
For the AppBase build, the fix hasn't been merged yet, so you need to do:
git pull ; git checkout 20180702-fix-vesting-withdrawals-steemd
(thanks to @bhuz for the tip)
Then recompile as usual.
Acknowledgment
Thanks to the Steemit team for quickly dealing with this issue, as they always did with critical incidents. And thanks to my fellow witnesses who were available with me to implement the fix as soon as it was humanly possible, despite the late hours.
Available & Reliable. I am your Witness. I want to represent You.
🗳 If you like what I do, consider voting for me 🗳
once.
Congratulations @drakos! We all have been waiting from past 90 days after sircork's prediction! Hard work, dedication and a little bit of arghhhhh!
I think I went to bed barely minutes before it started and I woke up when it had been several hours the blockchain had halted... discord felt like some kind of crazy afterparty.... I mean of course people were concerned... but it was more like " guys we are staying up late, there are people fixing this" some sung songs, others spread information around the servers... during the black out ;-) discord proved to be a backdoor to keep us informed... I cannot imagine the stress and hecticness among the witnesses... but here we are! up and running! Thank you!
I wish I had time to spend with you guys, but we had urgent priorities.
Yes, I can imagine it was a really hectic night and I hope you got a proper deserved rest... by the way congratulations on the placing on top 20 witnesses!
Thank you for a detailed explanation of what took place. I appreciate the time and effort you give to make sure that the Steem blockchain and community are fully functioning.
Thank you... you poor dear .. resteemed and upvoted
That is why you are one of the witnesses I put my vote in for.
Thank you for your sleepless night and tireless efforts.
I'm glad you hit the top 20. Well done. You deserve it. :)
I was wondering what had gone down. Thank you for keeping us posted and burning the midnight oil! :)
Wow, nice job. Thanks for bringing this to my attention I was pretty curious. I'll vote for you... all 1000 SP :D
That is a really bizarre bug. Why would the blockchain code even allow someone to withdraw negative vests?
I tried for hours to go on and i began to worry that it was the end to steemit. This is better clarification for the problem
with due respect for your wise and hardwork...
Rhymes with some Sherlock novel's name ...
Very sad. Can we get any action on those people like nigeah on steemit? @drakos
Stunning work my sire, sorry for the late night you experienced and many thanks for the hard work I am sure is much appreciated by all honest folks here. That account looks suspiciously like another famous nefarious 'crypto analyist' here it would not surprise me if there is a link, I am sure that avenue is been explored by you smart folks though.
Be blessed
Yours Always
Pricasso
Drakos top 20 ! \o/ :D
you deserve it
I could not imagine what a difficult life a witness has!
Well done @drakos! You successfully guessed the match result.
Click on the badge to view your Board of Honor.
Do you want to know your current ranking? Read this post
Do not miss the last post from @steemitboard:
SteemitBoard World Cup Contest - Round of 16 - Day 3
Do you like the SteemitBoard World Cup Contest?
Vote for @steemitboard as a witness and get one more award!
C'est bon de savoir que tu veilles sur nous pendant qu'ont dort.. comme ça on peut focuser à faire des beaux rêves. Top 20 by default is still a top 20!!!! Bravo
Someone should investigate @nijeah account to find the withdrawal memo and check the blockchain to see if that memo was used for withdrawal from other accounts.
Hint, Hint...
You got a 22.94% upvote from @upme thanks to @drakos! Send at least 3 SBD or 3 STEEM to get upvote for next round. Delegate STEEM POWER and start earning 100% daily payouts ( no commission ).
As usual, you showed up to the best of your ability – no matter the cost. Thank you for ensuring everything was back up and running as quickly as possible! Totally sucks that your witness was one of those that experienced a replay. But how awesome that this whole thing bumped you into top 20 for a minute. Foreshadowing of things to come, perhaps? ;)
As @yidneth said, we were all hanging on Discord. Turned into quite the raucous gathering. Seemed no one could sleep! I finally logged off after sharing a lullaby with all the middle-of-the-nighters on the East Coast and elsewhere – I sang Evening Falls by Enya. Thought of you!
I hope you get some good, well-deserved rest today!
@zipporah I missed you singing Enya!! I woke up and party was kind of over, but I saw your message crossing different servers and honestly was the first one I read to be up to date to what happened.
This morning, when Steemit did not work, users were very worried and discussed it in the chat rooms. For many, life stops at this moment ))) And how much joy when problems are eliminated!
Thank you for your work and support!
@drakos
I had started complaining that my network is the poorest, but steem wasn't working actually..
Badly addicted to steem..
I couldn't have had worse luck. Got on a 9-hour flight at pretty much the minute that it started and didn't even know anything about it till I got a call from another Curie member when I landed. Thankfully they were able to sort out a node while I was in the air. We are now back up and running on the original servers after a concerted effort by everyone on the team. As I said in themarkymarks's post, it was really good to see the community, top 20 witnesses and steeminc come together to get this fix expedited. Great work all round.
After the last witness drama Curie went through you must have been on your toes when thia happened. Seems like a small thing for us layman, but this must have been nerve racking..
And you guys pulled it off. Came together and acted. Bravo. ❤😇
I feel you on not being able to sleep. I also had to replay my servers to get back up to running, and I managed to accomplish that at 11am CST. Your hotfix was a blessing, @drakos. I did, however, take this opportunity to try my hand at building steem manually. It was a learning experience, and the next backup server I start is going to be built manually so that I don't end up dependent on docker like I did this time around.
Thanks for all the hours and effort you put in! You'll always have my vote, and congrats on making it to the top 20 (for a while) :D
I cant say i understood much of the witness lingo, but really it seemed like a bad time for you guys. We were just: "oh, steem is down lets drink coffee".
But behind the scenes its red alert.
Shows how important your work is.
Thx.
You bet it was red alert! It's our duty to work together to keep the blockchain up and running.
Drakos.. But did he manage to take the negative vests out? I see it says -10mil... Did he succeed at stealing? It doesnt say he sent anything out.
Well done @drakos! You successfully guessed the match result.
Click on the badge to view your Board of Honor.
Do you want to know your current ranking? Read this post
Do not miss the last post from @steemitboard:
SteemitBoard World Cup Contest - Round of 16 - Day 4
Do you like the SteemitBoard World Cup Contest?
Vote for @steemitboard as a witness and get one more award!
♥♥♥
thanks for everything you do for the community... it is NOTICED!
tip!
Hi @drakos! You have received 0.1 SBD tip from @paintingangels!
@paintingangels wrote lately about: But No Feel free to follow @paintingangels if you like it :)
@tipU voting service | For investors.
bahhh, I didnt know they leave infommercials about my stuff... ignore that part! lol.. just wanted to do the tip thang.. :D
I thought it was like a DDOS or something, they weren't giving out much info on Twitter. It seems like a big response to one account trying to power down to negative numbers that the whole system shuts down. I know it's still in Beta but i hope that doesn't happen again.
I think that explains why I could not access Steemit last night, now it all makes sense! Thanks for the information :)
I used your explanation to update Steem in a box. It worked even though I had to do a replay. I also to go bed quite late but that's the usual for me. Congrats and thank you Drakos!
discord is a lifesaver when it comes to sharing information, we were all informed about what was going on pretty fast and people were lovely with their acceptance that witnesses live across the world from each other and in different time zones. We waited patiently and you guys came to the rescue as soon as you could. Thank you for that 💚
On Monday night I was scared because suddenly I could not get into Steemit by any means and I thought it had been the government of my country that is blocking pages and this scared me a lot, then I realized that they were problems directly with Steemit.
Wow @drakos spend practically all the dawn working to solve this problem is admirable and more than they fixed it very fast, you are the best as I always tell you and you deserve the number 1 position of the witnesses.
Thank you for your work as a witness.
Just going through your steemian.info/witnesses list and noticed you were in the top 20. Congratulations......longtime coming. Well deserved.
The sparkling pirate is the best!! xD
I'm lucky I slept right through the issue haha