What an intense and draining night it was. So many witnesses had personal problems, traveling, in hospitals. I had a damn leaky pipe at 3am (when I discovered it), luckily it didn't burst. I had to cut off the main water valve for the house, then went back at my computer to follow the situation. Shitty times. Yet, despite everything, we did our best to be around to keep the blockchain alive. Teamwork at its finest.
Thanks for the information. I hope the platform is secure and stable over the long term. Questions are being raised about EOS, but I hope that's just a case of EOS being new with the initial kinks needing to be worked out.
So, the blockchain froze as the consensus was lost. This is a good thing then, which means that it did what it was supposed to do, right? Except the fact that buggy part of the software did allow the input of wrong value the rest went as it is supposed to……………. that’s good to know. Maybe you are one of the few witnesses that I follow so this the only post in my feed that gave me a level of assurance as to what went down wasn’t anything lethal.
Although apart from the general gist, I have really no idea about the technical differences between a replay and a restart. Except maybe that replay lasts for hours while a restart can be a matter of minutes. I have heard that the unlucky ones had to do a replay 😊. I never knew that missing the blocks maybe out of your hands then (I actually thought that it had a lot to with guys being lazy)
Thanks for sharing all this man. I spend quite some time and energy on Steemit so had a bit of scare, not much but it was compounded by the events of my personal life. I hope you drop by my latest post about my experience of the scare.
PS: Nice green hair 😊. Just a few days I ago I gave my first witness vote to you. Second was for @dragosroua. I’ll take a little more time with rest of the twenty-eight.
Thank you for being available all the time everywhere and helping out with information flow. Yes, response from the community in a crisis mode was absolutely amazing.
Thanks for the update. Communication timing is great.
Green is often spill-over from the background being too close or too brightly lit. If you can push it back, that is easiest. Otherwise, lowering the green lighting but keeping the background evenly list is tricky.
You were the first witness I voted for, and I know I made the right decision. Thanks for staying up all night, fixing thing and reporting back promptly.
While your bedroom looks like it just survived nuclear winter, I'm glad the Steem blockchain had a far easier time of it. ;)
Thanks @lukestokes really useful to get the news update - I've learnt some new bits there - and sounds like a great response to an emergency and it's impressive to get all the block producers onto the rollout so quickly. You have my vote and this update just gives me further confidence that it is a vote well cast. It's a good point about the missed block stats and I will also keep this in mind. Good work on the comms. Great leadership.
For several days now I cannot upvote any post or comment without getting an error. Have you seen this before? Do you know if anyone else is experiencing this?
Thanks for the explanation of what happened last night. I though maybe it was connected with the issue I'm having with upvote errors, but it was a bigger system issue.
If you can help me with why I can't upvote, I will then be able to upvote this post!
If you're getting the slider before the error: find something you want to upvote 100%. Drag the slider down a ways and then back to 100, then vote. After that you should be able to vote normally.
If you're not getting the slider I don't know that I can help; when it happens to me I glare at it for a while and eventually get the slider.
It's not related to yesterday's issues; @m-sargon has had this off an on for a month, and it hit my own account once.
Also when it doesn't work on Steemit I've been able to vote through Busy.
What error? Would help to know what it is in order to know what problem you’re seeing. You might also want to log out and log back in again (maybe you’re using your memo key instead of your posting key)?
I have only ever used my posting key and it's saved in my password keychain on my iMac. Below is a screenshot of the error. Thank you for taking time to respond to me.
well, thanks for all the hard work
I saw the blockchain down and as part of a witness team my first call was to our tech genius to let him know as well. As we are only around place 100 with @swisswitness we only missed one block but indeed the number of missed blocks is not always a way to see how good one is as a witness. But the pressure of being one of the top 20 is something else but then the rewards are also better...
I guess as witnesses the only thing we have to do is be vigilant and stick together to keep this blockchain up and running.
I saw all this last night, saw the tweet, went to bed confident in those behind the scene to correct whatever it was. Will it happen again? My guess is the back door is being bolted.
I think the whole issue has been seen as positive as the price of STEEM remained consistent throughout the resolution of the situation. This could be used as an indication of the confidence in place for the STEEM blockchain and its governance process. Thanks for the explanation and transparency.
If everyone just traded votes (follow for follow, vote for vote) then the quality of the whole platform would go down. Read the white paper. It should be based on value, not vote trading. You've given me no reason to vote for you as you've added no value to this conversation.
I think these type of events really create strong bonds between all participants. Even though we dislike software bugs and short of sleep, there's nothing better than solving issues as a team and being able to say: "We made it."
Thanks for your dedication and for keeping us posted!
As one of the newer users around here (26 days according to Steemd), it was interesting to see what unfolded here yesterday. I'm still not sure that I understand exactly what happened, but I have been reading everything factual that I can find about it, and your post and video here were a great help to my understanding, @lukestokes.
What I am curious about is how this decentralised platform had a single point of failure that took down all of the Witness servers at once? I thought the idea of decentralisation is meant to prevent such things from happening?
My thanks to all the Witnesses, and to Steem Inc., for getting the blockchain back up and running. It sure looked like a mammoth task, coordinating everyone to update their servers like that.
Decentralized means many individual nodes, yes? If all those nodes are running the same software and they all have the same bug, then they will all respond in the same way. This is by design. All the nodes froze because the transaction that was being processed was invalid and should not have been allowed. So it's not a matter of more decentralization would have fixed this or not. The BitShares chain froze about a year ago also. Bitcoin had a chain fork in 2013 which I remember well. These things happen.
Thank you for taking the time to explain that, @lukestokes. I'm coming from a background of having used (as an end-user only) decentralised social networks like Diaspora* and GNUSocial/StatusNet (and now Mastodon), so I compare it to one of the "pods" going down, but not taking the entire network down with it.
But it seems that what you're saying here is that if all pods were running the same software, and there was a bug in that underlying software, then the whole lot could possibly fail, especially on a sync event, am I right? I guess the crypto side of things here (which neither D* nor GNUSocial etc have to worry about) means that all nodes in the Witness network for SteemIt must be running exactly the same software, and exactly the same version, huh?
Ideally we're all following the same protocol, but there could be multiple implementations of it in multiple programming languages. We're no where close that yet at this point though.
It was very surprising... And there was a lot of people saying crazy things like "Steem is a scam" or "Steemconneeeect" lol. Many of us found out what was happening because of you, for example I spread your message in several communities, thank you for that.
Hi @lukestokes ! I imagined your mind when you got those messages and more over when you saw the issues were things never happened before. Great it was quickly solved by the dev team!
I wanted to let you know about an idea I had, and your thoughts are very welcomed:
Appreciate the straightforward explanation, as well as all your hard work! I can't even imagine what it's like to be the core of multiple social sites like these, coordinated between people all around the globe.
What an intense and draining night it was. So many witnesses had personal problems, traveling, in hospitals. I had a damn leaky pipe at 3am (when I discovered it), luckily it didn't burst. I had to cut off the main water valve for the house, then went back at my computer to follow the situation. Shitty times. Yet, despite everything, we did our best to be around to keep the blockchain alive. Teamwork at its finest.
Thanks for the information. I hope the platform is secure and stable over the long term. Questions are being raised about EOS, but I hope that's just a case of EOS being new with the initial kinks needing to be worked out.
So, the blockchain froze as the consensus was lost. This is a good thing then, which means that it did what it was supposed to do, right? Except the fact that buggy part of the software did allow the input of wrong value the rest went as it is supposed to……………. that’s good to know. Maybe you are one of the few witnesses that I follow so this the only post in my feed that gave me a level of assurance as to what went down wasn’t anything lethal.
Although apart from the general gist, I have really no idea about the technical differences between a replay and a restart. Except maybe that replay lasts for hours while a restart can be a matter of minutes. I have heard that the unlucky ones had to do a replay 😊. I never knew that missing the blocks maybe out of your hands then (I actually thought that it had a lot to with guys being lazy)
Thanks for sharing all this man. I spend quite some time and energy on Steemit so had a bit of scare, not much but it was compounded by the events of my personal life. I hope you drop by my latest post about my experience of the scare.
PS: Nice green hair 😊. Just a few days I ago I gave my first witness vote to you. Second was for @dragosroua. I’ll take a little more time with rest of the twenty-eight.
Thank you for being available all the time everywhere and helping out with information flow. Yes, response from the community in a crisis mode was absolutely amazing.
Thanks for the update. Communication timing is great.
Green is often spill-over from the background being too close or too brightly lit. If you can push it back, that is easiest. Otherwise, lowering the green lighting but keeping the background evenly list is tricky.
Huobi announces suspension of Steem deposit and withdrawal?
Well done, Luke.
You were the first witness I voted for, and I know I made the right decision. Thanks for staying up all night, fixing thing and reporting back promptly.
While your bedroom looks like it just survived nuclear winter, I'm glad the Steem blockchain had a far easier time of it. ;)
you guys jumped in an handled it very well. great attitude! gives comfort to know you guys are out there working so hard for this site. much love
Thanks @lukestokes really useful to get the news update - I've learnt some new bits there - and sounds like a great response to an emergency and it's impressive to get all the block producers onto the rollout so quickly. You have my vote and this update just gives me further confidence that it is a vote well cast. It's a good point about the missed block stats and I will also keep this in mind. Good work on the comms. Great leadership.
Frankly, I was pleasantly surprised regarding how calm and well everyone handled. Nice job!
For several days now I cannot upvote any post or comment without getting an error. Have you seen this before? Do you know if anyone else is experiencing this?
Thanks for the explanation of what happened last night. I though maybe it was connected with the issue I'm having with upvote errors, but it was a bigger system issue.
If you can help me with why I can't upvote, I will then be able to upvote this post!
If you're getting the slider before the error: find something you want to upvote 100%. Drag the slider down a ways and then back to 100, then vote. After that you should be able to vote normally.
If you're not getting the slider I don't know that I can help; when it happens to me I glare at it for a while and eventually get the slider.
It's not related to yesterday's issues; @m-sargon has had this off an on for a month, and it hit my own account once.
Also when it doesn't work on Steemit I've been able to vote through Busy.
Perfect! I'm back up and able to upvote again. Thank you!!!
What error? Would help to know what it is in order to know what problem you’re seeing. You might also want to log out and log back in again (maybe you’re using your memo key instead of your posting key)?
I have only ever used my posting key and it's saved in my password keychain on my iMac. Below is a screenshot of the error. Thank you for taking time to respond to me.
See the reply from @tcpolymath which may help. Looks like it may be related to this? https://github.com/steemit/condenser/issues/2926
Yes, that's the bug I'm seeing.
I was freaking out... not really but didnt made any post yesterday.
well, thanks for all the hard work
I saw the blockchain down and as part of a witness team my first call was to our tech genius to let him know as well. As we are only around place 100 with @swisswitness we only missed one block but indeed the number of missed blocks is not always a way to see how good one is as a witness. But the pressure of being one of the top 20 is something else but then the rewards are also better...
I guess as witnesses the only thing we have to do is be vigilant and stick together to keep this blockchain up and running.
liking the hulk hair though.
Thanks so much for the explanations. This shows how vibrant our community is
I saw all this last night, saw the tweet, went to bed confident in those behind the scene to correct whatever it was. Will it happen again? My guess is the back door is being bolted.
I think the whole issue has been seen as positive as the price of STEEM remained consistent throughout the resolution of the situation. This could be used as an indication of the confidence in place for the STEEM blockchain and its governance process. Thanks for the explanation and transparency.
Thanks for the update on the situation. It's great to see how quick a fix was implemented and everyones funds are safe and the chain didnt fork.
Hey Luke, thanks for your hard work and awesome communication! Keep it up!
Thanks Dan!
I vote for you, you vote for me. I like your post, your post is very good.
If everyone just traded votes (follow for follow, vote for vote) then the quality of the whole platform would go down. Read the white paper. It should be based on value, not vote trading. You've given me no reason to vote for you as you've added no value to this conversation.
Thank you, @lukestokes, I feel better about Steem.it today!!!
Thanks for your unquestioning efforts, Luke!
I think these type of events really create strong bonds between all participants. Even though we dislike software bugs and short of sleep, there's nothing better than solving issues as a team and being able to say: "We made it."
Thanks for your dedication and for keeping us posted!
Thanks Marly! It's encouraging to get such positive responses from people like yourself who have put so much into this platform.
@lukestokes
The freeze scared me for a while, until I realized it was for all
As one of the newer users around here (26 days according to Steemd), it was interesting to see what unfolded here yesterday. I'm still not sure that I understand exactly what happened, but I have been reading everything factual that I can find about it, and your post and video here were a great help to my understanding, @lukestokes.
What I am curious about is how this decentralised platform had a single point of failure that took down all of the Witness servers at once? I thought the idea of decentralisation is meant to prevent such things from happening?
My thanks to all the Witnesses, and to Steem Inc., for getting the blockchain back up and running. It sure looked like a mammoth task, coordinating everyone to update their servers like that.
Let's think about it this way:
Decentralized means many individual nodes, yes? If all those nodes are running the same software and they all have the same bug, then they will all respond in the same way. This is by design. All the nodes froze because the transaction that was being processed was invalid and should not have been allowed. So it's not a matter of more decentralization would have fixed this or not. The BitShares chain froze about a year ago also. Bitcoin had a chain fork in 2013 which I remember well. These things happen.
Thank you for taking the time to explain that, @lukestokes. I'm coming from a background of having used (as an end-user only) decentralised social networks like Diaspora* and GNUSocial/StatusNet (and now Mastodon), so I compare it to one of the "pods" going down, but not taking the entire network down with it.
But it seems that what you're saying here is that if all pods were running the same software, and there was a bug in that underlying software, then the whole lot could possibly fail, especially on a sync event, am I right? I guess the crypto side of things here (which neither D* nor GNUSocial etc have to worry about) means that all nodes in the Witness network for SteemIt must be running exactly the same software, and exactly the same version, huh?
Yep, that's essentially it.
Ideally we're all following the same protocol, but there could be multiple implementations of it in multiple programming languages. We're no where close that yet at this point though.
It was very surprising... And there was a lot of people saying crazy things like "Steem is a scam" or "Steemconneeeect" lol. Many of us found out what was happening because of you, for example I spread your message in several communities, thank you for that.
Thanks for the breakdown/information 👍
Hi @lukestokes ! I imagined your mind when you got those messages and more over when you saw the issues were things never happened before. Great it was quickly solved by the dev team!
I wanted to let you know about an idea I had, and your thoughts are very welcomed:
A Token as SMT for the Steem Ambassador and Promo-Steem Project
Regards, @gold84
Appreciate the straightforward explanation, as well as all your hard work! I can't even imagine what it's like to be the core of multiple social sites like these, coordinated between people all around the globe.