Lately, a few fellow Steemians complained (to me or to other people) that their post or comment just disappeared after initially showing as posted. It happened on the main Steemit.com website, but also on other channels like eSteem or Steepshot.
To understand why this happens and where is your post, you have to understand how the entire Steem ecosystem works.
So, first of all, just to get on with it: if the post /comment disappeared, it is lost. Forever. And ever and ever.
Second, this is how things are working in the Steem ecosystem:
Everything Is A Transaction
All the operations in the Steemit ecosystem are , in fact, transactions. Every post, every comment, every vote, every power up or power down, every transfer, everything. These transactions are not saved in a central database, like it happens in a traditional, database-driven website. Nope.
These transactions are packed, signed (with your posting or active key, so everybody will know they belong to you) and then broadcasted. Like a radio is broadcasting waves to all potential receivers.
This is where it gets interesting.
Every 3 seconds, a witness in the Steemit ecosystem is picking up these transactions, verify them and then pack them into a block. This block is then pushed in the blockchain, where it stays forever. If your post was picked (which happens in 99.99% of cases) it will then be available in the blockchain forever, at the specific address of the block.
If you happen to run a witness node, and if you look at the logs, this is what you see (edited for brevity):
handle_block ] Got 23 transactions on block 13532820 by timcliff
handle_block ] Got 39 transactions on block 13532821 by clayop
handle_block ] Got 28 transactions on block 13532822 by pharesim
handle_block ] Got 29 transactions on block 13532823 by timcliff
handle_block ] Got 33 transactions on block 13532824 by klye
handle_block ] Got 38 transactions on block 13532825 by jesta
Now, if your transaction (being it a post or a comment) didn't make it in any of those blockes, it won't be picked up again. That's what I mean when I said it's lost.
So, you see, this is not your average website. It certainly looks like an uglier reddit, but it's so much more than that.
There are many other things to be said about how witnesses are rotating and what incentives they get to support the system, but it will get a bit complicated, you can have a look at some of the posts at the end of the article where I describe this in a bit more details.
So, Why My Post Didn't Make It Into The Next Block?
Well, there can be many causes:
- your connection dropped before the transaction got broadcasted
- the client didn't properly signed the transaction, so it got rejected
- or, the most common: the witness responsible for picking up that transaction wasn't working
Now, the third one is the most common cause because the infrastructure needed to maintain such a big network of witnesses (at all times there are around 100 witness nodes working) is very, very difficult to control. Sometimes the software crashes, sometimes there's a network outage at the hosting facility (it suffices to be just as little as 2-3 seconds, and the block is lost), or the witness is having some internal hardware difficulties. The good news is that the whole thing is self-healing, every hardfork brings in stability fixes or improvements.
And that's the connection with the witnesses that I was talking about in the title.
If you want to have a healthy Steemit ecosystem, you should be aware that everything is relying on them. Your posts, your votes, your transfers, your power ups and downs, everything.
So please pay attention to the following metrics when you vote for a witness:
- how many missed blocks? (you can see that in any decent block explorer, like steemd.com or steemdb.com)
- how involved the witness is in the community (chances are that any witness will solve any technical issues faster if he's still involved with the community)
- how versed the owner is in the technical details (follow all the witnesses and pay attention to their regular updates)
- how often a witness is updating the community with his activities (follow the tag #witness-category)
Remember that a big healthy network is made of many small healthy nodes, connecting to each other.
I'm a serial entrepreneur, blogger and ultrarunner. You can find me mainly on my blog at Dragos Roua where I write about productivity, business, relationships and running. Here on Steemit you may stay updated by following me @dragosroua.
https://steemit.com/~witnesses
If you're new to Steemit, you may find these articles relevant (that's also part of my witness activity to support new members of the platform):
Great post as always. Very informative. You're always on the money and why you got my vote for witness. Keep up the great work!
Thanks :)
Very interesting thanks for clearing that up for me. I will def be checking for missed blocks when voting for a witness next time. Great post!
Great post! It's very important to consider the best witnesses. I have dedicated my time and money to steem for well over a year now, and I am finally witnessing for both steem and peerplays.
If you haven't already, please show some love by voting for my witness! You will have to type in "tyler-fletcher" and click vote!
My peerplays witness is "firecrab-witness".
Thanks!
THANK YOU so much for your article! I experienced some missing 'transactions' yesterday and @creatr shared a link to this article to help me understand. I get it now :)
Thanks for the information, I think now Steemit is becoming more familiar with the general public there will be more childhood diseases. greetings from Belgium
This is a truly bizarre sentence, I'd love to hear if it was intentional or some kind of beautiful typo.
Looking at their profile, they don't seem to be some sort of bot. Truly truly bizarre! I'm guessing maybe autocorrect?
Sorry for the bad translation, startup problems with a new system, we call 'childhood diseases' it's kind of saying over here. greetings from Belguim
Haha yea must've been an insane autocorrect typo
The wide world of the interwebs and bizarre comments Hey Matt Sokol
LOL
I still love your user name so much, it brings me much much joy every time I see it.
you're a crazy guy barry O_O but I'll take the compliment!
Sorry for the bad translation, startup problems with a new system, we call 'childhood diseases' it's kind of saying over here. greetings from Belguim
Ah cool, now I learned something! Thanks for coming back to explain.
haha this comment though
Sorry for the bad translation, startup problems with a new system, we call 'childhood diseases' it's kind of saying over here. greetings from Belguim
Hi @dragosroua, thank you for expanding my noob understanding of the inner workings of steemit a little bit further.
One quick noob question - is there a limit to how many transactions can be held in each block? Can the system get overloaded?
Nope. There is a link between the block size and the number of transactions, obviously, but witnesses are monitoring this and they will increase the size as needed. The system can handle dozens of thousands of transactions per second.
Thanks for that extra bit of info. Gradually filling the gaps in my knowledge.
This happened again to me today..... and I see more of it last week too, it was pretty good for the last few months!
Great post, top to bottom, @aunt-deb sent me here!
Please explain this then:
More than once, I have posted a comment and had it appear to be visible under a post, i.e. it appears to have been "accepted" by the network...
Then, later on, I go back and it is gone!
Based on the combination of your article and my observations of this phenomenon, would it be true to say that the Steemit User Interface may temporarily display a certain state before it is posted to the blockchain? And that, afterwards, if the block is "dropped" that apparent state is reversed and disappears?
Thanks for the clarifying description... ;)
😄😇😄
you are asking the right questions!
@dragosroua thank you so much for clarifying this issue. It happened to me too however I have a better understanding now
This makes sense except for a couple of things.
First the post I just posted... it actually showed up in my blog feed for a few seconds and then disappeared.
When I refreshed the browser it was in my feed again for a few seconds.
When I refreshed it again instead there was a resteem icon with not title image or anything for a few seconds...
and then it was gone...
Correct me if I am wrong but nothing should show up in my blog feed unless there is actually something there for a moment.
It seems to me more like it WAS accepted by something for awhile and then a consensus check decided it was invalid and tossed it.
So, that's why my post disappeared yesterday. Some "witness responsible for picking up that transaction wasn't working"
That comes to saying that there might have been a short outage, which translated into a consistent latency (>2-3 seconds) at the hosting facility. That's the most common reason for missed blocks.
That's why i'm using external markdown editor with saving option. Also i think this could be handled by Steemit, system should try to repost once again.
Thank you @dragosroua, now I have a perfect explanation why my comment to this very post 15 minutes ago is missing, that has never happened to me before... Ironic, huh?
This is a great explanation!
The day before yesterday I mysteriously lost my post. That is, it even reflected in the "New" tab, but when I clicked on it, I got out a record that such a page does not exist!
Do you have any proof of reversible transactions actually happening?
Yes. I don't know what you mean by "reversible transaction" but I published a post today, saw it for 3-4 seconds on my own feed, but when I reloaded it disappeared. For me it means it wasn't on the "last reversible block".
Yeah, I've heard this story too. But for all I know, it's entirely a UI bug. I have no evidence that these transactions ever got accepted by a node, then "lost." But I could be wrong.
Good explanation, Thank You, Had to redo mine about a week ago , and as I do them off the cuff, not exactly the same!
Sir @dragosroua, thanks for the information.
I was curious why does it happen. And my friends asked me about it a couple of times.
this happened to me yesterday , was writing about crypto in my spare time .. 3hours gone down the drain
Never thought the blockchain is this complicated... (pardon the ignorance). Great to learn something new everyday...
These posts are a great reminder for me to keep paying attention to witnesses. It isn't enough to vote once and forget - any passionate Steemian should probably set a calendar reminder once per month to check up on it and see if all the votes are going to good people.
Easier said than done... but I'll try to keep up with my duty to be an informed voter :-)
Thanks for the detailed explanation. Speaking of which, I made a comment just a few minutes ago and it's gone. That's the second time this happens to me. Good thing I wasn't receiving 100000000 STEEM from @dan lol.
:))
Thanks, appreciate you giving deeper insight. Consistently good information!
Excellent post. I asked a question related to this just last night and someone directed me here. Good stuff. Cheers. Are you a witness?
Yeap :)
Same happened to me yesterday @dragosroua
I think one should save the blog somewhere on their PC so they can post it again. I was just working on steemit, and when I posted, it just disappeared, and I had no way to repost it.
Good information and very well-written. I will definitely keep this in mind next time I will be voting for a witness :] Upvoted and RESTEEMED
I
I had a bit of a different and strange experience.
I made a long post and it appeared on the platform. I went back to fix a spelling error but after I did, the post got replaced by some strange symbols and the old and new spelling. I guess what was recorded was just the info about the edit, but not the original post somehow.
I tried to salvage the original version through steemdb, but didn't manage to do so. In the end I wrote the post again (hopefully a bit better).
Fantastic explanation, thank you!
I too experienced a lost post ( hopefully I was able to recover it, because I wrote it using word ). This article is pure gold ! Thanks
That was a good tutorial of how steemit works, learned something. Thank you for sharing :)
@dragosroua your post leaves me hoping that the Steemit community will put more thought and effort into selecting Witnesses than most people put into choosing a political candidate.
You've given me insight into how important Witnesses are that I didn't have before.
Hopefully, there are enough members who also understand that Witness voting isn't just another popularity contest.
Upvoted, resteemed, following and I'm also bookmarking this post so I can refer back to it later.
Oh, yeah and you've got my Witness vote too @dragosroua.
Upvote for the info dump and easing my sanity a little. I've had this happen about 3 separate times in the last few days. Could it be a symptom of a bigger problem? Overloading? I hate the thought that comments that I type are being lost and I not know unless I start going back to check myself. :-(
@dragosroua - this is the best explanation I've seen. Resteeming.
I was wondering why some of my upvotes and comments had disappeared. I had attributed it to having a flaky moment. Now I know...
I'm still sane! bwahahaaa
Exactly - I had this happen to me last week with a comment I posted, then the chaos that came after when people said they got a message that I was not a valid steemit account when they tried to reply back to me. Now I know why, even if I don't totally understand why. :-)
Thank you @dragosroua for the explanation on this, I had it happen to me last week. My comment was there, then it wasn't. Other members tried to reply to my comment and it gave an alert that I (the author) was not a valid account on steemit. It caused a bit of confusion for a while. Now I understand a bit more of how/why it happened.
@dragosroua, thanks for the detailed explanation... it's really the perfect response to my question-post from a couple of days ago. It's nice to understand how the underpinnings of Steemit really work.
A semi-newbie question: The fact that these "drop outs" are happening more often, is that the result so many new members, so many more transactions? Would it be solved by adding more witnesses, or is this issue not related to overall network capacity?
It's a process of adjusting. Increasing block size also helped. I don't see it quite often lately so it means the network adjusted.
Thank you for this valuable information. I will have it in mind next time I vote for a witness.
I had no idea that this was all so complicated...and simple. I'm one of those non-tech users of the platform and not very computer literate.
Thank you for this post, it gives me a better idea of what's going on in the back end here. It also drives home the importance of people such as myself to take the time to at least get a rudimentary understanding of the mechanisms involved...and the importance of their vote.
Thanks so much...glad I'm a Steemit feed surfer...Feed me more!
UV/RS & following
Now that I'm actually thinking about this for the first time.
Is there any way there could be an automated filtering process that would omit human verified serial scammers, spammers and undesirables at each node?
A regularly updated 'shitlist' if you will. Not sure it will help with the disappearing activities but it may help prevent bloat on the blockchain...or however you choose to term it.
A simple no or non answer will suffice if this is just impossible. ;)
Just a thought...peace out...in
check out @steemcleaners, they're already doing it
So if the new post is broadcast out to several witnesses, why is it that only one of them catches it and posts it to the blockchain? Isn't it more blockchainy for several of them to do so and then build consensus on which version of the post is correct, or am I just completely misunderstanding blockchain tech here?
Nope, you got it right, but the way witnesses rotate is a completely different thing.
The top 19 witnesses are rotating faster than the rest, they get scheduled (pretty much) every minute, then the runner up witnesses get scheduled in a decreasing algorithm. A witness in the 50 position signs about 20 blocks / day. This is all written in the blockchain logic It makes things way faster.
Thanks for your clear explanation -- but it sure raises a lot more question for me! The value of Steemit, from the white paper, is supposed to be in the speed and reliability of the blockchain. If it's not meeting expectations, that would be an issue.
I see comments all over the place about failed transactions recently. It seems like much more of a problem than it used to be. Is there any sense of why this is happening more frequently in just the last week or so? Is anyone even tracking how often this is happening? Are more witnesses failing on their end, with underpowered equipment? Is the system up against the limits of its transaction processing capabilities?
Thank you for the info, so interesting, upvoted and followed 😁
Great and intetesting info and it did happen to me a few times that I wrote a comment and then it was gone. Big deal I did it again and there it was :0)
I wondered why bit it didnt bother me.
But now I know so thank u so much for clearing it up.
Good explanation!
Was wondering if I could translate your post to Spanish. People in the spanish chat are also questioning this with no real answer. Would be a good info. Of course with a link to this post and informing it is a translation....as usual if I do a translation :)
One example here https://steemit.com/spanish/@trans-juanmi/el-fin-de-la-propiedad-intelectual-en-la-imaginacion-la-inteligencia-artificial-y-la-generacion-por-procedimientos-traduccion-de
this is quite interesting. thanks @dragosroua
Brilliant post. Thanks!
click me up