A follow up on: HF20: Why STEEM software development process is broken and why to fix it NOW!
Now that HF20 seems stable and mostly everyone is back on business, can we finally talk about preventing another traumatic hardfork?
How about implementing a working Testnet?
I'm not against it but some people stated (some top20 witnesses included) that it is almost impossible to fully test a hardfork upfront, and I want to say contrary, it is perfectly possible and we can start right away.
Really?
Yes!
Show me the way!
OK... let's see. I'm looking for the right language to explain myself. I talked a lot about cows in the past, but we need something more attractive...
Let's talk about Italian luxury cars!
No, no, no, no, I don't like velocity that much (it remembers me of HF20), I prefer to build something for mass adoption... I have it on the tip of the tongue.
Tesla or Toyota, both are lean
@ned gently answered one of my provocative comments the other day. There he confirms (in a very businessman speech style) that he likes the agile approach. That was enough for me, I already know what to expect from agile companies and agile teams. The infamous release of HF20 is a perfect example. So, I quickly asked for a Lean approach (without mentioning it of course) but I'm still waiting for an answer. Full story here: https://steemit.com/steem/@develcuy/re-ned-re-ned-re-sapphic-re-ned-re-ats-david-re-steemitblog-hf20-update-restoring-continuity-20181001t004050638z
Getting back to cars, have you ever heard of a car manufacturer stating: "We can't test this car before mass production". That would be nonsense don't you think? But, how things work in software, are things really that different between manufacturing and software development?
What is Lean in the first place? Oh wait! If you ignore what it is and you happen to be a witness, a STINC/Steemit Inc. staff or @ned, please educate yourself: https://www.lean.org/WhatsLean/
lean means creating more value for customers with fewer resources.
Lean.org
Resources are always scarce, then it makes sense for mostly all companies to at least learn about Lean and to evaluate whether it is the right strategy to adopt Lean.
Guess who invented Lean... Toyota. Guess what is Tesla's manufacturing approach: Lean. Should I say more?
Testing a car before mass production
By the way, nothing is perfect, a Lean company might fail:
We had better get back to the roots, let's build the Toyota way:
The right approach to test future hardforks and softworks: Simulations (or fake steem)
Straight to the point this time. My proposal is to launch a fake-steem, where everything is obviously a simulation. Let's prefix everything "fake-", so that we have: Fake-STEEM(fsteem), Fake-SBD(fsbd), Fake-witness(fwitness), Fake-market(fmarket), fake-steem.com, fake-steemconnect.com, fake-you-name-it. If we make everything functional, we can have a working simulation, where we can freely experiment without the worries.
Making this to happen is not difficult, it just a matter of having the right people willing to do it. But not a hard deal.
Don't copy steem blockchain, just start fresh on every hardfork
Look, I know that maintaining an steem node takes time and that the blockchain has a fair amount of gigabytes, but we don't need to copy the current blockchain with all its transactions. If things are well coordinated, we can have an scheduled launch date for fake-steem HF21. We have a good crowd of spammers and noisy stars in this platform, let them own the testing accounts, they'll generate good noise so that the crowd of STINC along the top20 real witnesses can squash some bugs on softforks and get the real HF21 ready for a smooth release.
Not only the above, there is other approach that would work in parallel: Artificial Intelligence. How about training bots to pull content from real steem? They could generate acceptable input for testing. Also, let the general public to register in fake-steem, so that anyone can contribute real content.
Who will pay the bills?
Ok, running such infrastructure is not that cheap, but not so expensive either. I think that STINC should control fake-steem.com and fake-steemconnect.com. It requires some decent IT infrastructure as well, but they know how to deal with it already.
Testnet vs. Fake-steem
This is most tricky part of my proposal. Both concepts are pretty similar but I want keep the concept of simulation as an invitation to create an alternative space where everything works exactly the same as the real one, except that everyone is aware that it is a fake environment. When manufacturers want to create a new car model, they design it, build it, test it, they find defects and opportunities for improvement, then they design a second version, build it, and so on. At some point they are happy with what they have and order mass production. That is what we need so we should do it, no reason to wait any longer.
What would you publish in fake-steem.com?
Let me answer first: I would copy cat and yell a lot. But I'm definitively going to use it.
How about you?
Interesting points @develcuy! In an MMORPG that I used to play a long time ago, they had separated servers, where they could test new features and fix bugs before rolling these changes to the players. When you see how GNU/Linux Distros work out, they test extensively the software before deploying anything to the public. Both are examples of the same process in a low and a large scale organization to guarantee QA in their software.
I can't understand why Steemit heads refuse the proven methods and good practices in Software Development.
I agree with you a 100% and if Steemit needs more resources to implement such system, they should take a little bit of the pool to finance it, or at least study the way they can do it. It would do a lot of good to have a system like that in place.
Thanks for sharing @develcuy!
This is a good idea, but reseting the blockchain state has some issues:
BTW replay attacks are protected against already. Chain IDs prevent them.
@smitop, excellent food for though! You are tackling two very important issues:
What you describe as far as the testnet was pretty much what was done for hardfork 20. We had a working testnet that was a clone of the mainnet, did not have the full historical set of data (to keep the block_log size low), and was even getting a copy of the new mainnet transactions mirrored over in real time. There was also a testnet condenser: https://condensertestnet.steemitdev.com.
Yes @timcliff, you are right, so I should elaborate my proposal a bit more. To be more specific, I'm against straight dumping content from mainnet into the testnet, that is why I suggested AI and I should add statistical analysis as well, which should mimic human behavior with an smaller database of content "recycled" (statistically down-scaled) from the mainnet. In means that we should end up with a lightweight fake steem blockchain that can still provide meaningful feedback. I know that it might sound very complicated, but let the big data experts debunk this alternative. I still prefer to let the community play with fake steem freely, AI can come later.
I get what you are going for, but I still don’t think it would generate the conditions we need to simulate a “real” test.
The community had access to test hardfork 20.
How was this done ? Is there any code or documentation available ?
The official Steemit, Inc. testnet was originally announced here:
https://steemit.com/hardfork20/@steemitdev/hardfork-20-testnet-details
It bootstrapped and activated HF20 the same day.
There were specific things that we were looking for with the original testnet deploy. We wanted to know how it transitioned from HF19 to HF20, and that looked good. We wanted to know how transactions from mainnet copied over to testnet would fair, and that looked good too:
https://github.com/steemit/tinman#gatling-transactions-from-mainnet
One of the things we could have probably tested in the month prior to HF20 was to delay transition between the bootstrap of the testnet on HF19 to HF20. Maybe a 5 or 7 day HF20 activation delay would have revealed a hint about the difficulty in the new features, but we didn't test that scenario.
So, there was possibly a lack of imagination. But at least we have that knowledge moving forward.
That's pretty much what we do by using a
chain-id(without the prefix, just using different asset symbols instead). We use a uniquechain-idfor the testnet so that transactions cannot be mixed up between blockchains. Have a look:https://condensertestnet.steemitdev.com/
Indeed @inertia, the concept of testnet is very similar but it still needed some adjustments IMHO. More than the technical aspects, my proposal is about how to involve all the steem community on the testing process. That is even why I use an easy word for the non-tech savvy: "fake". Also, we should actively invite people to participate in testing, don't let it pass unseen, it should be a testing party or festival actually (easy words again). Otherwise, it makes no sense to open source the project, let's make it easy for regular people to contribute something! Testing is a great opportunity.
Oh to that end, there's also talk of adding a financial incentive to use the testnet (one or the open issues in that repo). But that would be implemented by the community, if it happens.
Thanks @inertia, this will help us a lot. Working on creating public environment for testing hard forks with @bobinson and on ways to motivate steemians to help with testing.
Thank you @inertia and @timcliff
@inertia : Tinman has under gone lots of changes from the last time we were using it! We had reported an issue in May ( https://github.com/steemit/tinman/issues/59 )
Syncing the transactions is a wonderful feature you guys have added. I will revert with comments/doubts after going through the all "wizarded-up Tinman! "
@intertia we have few doubts and giving them below. I hope this will help to do a RCA.
cc: @timcliff @ura-soul @yuriks2000 @yehey @develcuy
Questions
Date on which TESTNET was live ? ( 2018-08-25 as per the @steemitdev post)
what is the version of the offiical TESTNET ?
How many transactions are happening in the MAINNET per week ? (7 x 86400 seconds)
How many transactions happened in the official TESTNET ?
What was the highest TPS on the ofifcial TESTNET ?
Can we create scenarios like upvote which will test the official testnet ?
Do we have a list of scenarios and transactions that cannot be effectively tested on the TESTNET ?
so can we identify the difficult areas like RC and test them ?
Which are the difficult scenarios that was not tested by the automated test environment ?
See: https://steemit.com/witness-category/@inertia/re-bobinson-transaction-generation-on-the-steem-testnet-step1-root-cause-analysis-20181006t180941432z
I don’t know much about it other than what is here: https://github.com/steemit/tinman
@inertia may be able to explain more.
Thank you @timcliff Do you think the witnesses can help to increase the number of users and test cases + code coverage on the official TESTNET ?
We are planning on that and a lot of other things for the next hardfork.
"We" as in ?
Also, I hope you have seen this work ( https://hackmd.io/s/ByT1BuG5m)
the witnesses
yes
Working on this kind of environment right now with @bobinson. Everyone is welcome to join and help. Thinking on ways to motivate testers/steemians.
Will write about it in more detail soon.
Congratulations @develcuy! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :
Click here to view your Board of Honor
If you no longer want to receive notifications, reply to this comment with the word
STOPCongratulations @develcuy! You received a personal award!
You can view your badges on your Steem Board and compare to others on the Steem Ranking