To fork or not to fork? Is that even a question?

As I've posted about many times, I'm a big fan of DPoS and on chain governance. I've given talks at conferences about DPoS and plenty of interviews, some of which you can find here, which include DPoS discussions. This governance model allows token holders to support witnesses directly and through them, vote for or against proposed code changes.

In my opinion, this is a much better governance approach than many other consensus algorithms. Bitcoin, which uses PoW (Proof of Work), only has a user activated hard/soft fork concept which, in a sense, holds the block producers hostage using full nodes. We're much more civil here. :)

That said, I wanted to explore an often mentioned criticism I hear when I attend conferences or talk with people who are deeply entrenched in other cryptocurrency projects (something I often call crypto tribalism). They talk about Steem, EOS, and BitShares being centralized (which is essentially a critique on DPoS itself). Given how many independent block producers and witnesses involved in creating consensus compared to a limited number of mining pools on other chains, I don't think this is a valid criticism.

What may be a valid criticism is evaluating where the code comes from.

When it comes to technology, whoever ships the code defines the technology. When there's a disagreement about something, the code wins. Usually, the concept of Code is Law rules the land (meaning, whatever the code does is the law, even if it was a bug). EOS is exploring ideas such as intent of code is law via Ricardian Contracts and arbitration, but that's still at very early stages and makes many in the blockchain space very uncomfortable.

Here on Steem, most of the major code changes come from one organization: Steemit, Inc. They are the for-profit entity behind this blockchain. In a sense, they are a centralized creator of the code which defines the blockchain. EOS has similar centralizations with Block One creating the code (or contracting firms such as Object Computing to create the code). In situations like this, when a hard fork is proposed to the community, witnesses and token holders have a limited number of options:

  1. Do not upgrade in support of the Hard Fork until changes are made.

    In the short term, this essentially maintains the status quo and does not improve the blockchain for future growth and performance. As you can imagine, this is usually a very unpopular decision for a witness to make and it can only be made effectively if there is a lot of support and consensus by token holders to keep that witness active. The fun thing is, this actually happened with Hard Fork 17. It had changes not everyone agreed to and was postponed until Hard Fork 18.

  2. Test everything you can, do a code review, voice your concerns (if any), and go forward with the planned upgrade.

    This approach, it seems, is the most common one. Does it mean witnesses are "rubber stamping" anything Steem, Inc. creates? Does it take anything away from the effective governance of Steem DPoS? Do we have any guidelines for what testing is done and how much code review is enough?

When witnesses go with option 2 and things break (complex software almost always breaks, it's just a matter of how often and how badly), it can be tempting to blame Steemit, Inc for putting too much into one change. It can be tempting to blame witnesses for not going with the very unpopular option 1. I've been talking with a number of top witnesses over the last few days about this hard fork and there are certainly some concerns.

  • The new resource credit system I mentioned in my last video is a huge change and requires some time to reach equilibrium. As the latest blog from Steemit, Inc. explains, we may have to adjust our expectations.

  • There's been some concern about the lack of tooling in place for witnesses to set the new properties needed to adjust the resource credit system (the functions are not available within cli_wallet). Thankfully, literally hours ago, Beem for Steem created and maintained by @holger80 figured out a way to properly updated the parameters.

  • The Steem Testnet is a huge win for the Steem ecosystem! That said, I get the impression not that many witnesses have really interacted with it extensively. The account creation fee is currently set to 0 which makes testing certain aspects of the resource credit system related to account creation difficult. It also started with some distorted resource measurements due to the way the content from the live chain was ported over, making testing of the resource credit system under real-world conditions quite difficult.

  • There are a lot of changes in Hard Fork 20. A lot of changes. This is a tricky balance because in order for a vote for or against a hardfork to matter, it has to be a vote on a single issue. If, for example, 10 important things are included, but 1 is contentious, it's difficult to hold back innovation on the 9 things just to make a strong point about the 1 thing. Some argue we should have more frequent hard forks specific to individual issues. Others have concerns about too many hardforks causing too much maintenance demands on exchanges who then would have to upgrade their Steem nodes on a regular basis. If that's too much work, they may consider it easier to drop support for Steem/SBD.

Are these concerns enough to prevent us from moving forward with Hard Fork 20 and working through any issues we find along the way?

I don't think so, which is why I'm currently signing on a v0.20.2 node. I have a backup on v0.19.12 just in case everything breaks.

This is probably a good time to outline my standards for upgrading to code supplied by Steemit, Inc. (or any other developer, for that matter). Ideally, I'd like to have more contributors to the code than just those employed by a single company and thankfully people like @timcliff have been doing just that. That said, with more decentralization comes more coordination costs. So far, I think Steemit, Inc. developers have only put forth code which is in the best interest of the Steem blockchain community. So here are my current guidelines for if I go with option 1 or option 2:

  • Code review the changes being deployed on Github. Look for anything obvious/confusing/concerning and ask questions of the team as appropriate. As you can see with this comparison between v0.19.12 and v0.20.2, that can be quite difficult to do all at once as this change has 2,657 changed files with 569,910 additions and 31,139 deletions:

  • Run the code on a seed node for a period of time and monitor the logs closely. Are there any changes, issues, or unexpected problems? If so, discuss with the other witnesses, community, and Steemit, Inc. developers and create Github issues as needed.

  • Run the code on a backup witness node that is not actively signing blocks. Again, monitor the logs and performance closely.

  • Now that we have a testnet, I'm also including running some tests for existing functionality against the testnet via my own node and through the testnet condensor. For Hard Fork 20, I also created a new account lukestokes.test, configured it as a witness, and signed some blocks.

  • Test new functionality on the testnet (such as the new rc_api plugin in HF20):

  • Ask application developers if they've tested their applications against the changes, and if they are comfortable with those changes. I could certainly do more of this and if you're a Steem application developer reading this, please know I'm always available to you on Discord, Telegram, Steem.chat, and more.

  • Discuss with other technical witnesses their own testing and confidence level with the proposed changes. Since DPoS requires consensus and we all want whats best for Steem, it can be helpful to get multiple perspectives and insights to inform your own opinion.

  • Ask questions of the Steemit, Inc. development team to determine their confidence level and figure out if there are specific areas of the code which they may be most concerned about (and which I should focus my testing).

So those are my standards as of right now, and they adjust based on the complexity of the code changes. Are they enough to catch every possible potential problem? Of course not. I've been doing software development since 1996, and I know, there is no such thing as perfect code. Sometimes you do the best you can, deploy, iterate, and improve.

I'm posting this right as Hard Fork 20 goes live in the code. That means a whole bunch of code previously hidden behind an "if HF20 = true" check will now run. What will happen? Let's find out.

Edit: Right as I tried to post this entry at 10:00am Central Time, I was greeted with this message:

Checking my account, it seems I have negative resource credits!

curl -s --data '{"jsonrpc":"2.0","id":1,"method":"rc_api.find_rc_accounts","params":{"accounts":["lukestokes.mhth"]}}' localhost:8090 | jq -r
{
  "jsonrpc": "2.0",
  "result": {
    "rc_accounts": [
      {
        "account": "lukestokes",
        "rc_manabar": {
          "current_mana": -1384119289406338,
          "last_update_time": 1537881810
        },
        "max_rc_creation_adjustment": {
          "amount": "10000",
          "precision": 3,
          "nai": "@@000000021"
        },
        "max_rc": "57193602393"
      }
    ]
  },
  "id": 1
}

It's going to be an interesting day. I'm trying to post again from my witness account.


Luke Stokes is a father, husband, programmer, STEEM witness, DAC launcher, and voluntaryist who wants to help create a world we all want to live in. Learn about cryptocurrency at UnderstandingBlockchainFreedom.com

I'm a Witness! Please vote for @lukestokes.mhth

Sort:  

Hey, @lukestokes.mhth.

Glad you were able to post this, and it's also good to know how you walk through the process of witness decision making.

It would seem to me that a balance could be struck between a massive hard fork and incremental ones, where maybe they happen once every six months, or if that's too long, once a quarter. I've tried to figure out when the first 19 took place, but it seems to me they all happened within a year or so, or at the very least, the majority did. That would seem like a lot of updating to do, although it may have been necessary given how young the blockchain was and how much may have needed to be changed as different issues and situations came up.

Now that so much time went until this hard fork, it would seem to me a happy medium could be struck somewhere in between, so that there's not so many changes to reach consensus on, but not such a need to update all the time.

I've been among the many who haven't been able to do anything on the blockchain, so just commenting here finally feels like an accomplishment. Not sure how many more I'm going to be able to do each day, so I'll need to watch all that while the equilibration (or whatever) continues to take place.

Yeah, I think waiting a year is far too long. Maybe quarterly is a good approach. I don't have direct relationships with exchanges to know what they are willing to accept which, to me, seems like the main limiting factor with required upgrades. Hopefully the modularization of appbase allows us to make changes without exchanges needing to do a full replay or upgrade. We'll see, I guess.

I would hope that what ends up being good for the STEEM blockchain, ie, hard fork or updates of any kind, would not be impeded by exchanges, but that it would be considered a good thing by them too. I don't know. There should be a happy medium between massive hard forks and tiny ones where all are fine with the replaying. If appbase is the answer there, wonderful. There just seems to be one thing that prevents something from happening. Is that one thing that stoic or important, or can it adapt, modify behavior, whatever? I don't know. Honestly looking for solutions and answers that don't end up with ill will and most people locked out of the system, or so limited that they might as well be.

test HF20 comment

apparently it works ;)

It'll be unrelated to post so, sorry about that. But you should be more active on radio shows/talks etc. I just listen you on Utopian Open Source Radio Show, and there goes my little witness vote.

ps. you possibly active enough but this was the first time I heard what you do and think about what should be done in general.

Thanks. I've been on quite a few Discord shows, but I'm also building stuff. Sitting around talking and educating people is great, but we also have to balance that with building stuff. :)

I certainly agree with your options and the approach you have decided to take but the question remains how many other witnesses have a similar path and how much do people who vote know this about their witness votes to ensure that they are taking their responsibilities seriously regarding governing the blockchain. The fact that the Steemit Blog pointed the Resource Credit issue out the day before Hard Fork 20, shows that they had some concern regarding the implications and I think that they failed both the witnesses and the community by not expressly saying what was going to happen. We may not know if they knew all the details but I feel they did not clearly state the impact this was going to have to the majority of the community or else I think it would have been handled differently as the churn I expect from this issue will impact the activity in the ecosystem.

I agree. If you see my comment on their post, I tried to clarify that what they meant by some disruption really should have been "you'll see errors and the site will be unusable." I did a video trying to help people prepare, even before Steemit's announcement. I figured the Resource Credit system would have some initial issues, but I didn't imagine it would completely break the site. I think we're all going to demand more testing in the future. I hope we all participate in it as well.

Hello @lukestokes,

You are one of my Steemit witnesses, and as such I hope you'll pardon this off-topic contact.

I'm here to inform you of a fantastic opportunity to promote STEEM.

Please check out the information in my article:

STEEM On Netcoins Contest: EVERYbody Wins!

I'm not looking for your vote or reSteem...

I am asking you to take action to promote this!
It seems clear to me that it will benefit the entire Steemit community.

Thank you in advance for your consideration!

😄😇😄

@creatr

You do tend to drop in with unrelated comments, huh? Heheh. Looks interesting. I went ahead and voted, but I am highly skeptical at this point that any mass adoption will come through that route. I spent about 5 years trying to get merchants to accept cryptocurrency via my company FoxyCart, and it's just too early. The protocols and user experience just aren't there yet.

Hi Luke,

Thanks for taking a positive, even if measured, response.

I understand your skepticism; however, I feel we need to keep chipping away at the avenues that are available. This particular approach seems promising to me because I have, at one time or another, tried pretty much all the others.

I realize that Netcoins is focused on Canada right now; perhaps they'll never make it to the US. Even so, for the sake of every Canadian Steemian (and I know many personally) it warms my heart to think that they one day may be able to walk into a local shop and spend their STEEM, or cash it in for fiat to pay a water bill...

I'd love to hear or read more of your FoxyCart story; have you written about it on Steemit? If you have, or if you do, I'd appreciate a link.

On the Netcoins thing; earlier today, STEEM was ranked 11th in that contest. As of this writing, it is now 5th, thanks to the efforts of Steemians rallying to the cause. And so, I'll close with this observation: If STEEM gets a (reputedly) $30K account with Netcoins for free, hey, it sure won't hurt anything, and it might just help... :D

I'd like to see STEEM have that chance. ;)

Thanks for not throwing me out on my ear for contacting you through this "side channel." :D

No worries. Thanks for working to spread the good word and help promote STEEM. You can read part of my FoxyCart story here and how I turned my code into a company here.

Hi @lukestokes.mhth
Just decided to see how have you been doing, since I didn't hear from you in a while. I noticed that you didnt post anything in a while.

Did you give up on Steemit? :(

Yours,
Piotr

Congratulations @lukestokes.mhth! You received a personal award!

Happy Birthday! - You are on the Steem blockchain for 2 years!

You can view your badges on your Steem Board and compare to others on the Steem Ranking

Vote for @Steemitboard as a witness to get one more award and increased upvotes!

Congratulations @lukestokes.mhth! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s) :

You got more than 10 replies. Your next target is to reach 50 replies.

You can view your badges on your board And compare to others on the Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP

Do not miss the last post from @hivebuzz:

HiveBuzz supports the Austrian Community Meetup