Update after HF21/HF22: September 22nd, 2019 - Witness nodes back online and updated to v0.22.1

Hi Steemian friends! :)

This is my Witness Update, going through all the latest details about my Witness challenges, and for those of you who don't know me yet, here is a photo without my long beard! :)

I am a life long Software Developer, working on Blockchain for last years on several projects, but I am also a DJ and Electronic Music Producer, my alter-ego "Kolatz" that you can browse for if interested... :)

I am the creator of DSound, which was the first and still the only completely decentralized sound platform in the world, that continues helping musicians and sound creators around the globe to monetize their content besides the state of the crypto economy actually, and the next release will be the first "beta" release with loads of new features! :)

If you got curious about DSound, please check it out: https://dsound.audio

Update after HF21/HF22

Yes, this last month has not been easy. Hard Fork 21 came about and with it loads of challenges to all the witnesses as usual! ;)

The update to HF21 was not an easy task, I tried to do it and all my nodes got stuck at the replay stage... Never ending story...

Also it was the first time I decided to experiment with MIRA, the implementation of RocksDB data layer for the Blockchain state, that allows a node to use mostly disk instead of RAM for the state. This was definitely something that I would love to use, since it will free up RAM resources in my bare metal servers to possibly make a dedicated full node for DSound.

MIRA configuration

For a 16GB node I used the default configuration, as suggested by Steemit Inc. My database.cfg looks like this:

{
  "global": {
    "shared_cache": {
      "capacity": "5368709120"
    },
    "write_buffer_manager": {
      "write_buffer_size": "1073741824"
    },
    "object_count": 62500,
    "statistics": false
  },
  "base": {
    "optimize_level_style_compaction": true,
    "increase_parallelism": true,
    "block_based_table_options": {
      "block_size": 8192,
      "cache_index_and_filter_blocks": true,
      "bloom_filter_policy": {
        "bits_per_key": 10,
        "use_block_based_builder": false
      }
    }
  }
}

A special note on MIRA: thank you @steemit for the great work on implementing this. My nodes are running great after I found out the correct configuration (details below). A big shout out to you guys! :D

Steem Blockchain code and compilation

On my nodes I always compile the source code from GitHub.

I think that it is a very important responsibility of a Witness to check the code and the changes introduced in a new version of the blockchain code. I do it all the time an update comes about, and only after that I decide if I will upgrade to it or not.

Then I build it from source and do the necessary actions to upgrade the nodes to the new version built by me, without any docker or images from thirdparties. I like to have my nodes configured by me completely, so I know what is going on at all times.

Steemd daemon and Linux systemd

On all my nodes I configure a systemd daemon for running steemd.

This allows me to login and out without any trouble with screen sessions or needing other thirdparty tools to manage the lifecycle of the blockchain daemon. It also makes the management of Steem daemon as easy as any other service on the Linux box.

If you would like to try this configuration yourself, my steemd.service is like this now:

[Unit]
Description=STEEM daemon
After=network.target

[Service]
Type=simple
User=steem
LimitNOFILE=65536
WorkingDirectory=/home/steem/
ExecStart=/usr/bin/steemd
Restart=always

[Install]
WantedBy=multi-user.target

Hunt for the "Too many open files" limits issue

Thanks to @guiltyparties having a brainstorm about this limits issue, I got in the correct path to identify what was going on... Great help mate! :)

The problem was that, as my nodes are configured with a systemd service to start/restart steemd daemon, the regular session limits are not applied to daemon sessions...

Daemon limits need to be explicitly defined in the systemd service configuration with a line like LimitNOFILE=65536, as you can see above.

So when we use a systemd service to run steemd, all other config files like /etc/security/limits.conf or others are pointless, only the line in the service configuration matters...

This issue took me days in so many unsuccessful replays of the Steem blockchain... uff... ;)

The final glory

It has been a week and all nodes are upgraded to HF22 version 0.22.1, running MIRA and producing blocks like a swiss watch! Great improvement to Steem blockchain! :)

What I am working on

I am currently working on DSound first "beta" release, that will be out soon, and a complete new business model and economy for DSound... Loads of news very soon, so keep tuned my friends! ;)

We are building the evolution of DSound while the crypto winter has pushed away so many great people from our 9000+ community, and also gave us more indicators to re-think our vision, business strategy and the path we want to go to achieve our goals as a team and community leaders...

Many improvements have been made on the social media side and to keep "DSound Tribe" Discord healthy... DSound is always improving and our team is growing! :)

Why voting for @prc and @dsound for witnesses

@prc is my personal account and I run my witness nodes since November 28th, 2017. In these near 2 years only 314 blocks were missed by my nodes, mainly due to a chain hang when I was higher in the witness rank, and our top 20 folks got their big chunk of misses as well.
@dsound is the DSound project account and our team runs our project witness nodes since December 11th, 2018. Since then only 36 blocks were missed by our nodes, mainly due to a problem with our power redundancy in our datacenter. This witness helps funding DSound community initiatives and platform development.
All our servers are bare metal boxes, not VPS's on some thirdparty datacenter. We manage our own datacenters, now with 2 separate sites located in Portugal, and state of the art technology.
We are independent, non-related to any other group of people with any other joint agenda.
Our mission on Steem is to help grow the music and sound creators user base and help onboard more to the platform.
We have been around. Our team has more than 30 years of experience in high availability and security, and we are fully committed to crypto and blockchain since 2016.

So If you trust us and approve what have been doing for the Steem ecosystem, please give our Witnesses a vote to help us continue...

Design by @aleister

You can vote here: Vote @prc with SteemConnect - Vote @dsound with SteemConnect

... but if you prefer the oldfashion way, please go to:
https://steemitwallet.com/~witnesses

and vote for @prc and @dsound ;)

Alternatively you can set @dsound as your proxy, so that you follow the witnesses we vote as well. This might be interesting for all of you that don't know that well for which witnesses to vote... To do so use this field:

Thanks a lot for your support. :)