Hello fellow Hive community!
I am very excited and proud to announce that after unfortunately missing my first two potential blocks to be signed, as of about 7 hours ago I signed my first successful block as a Hive witness!!
The Witness Journey So Far
My witness journey certainly didn't come without some painstaking hours of troubleshooting and hunting through logs.. One of the purposes of this post is to note that becoming a Hive witness is not the easy money job it may seem at first glance. Some had asked me at first if the goal was just to "turn it on and get rich quick", and it certainly is not. The reason I became a witness in the first place was to do my part to help secure the blockchain and add the value generated back into the greater Hive community, and it certainly is not an easy task.
After enabling my witness about 40 days ago, I received my first block to sign after about 28 days. After waiting 4 weeks for that first block (impatiently I might add), I was pretty upset when I had checked my witness status to learn that I missed the block, and I had no idea why. I double, triple, and quadruple checked my config, and everything seemed to be in the right place. So next I decided to check out the generated logs of the witness, but alas after spending hours going through each line, I couldn't find anything that pointed to an issue. I was starting to feel defeated..
At that point I decided I needed some help. I reached out to a few folks/friends in the community who had helped me out in the past (which I greatly appreciate). After chatting with fellow witnesses @shmoogleosukami and @someguy123, I checked a few parameters of my config where mistakes were commonly made, but everything seemed to be in the right place. After that I was able to have a chat with @rishi556, who mentioned a few more common issues that could cause blocks to be missed, including what ended up being the issue all along, server clock time synchronization (but at that point I still hadn't quite fixed it yet, though it lead me in the right direction).
After feeling a little defeated, I upped my server resources well beyond what seemed necessary, rebuilt my witness node twice, including from source, and made sure my updates were in place for HF25. At this point, I was close to getting my second block to sign, which I missed... again. At this point I felt absolutely defeated, and felt as though I was letting my witness voters down. I had approximately 20 days before my next block to figure out a fix, and I was going to do everything in my power to figure it out.
I read article after article and every piece of documentation I could find that had been written about different node configurations and what not. At that point, I felt like I had a PhD in Hive witness configs, but I still had no idea why mine wasn't working. Then, just last night while having a beer for #beersaturday and just about to write a post for it (which I missed posting after focusing on my witness fix, sorry @detlev haha), I hit my eureka by accident.
Feeling fairly defeated and being so close to signing another block, I decided to take a step to get a backup server from the Privex team as I knew their all star support and knowledge would definitely make sure my witness was up to snuff for signing my blocks in case my primary witness node should fail again. After ordering my node, I received an email with a couple of attached articles, one of them being about editing the NTP server time synchronization to help keep you within 10s instead of 100s of milliseconds of blocks being added to the chain.
What?!? This seemed so simple, and what if it was my issue all along? I decided to give it one last shot, else I had planned to temporarily switch my infrastructure to a privex pre setup node for the time being. I setup the privex node as a backup and setup hive witness essentials to switch between nodes should one fail. Then it was time to apply the fix. EUREKA! Two lines added to a single config file, and next thing I know, I signed my first successful block!!
Guide for the quick Time Sync fix
If running/setting up your witness node on Ubuntu Focal (recommended, 20.04), you likely don't have NTP installed or configured with a high enough polling rate to keep in a proper time synchronization with blocks being added to the chain by default, which may be a big contributor to missed blocks for your witness, as the time sync seems to need to be under approximately 950 ms to ensure proper block signing (atleast from what I observed). Applying this fix solved this issue of lowering my block addition time sync from about 975 ms to 50-250 ms.
First, install NTP if you have not already:
sudo apt-get install ntp
Next, edit the config to ensure a more frequent polling rate:
sudo nano /etc/ntp.conf
and add to the end of the file:
minpoll 5 maxpoll 7
And save.
The above parameters are in powers of two, so this would cause the server to poll and synchronize its time every 25-49 seconds. THis is overkill in most cases, but semi necessary as a block producer.
After completing the above, enable and restart the process like so:
sudo systemctl enable ntp
sudo systemctl restart ntp
That's all, such a frustratingly easy fix for hours of troubleshooting haha.
Conclusion
Becoming a Hive witness is definitely not just an easy set-it-and-forget-it journey. It is one that takes careful configuration and maintenance in order to work properly in the way the community deserves. I want to thank everyone mentioned above for their help in getting me to this point and all of my followers/supporters who have moved my witness up to the 137th active witness spot!
My goal is to hopefully move into the top 100 active witnesses within the next 4 months. I don't expect this to happen without being earned, and I will continue to contribute value to the Hive community to help improve our wonderful blockchain ecosystem and earn a few more witness approval votes along the way. If you currently have witness votes to spare, consider voting for my @sicarius witness: https://vote.hive.uno/@sicarius! It would be greatly appreciated!
Also consider voting for the all star witness nodes below, whose help was greatly appreciated:
@hextech
@someguy123
@shmoogleosukami
Until next time,
~ Sicarius
Glad to see you are finally up and running. Man I wish I had remembered that time sync part of the setup. Completely slipped my mind that it was one of the key things needed for a witness.
Personally I use chrony as it was recommended to me but whatever works, works.
Are you using a native linux box to run your server or are you running via WSL or virtual box?
!PIZZA
Thanks for the help and the pizza, I appreciate it! Being the tech nerd that I am, I have a few ~2013-14 generation HP proliant and dell poweredge servers in my home office that I have picked up second hand as retired gov systems, but they are in great working order.
I am running my node on a native linux ubuntu installation on a semi souped up dell poweredge T320 that I just added 64 gb of RAM, though I may move it to my old dell R730 virtualization server in the future as I have more resources on that. I recently got my ISP to give me a static IP and a 2 gigabit direct fiber line. I'm starting to feel like I have my own datacenter in my home office lol (it runs a solid 82 degrees in there in the summer and I don't have to use my furnace in the winter anymore, which for context, I live in an area that has frequent sub-zero temps from Nov-Mar 😂)
Haha I know that 82 degree feeling, that's pretty much the min my room currently gets too.
Wish I could get a static IP but UK ISPs are really arsey for giving home broadband a static IP cause it's a 'businesss' thing, hense why I currently don't have a public facing api.
I've always wanted to run a server rack in my room but the noise and heat is a be nono :D My computer is hot and noisy enough as it is. I used to have a old intel server with like 8 fans in it and it sounded like a bloody jet plane.. Behind 3 doors...
Those old intel servers were absolutely terrible about noise, I have a couple that run a little louder but the server with my witness on it is actually super quiet most of the time. Just harddrive spin noises when there is any sort of load. Its all contained in my home office though, which pretty much only I occupy most of the time (occasionally maybe my fiance does some masters homework in there lol) so if there is a lot of noise and heat, I am the only one for it to bother haha. My next home improvement project is going to have to be to build a ventilation system!
I actually lucked out with the static ip, most ISP's in the US here also usually won't give out a static ip on a residential line. I live in a more rural state (South Dakota, home of Mount Rushmore and not much else) so some ISP's can be a little more lenient if you find the right person to ask a favor, and I stumbled into the right person. :)
@sicarius! I sent you a slice of $PIZZA on behalf of @shmoogleosukami.
Learn more about $PIZZA Token at hive.pizza (1/20)
Congratulations @sicarius! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s) :
Your next target is to reach 600 upvotes.
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPrunning the witness is more important than another beer.
But a !BEER might help to run the witness node
Haha thanks! Beer was definitely helping fuel some creative troubleshooting solutions last night, and probably ultimately my node 🤣
Definitely not forgetting my post this next week though, barring anymore technical issues
Love to read this story for the next #BeerSaturday
View or trade
BEER.Hey @sicarius, here is a little bit of
BEERfrom @detlev for you. Enjoy it!Learn how to earn FREE BEER each day by staking your
BEER.Your content has been voted as a part of Encouragement program. Keep up the good work!
Use Ecency daily to boost your growth on platform!
Support Ecency
Vote for Proposal
Delegate HP and earn more
Congratulations on signing your first block 🎉
I appreciate it! It definitely requires a little more behind the scenes work than some people might think haha.