Witness Update - First Block signed & Tweaks for Windows Home Witnessing

in #witness3 years ago

An update on Hive for the masses, by the masses - using mass market PCs for Hive nodes.

My witness node on WSL2 on a Windows 10 Home PC has been running for 5 days now, almost flawlessly, and this morning signed its first block. I was very lucky to catch it almost real time.

image.png

You can read details of How to set up a witness on a Windows 10 PC here.

As usual with doing something that's never been done before, there were a few tweaks and teething difficulties which I had to work out the solutions for.

I'm sharing them here for benefit of others who want to set up witness nodes on Windows 10 Home PCs.

WSL2 Clock Drift

After my witness had been running for a few days I started to notice red exceptions in the logs that referred to blocks being old.
I also noticed that the block offset times had increased from a few hundred milliseconds to close to 5000 milliseconds.

Screen Shot 20201117 at 14.44.23.png

@deathwing suggested that my system clock was off.

I tried synching my Windows clock with an internet time server but it didn't fix the problem

So I did some research and discovered that WSL2 is subject to time drift problems, especially when the host Windows 10 is suspended or resumed.

But the initial fix is quite simple. Just enter the following commands in Ubuntu to resynch the WSL2 clock.

sudo apt install ntpdate
sudo ntpdate -sb time.nist.gov

But I didn't want to have to do this manually every day or so - there had to be a way to automate it.

I did some further research. Crontab can be used to automate Linux tasks that need to be run on a regular interval

The problem with automating commands that require a sudo via crontab is that you have to find the location of the command's file so you can provide the full path in crontab - a task I don't know how to do quickly and easily.

I discovered a better solution.
I turns out that WSL 2 commands can be run as bash or batch file commands in Windows and you can specify the user (including root)

So I wrote a simple Windows batch file which included the command

wsl -u root sh -C "ntpdate -sb time.nist.gov"

I then used Windows Task Scheduler to run this twice a day.

Clock drift solved.

Automated Snapshots - Essential for Windows PCs

One of the great new features of Hive HF 24 is the ability to save a snapshot of a Hive node and then relaunch quickly from that snapshot.

It avoids the need for the dreaded replay of the blockchain, which can take around 18 hours on an average machine.
Replays times are pretty linear with CPU single core speed, so slower CPUs will take longer and 32 core server chips won't help - it is single core performance that counts.

I loaded up my Hive witness node originally from a Privex snapshot as part of @someguy123's HIAB.

But that snapshot was more than a week old when I used it and took about 1 hour 20 minutes to load up.

I wanted to take my own regular snapshots so I always have a recent snapshot to restart from and to allow me to quickly get a backup witness node running.

This is particularly important because, as @themarkymark points out, Windows 10 has this annoying habit of forcing updates on users and rebooting your computer without permission.

This can be handled by regularly updating your Windows 10 PC in a controlled fashion and taking a snapshot just before going through the update process.

I just tested the speed of a restart from a 24 hour old snapshot and my witness node was up and running in 14 minutes from typing the command:

./run.sh loadsnap

In order to automate the process I used crontab in Ubuntu.

crontab -e

My crontab configuration now runs snapshots 3 times a week and deletes the oldest snapshot to save SSD space.
As there is a risk of corruption every time you stop Hived, this process ensures you've always got two good snapshots.

I've also used a crontab function that allows an automatic snapshot load every time the computer is rebooted.
Currently that is using the Sunday snapshot but I'll write a script soon to make it use the latest one.

My crontab config now looks like this:

m h dom mon dow command
40 14 * * 0 /home/apshamilton/hive-docker/data/witness_node_data_dir/snapshot rm -r sundaysnapshot
40 14 * * 3 /home/apshamilton/hive-docker/data/witness_node_data_dir/snapshot rm -r wednesdaysnapshot
40 14 * * 5 /home/apshamilton/hive-docker/data/witness_node_data_dir/snapshot rm -r fridaysnapshot
45 14 * * 0,3,5 /home/apshamilton/hive-docker/run.sh stop
46 14 * * 0 /home/apshamilton/hive-docker/run.sh dumpsnap sundaysnapshot
46 14 * * 3 /home/apshamilton/hive-docker/run.sh dumpsnap wednesdaysnapshot
46 14 * * 5 /home/apshamilton/hive-docker/run.sh dumpsnap fridaysnapshot
47 14 * * * /home/apshamilton/hive-docker/run.sh logs
@reboot /home/apshamilton/hive-docker/run.sh load_snap sundaysnapshot

Automated snapshots done.

I now feel that I've overcome the initial and potential issues with using Windows 10 Home to run a Hive Witness Node.

The machine running the Witness node has proved very stable running other software that needs a high level of uptime.

It is a high end consumer motherboard using the X99 chipset and an Intel i7 6800k CPU (12/6 core) with 96Gb RAM, Radeon Pro Duo GPU and 500Gb NVME, 500Gb SATA and 1TB HDD. A typical 3-4 year old high-end gaming PC (apart from the massive RAM).

I now have the ability to re-start very quickly from a reboot forced by Windows or needed for other reasons.

I will soon be setting up a witness / seed node on a machine using a B365 motherboard and i5 8400 CPU (6 core) and then plan on upgrading the 6800k machine to an API node.

I'll keep posting updates on Hive for the Masses - Hive Nodes on Windows 10 Home PCs.

Please vote for my Hive witness. (KeyChain or HiveSigner)

Witness Vote using direct Hivesigner

Sort:  

My witness is running on Windows 10 as well, using a Debian subsystem. I solved the time issue by decreasing the schedule time for NTP from a day to just 10 minutes between updates.

By the way: I just voted for your witness.

you can use which <command> to find the full path of a command

image.png

Thanks. That will be very useful. I've got a lot to learn abut linux but I did learn to program on Unix machines doing computer science at University 30 years ago.

Congrats! These are excellent news! Well done

For a serious witness campaign I would strongly recommend renting a proper server in a data center.

One of the things I intend to prove is that this is unnecessary and anti-decentralisation.

Centralised platforms need the promised 99.999% uptime that a data center can provide (at a huge price), but de-centralised platforms don't need it.
The whole point of being decentralised is that redundancy is provided by the very large number of witnesses and nodes provided by independent people.

Having so many witness nodes at a very limited number of data centres is a huge problem for decentralisation, censorship resistance and freedom.

What if Privex gets a Court order to censor your posts?!

Privex is kind of expensive anyways...

Take a look at the pricing for DigitalOcean, Vultr, Google Cloud, RackSpace and Microsoft Azure for the closest equivalent of our $50/mo witness package - you'll see they all charge $100's for what we charge just $50/mo for.

Privex has extremely low profit margins, the only hosting providers that are cheaper, generally either have much lower quality hardware/network, awful customer support, a strict refund policy (or no refund policy at all), or they're unsustainable scams that'll probably disappear within a year without notice.

I was one of your first customers.

Hetzner would have been half the price, but less convenient for me.

What if Privex gets a Court order to censor your posts?!

Heh, this is exactly why we chose Belize, and not the US, UK, or any other country known for surveillance, and overreach of governments / law enforcement.

We are not legally required to comply with court orders sent to us from the US, UK, EU etc.

Under the laws of Belize, the only court orders we have to comply with, are ones from the Belize courts.

Plus, as a privacy + freedom of speech company, if we ever did receive a court order, we would have it verified by lawyers, and attempt to fight it and publicize it (AFAIK there are no gag orders in Belize).

The way that international legal claims work, is using legal agreements known as MLATs (Mutual Legal Assistance Treaty's). These are agreements made by a country with other countries, where country A is allowed to request legal enforcement in country B, and vice versa. E.G. imprisoning a known American terrorist who fled to Canada.

Belize only has ONE MLAT - where-as most European countries and the US have 100s. Belize's only MLAT is with the USA.

Despite having an MLAT with the USA, the terms of the MLAT are quite strict, preventing the US from using the MLAT frivilously - and from what I've read in the MLAT agreement, Belize gets the upper hand and has the right to refuse any MLAT request from the US if they don't agree with their legal request, e.g. low level crimes such as shoplifting, hate speech etc. where it wouldn't be worth the time or money for Belize to enforce the request made by the USA.

I can confirm under oath, that in the entire 3 years that Privex has been operating, we have received ZERO legal requests or court orders from ANY COUNTRY. As soon as a country's government or law enforcement notices Privex is based in Belize, they quickly give up hope on attempting to get any information from us, or threatening us with legal crap.

This is really good to hear and I never had any doubt that Privex would do its utmost to prevent censorship of Hive posts.
I think the Privex does an amazing job and full appreciate all the support they give to the Hive community.

However the point remains that even with all the best efforts, any point of centralisation is vulnerable. Also companies providing data center services are vulnerable because they are easily identifiable.

Hive is pretty under the radar at the moment but if Hive ever achieves the large scale adoption that we all want to see, enemies of freedom will start attacking all vulnerabilities.

It is almost impossible for governments to stop a blockchain that is fully decentralised across thousands of nodes run by individual people in their homes all across the world.

However if they only need to take down an handful of identifiable companies with fixed facilities then they will find a way. Even in Belize.

Also if the owners of a data center company are in the USA or a jurisdiction with easy extradition to the US then it can be a problem, as Bitmex recently found out.

This is why the ability to run a witness node (and hopefully soon an API node) on a Windows Home PC is so important.
We need to have thousands of active witnesses and hundreds of API nodes to be fully secure and decentralised.

The Hive Devs have done amazing work reducing the resource requirements to run a Hive node to the point where this is possible.

Now we as a community need to take this opportunity to more fully decentralise.

This is why I'm running a witness on a Windows Home PC - to prove that it can be done and can be reliable.

I appreciate the enthusiasm.

I had my pricefeed and other scripts running at home, while the witness was in a data center. This was, because I thought I needed a static IP for my node - that does not seem to be the case (?). I had noone to ask back then.

For a full API node (I would recommend a seed node first) you would need a fixed IP or at least a fixed adress to route to your node from, because otherwise people will not find your node - and then what would be the point of it ?

Running these things from home might sound and feel easy at first, but after 2 years and hundreds of missed blocks, I gave up.

If you stick with your plan, I can recommend looking into a router, that lets you also plug in a SIM to fall back on, in case of network problems.

What about uninterrupted power supply ?

In case you are on the road, you want ssh access to your servers anyways, because things always go wrong, when you are not at home ...

I will vote for you later on, but I believe you will migrate your setup once you come to your senses :D
There are other options than privex and you do not seem to need their privacy, which is why I chose them, back then.

Thanks for your advice and suggestions.

It is a lot easier to run a Hive node at home post HF24.

Snapshots mean that one can have your node running again after a reboot in less than 15 minutes, rather than having to replay for days.

I plan to run a seed / backup witness node as well quite soon, which will allow almost instant changeover in case on main node going offline. I have heaps of quite decent PCs parts lying around from GPU mining days.

I have very good internet with fibre to my desk and look at these download and upload speeds - Way better than most data centres provide for basic customers.

I have a 4G USB modem that can provide backup internet access but I also can access my friendly neighbour's wifi to provide network redundancy.

Power outages are extremely rare but with HF24 you can run a backup witness on a laptop with 8Gb RAM at a pinch and easily with 16Gb RAM. The laptop can get internet access over the 4G Modem if the whole apartment complex was to lose power so that neighbour wifi wasn't available.

I work from home and rarely go anywhere these days with COVID and 5 kids including 3 little ones (actually they are the main threat to be defended against).

So you see, I've considered all the things that a data center provides and realised that I have better everything at home, for free.

I also note that you are voting for a @liberosist who has been a dead witness for over a year. Maybe you can give me that vote.:-)