I'm running two witness servers (primary & backup) on PCs sitting right next to me as I write.
I'm in the process of spinning up another backup witness on an older MacBook Pro which will provide battery backup and the ability to switch to other internet (wifi / 4G) if my home fibre internet was to go down.
Once I get that working I'll convert my primary witness to an API node and Hive-Engine Node.
In terms of reliability my main issue has little kids pressing buttons and rebooting the PC but I've changed the settings now so the buttons don't do anything.
Since I got the backup working 2 weeks ago I've only missed one block.
Great to hear there is some progress on your side. Missing only 1 block sounds promising - it would be interesting to see how your setup would be performing on a higher (top 20?) load.
Let's just hope your kids do not unplug your machine ;)
LOL.
I don't think they are capable of pulling a plug.
The flashing LEDs next to the buttons attracted them.
I've got automatic failover to the backup witness node so I shouldn't miss more than one block even if I was a top 20 witness.
If I get that high I'll run a third backup.
When I saw the mention here I figured this was going somewhere else. That IPFS document was unacceptable to have been passed and promoted as quality by Jack from Twitter without further review. I understand how it happened but it still pisses me off. We can't undo it and I'm not about to cry over it publicly or go back to them with corrections. Hive has everything going for it.
Regarding the witness question: We've always been on our own various servers. During the HF and immediately after we were cycling through our 5 different backup witnesses + main (although having them all crash in sequence in one day was not fun). Now we're down to fewer due to the need to upgrade the servers. But they are all decentralized, hosted in different areas of the world. The reason for that is lessons learned during the blackout in the early 2000s that took out the power grid in our area and kept it off for about a week. We also realized the difference in datacenter upkeep and tech physical response time between countries and services with the pandemic.
Having physical access to the main witness is not a good idea for most people. That would mean it's running out of their house or out of their office or a local datacenter. We tried it (local datacenter) and it was sub-optimal, mainly due to sharing the same infrastructure issues as the datacenter. Backup machines, however, can run out of a house or some local building very well.
The issue with some witnesses is that running a node is challenging, paying for it is challenging, and some like to start off with the 'in the box' step by step guide or to buy a ready-made one from Privex (who does a great job managing them). Where some witnesses upgrade once they learn more others just become complacent and shift their attention elsewhere. None of this is a simple matter to go through.
(Just to be clear here; a witness is not one server. It's one main server and at least one backup server or secondary server, usually more than one. It takes a long time to replay. You need to manage snapshots. There's a lot involved there.)
Thanks for the great summary about the current situation here on HIVE. I am currently running my witness on a server I do NOT have physical access to. But it is running on a small data center and not on one owned by a big company. So I don't really see any shortcomings in relation to running it on a server I have physical access to. Additionally I have a backup witness running on my computer at home, in case the other server has problems, or if the other server should be shutdown for some reason. I am using the server as my main witness, because of the much higher reliability. Also if someone was to attack my witness server, I guess they could also be attacking it if it was running at my home. Once again - I don't really see any shortcomings running my witness node on a small data center owned by a small company, in relation to running it at home. What do you think?
And here I go. I said I wasn't going to comment, but since you asked . . . 🙂
I think you'd need to have an extremely high degree of confidence in that small data center - at the end of the day, it's still a third party you have to depend on. With your home backup, you're looking good. Please, don't get me wrong, everything can still go wrong even if you've got it all on your own premises, but the more autonomous the witnesses are, and the more spread out they are, the less importance any one failure has to the entire ecosystem (and, of course, that failure is recognized, accounted for, and recovered from). The key idea I want to impress on everyone is that the more we are setup in that direction, the better, and the more we depend on "cloud" infrastructure, the worse. It sounds like you are somewhere in between, leaning towards complete autonomy - I even get the impression that you have a personal face-to-face relationship with that small data center (for those who don't know, don't laugh, I've worked personally with small data centers - they do exist).
Thanks for the question, but hopefully no more comments from me if I can help it. I really prefer to have more knowledgeable and expert people responding to these kinds of questions, if possible.
Yes I am depending on a third party and I would like not to, but unfortunately my internet connection is not all that good. If is failing regularly with complete outages of a few minutes.
To make HIVE more decentralized and for more witnesses to join, I guess a larger portion of the rewards would have to be payed out to the >20 witnesses. For me the decision to become a witness was not because of the rewards, but because I wanted to support the ecosystem (I will most probably never reach top 20 - I am happy if I am break-even at some point). However I guess that a lot of people think the opposite or don't have the opportunity to do so. Maybe an increasing value of HIVE would facilitate that.
Thanks for the great informative post. I run my witness on a server through @privex And it has been amazingly reliable and I have nothing but confidence in their company. But in a just incase scenario I could start getting a back up witness running on the PC I am typing from. Once I get another PC I do plan on having a back up witness 24/7 with 1gb/1gb package. Currently I have been running almost a year on Hive with @privex so that's my plan for NOW. I am going to prepare myself through the information provided to get a back up, up and running.
I run 4 witness nodes in 4 separate locations/countries.
2 are hosted in 2 different Cloud Service Provider's datacenter, mainly because they provide reliability and high availability for power and connectivity, with SLA. This is a key factor when you are a top 20 witness.
The 2 others are hosted in different secured professional IT rooms, which I have physical access to, one of them being under my sole control and ownership.
Both internet connections I have access to are abysmal (10 mb down 3 mb up each) - therefore I can't host my server at either of those places. Fibre/faster internet is not available in either place either, so I can't do that.
My witness node is currently hosted on a Hetzner Dedicated Server which I do not have physical access to unfortunately (but at least it has decent internet)
There are a lot of witnesses using Hetzner and 75% of API nodes.
I am sure they are good but its a major single point of failure / attack at the moment.
While I agree 100% and this is something I referenced in my comment as well, they do have a lot of data centers, even in the same area.
A lot use Privex as well (as they can't/won't maintain their witness) which also uses them but also uses other servers as well.
As I stated elsewhere, running the witness is pretty insignificant part of the equation, being there when there is a hardfork/emergency, knowing wtf you are doing when there is (beyond just following the 8 steps for Hive in a box), and actually providing intelligent insight for changes/issues, and perhaps even reviewing hardfork code. That is what a witness really is, but unfortunately it's really just boils down to running Hive in a box or renting a server from Privex for most of them.
I don't have physical access to my witness server, since my internet connection is not reliable and/or fast enough (I have a 20/5 Mbps connection with a 300 GB/month limit (it's capped at 5/1 Mbps after)) to run it at home.
If something extreme happened and I couldn't host my witness at any server provider, I could get a witness up and running on my personal PC in under 3 hours, and I could get my own hardware & better internet connection in about 1-2 weeks.
But unless something really extreme happens, I prefer renting servers since it's much faster to upgrade/switch to a new provider and they have more reliable power/internet than most households (and it's cheaper).
I won't be able to find reliable internet connection at homes in India and thus will not be possible for me to host the witness on own hardware.
Getting high availability (different from decentralization) is quite easy with available software as @@@blocktrades had mentioned. Find multiple locations, get a DDNS, get a heart beat tool (there are many) + get your domain name registered in say https://handshake.org/ and done.
Is more related to API nodes. I think he is mostly referring to Witness Nodes which should not be public facing and thus High Availability via Load Balancer software like HAProxy is not relevant.
Consider the scenario of running 2 nodes in 2 buildings, depending on which network interfaces are active, we can switch - but yea this is going to be very ugly hack and going to be better in the case of API nodes.
Hi, I decided to rent virtual server from @privex, where they hosts servers in different locations and I hope they can move VPS from one datacenter to another when needed.
I also have small server in my own office for testing and I can switch to it in short time if necessary. But I have only ADSL connection there and it is not enough for running witness node. But it's an option I have and I can bridge few days with it.
I have some plans for backup witness server and public node located in Czech Republic. I subsidize running of witness node by my own so first of all I have to work on financial security of the planned servers.
Let's see what we hear from our witnesses. Even though some are not running on decentralized physical servers they own, how can attackers take over them ? They would still need lot of hive stake to control, right ?
If the tech companies started to take hive nodes down because they are on their servers, like AWS did with Parlor, that is what the topic of the article is about.
i think the idea is to be able to host in few different countries. different laws, different interests... If Parlor was hosted somewhere outside the US it would most probably not happen to them.
A proper witness server only has SSH exposed, even that can be blocked by a firewall. The likelihood of a witness being attacked is slim to none if done properly, but most witnesses are just running Hive in a Box and really don't know much more than following the 8 steps to get it up and running or securing their server(s).
I think the real threat being presented here, is a data center or service provider telling you the take a hike. This happened with SSD Nodes, they ninja changed their service policy and said no one can use crypto projects on their server and cancelled anyone running a witness or anything crypto related without refund or notice. I had 8 servers there at the time they shut a few of them down without warning or refund and referenced a Terms of Service change that was made without notice and said it was my responsibility to read their terms of service daily.
Yes, the data center threat is political not technical.
While its obviously important to have some witnesses with a high level of technical skill able to help with hard forks, having lots of HIAB witnesses running their own equipment mitigates the political threat of Big Tech censorship.
Hive shouldn't be doing any hard forks at a time that it is under political attack. As the system matures hard forks should be much rarer than they have been in the past.
I have my own server sitting next to me that serves as a backup, and my main server is being hosted on Privex. Both work very well, but I keep paying the server bill because I like the freedom it affords me in handling things like forks and replays.
All due respect to Privex and people that use them, but it's concerning to me on their home page they are offering pre-packaged witness nodes. It begs the question: how many people such as yourself are using them?
Ironically this is the blockchain I've spent the least amount of technical time on. I'm assuming I could spin up a node and run a tcpdump on it and figure out all the IPs. Is it like other blockchains where there are a few pre-coded seed nodes and it just discovers the rest?
I have a variety of hosting providers that I use with servers available in multiple countries. I am not running on dedicated hardware that I control though.
Although there are big advantages to running witness servers with experienced and reliable hosting providers: DDoS protection, network uptime, power uptime. As long as there is sufficient distribution of witness servers across hosting provider companies, I think the network is more robust using top tier professional hosting services.
This is the truth, but prefer to be berated based on my experience with the author recently.
Just saw this scrolling up. Please refer to my reply to your main comment below. And stop making a fool of yourself (that is actually good faith advice; now you've just got to think about it - it's still not too late for you to shut that trap, turn the page, and just move on with what dignity you have left and start rebuilding). Do you think everyone else is an idiot? Below I said goodbye to you again.
Okay, I'm going to scroll higher again. Sure hope you haven't trolled this entire thread, not that it would really matter though: everyone would see it for exactly what it is.
And I really thought there were more of us who could say that, but it looks like the reality is that we're still very much in the dark ages in much of the world. I'm 5 kilometers from a small town of 70,000 and we've got the best fiber connectivity and stable electricity, man, from what I hear, I guess I could say "the best in the world!" And 55% of the world's population lives in urban areas! So I guess we have three possibilities here, either good broadband and power supply is much rarer than I thought, or the average HIVE witness is somewhat of an anomaly (in the sense that they are somewhat "off grid", so to speak, or they simply aren't representative of the typical urban dweller - neither of which is necessarily good or bad - it simply is what it is), or a combination of both (which is the most likely). Bottom line is: given that folks like us are the exception, expectations on this subject need to be revised. All up coming in my "report" that will be posted tomorrow. Thanks for the comment!
My witness nodes are running in datacenters where I don't have access... but what I have is a stable network and redundant electricity and these are major factors for running services 24/7.
I rent a server from Privex and have full faith that if something did happen there, @someguy123 and his team would be on point and handle it with discretion and swift action.
Hosting a node at home is not ideal, in my area broadband connections are crappy, we have several blackout a year due to weather conditions. Renting server in a data center in the country (Australia) is very expensive compared to EU or USA.
We just need more witnesses with more diversity of providers. But a backup witness is not paid well enough to attract new witnesses especially when going up the ranks is a slow process.
I kind of sort of agree more is better, but honestly only 21ish witnesses are actually getting used at any given time. API servers are a different beast, since you actually have to point your applications to them. I'm more worried about the 21 witnesses we have voted in not being able to sustain a DDoS, and how it would play out if multiple top witnesses are DoS'd. I whole-heartedly agree that any top witnesses running on underpowered or vulnerable hardware need to be weeded out. At some point they'll have to cross the threshold from hobbyist to institutional grade. We need the witness votes to age out to keep constant pressure on them to stay ahead of the curve.
Yet, as you can see from most of the responses, it is in fact a big issue for most people, in fact the biggest usually (as I stated). One of the two responses that it isn't, runs both a primary and secondary on the same network, so if the Internet does go down so do both witnesses.
Either way, I disagree with your post, I have witness servers in four different countries. If one of them does get affected in the slim to none chance, I'll have another up somewhere else within the day.
The real problem is most everyone is using the same provider.
The irony of me leaving this comment on a post talking about running a witness at home resulted in you throwing a fit and berating me about it. Source: thread from hell
Yet, as you can see from most of the responses, it is in fact a big issue for most people, in fact the biggest usually (as I stated). One of the two responses that it isn't, runs both a primary and secondary on the same network, so if the Internet does go down so do both witnesses. Either way, I disagree with your post, I have witness servers in four different countries. If one of them does get affected in the slim to none chance, I'll have another up somewhere else within the day.
The real problem is most everyone is using the same provider.
Thank you for the "summary" (as if you were the only qualified person to do so), and your continued innuendos (a "fit" for you is when someone insists on the same question when you do everything possible to avoid answering, even to the point of being obnoxious, offensive, and resorting to personal attacks). The "irony" of this comment is your acting out once again, announcing to the world that you are king, that you know best (because you say so, and reasons, explanations, and examples are beneath you), that you are so above the rest of us and that we need you to summarize and clarify for us, even before giving the original author time to do so him or herself (it's in the works, BTW, and it's a bit more comprehensive, which I know we couldn't ask of you). The "irony" of your comment is your need to show us all just how big a chip you have on your shoulder.
When someone says GOODBYE to you, as I did in that painful thread you call the thread from hell (thanks so much to your unwavering efforts to say a thousand unrelated words to hide having said nothing) where it took 20 comments to get a straight answer from you (not just from me by the way - the immediate reply to your godlike reply wasn't me, by the way – and, if anyone doubts my word - I'm not going to rehash it - it's reblogged on my blog!) and always finishing on a negative (as you do here, once again), you really should take the hint and go your own way (or at least be smart enough to not deliberately create problems with that person when in their presence - unless that's your modus operandi, in which case, I guess we're all screwed).
Don't you have any self-respect?
Please, get a clue. This is my third official "goodbye" to you. Don't bring your crap into my life, okay? Take it somewhere else. Is that clear? Do you understand? Again, as I said the first time, from my point of view, it’s nothing personal, it’s more about time and productivity. It’s just that I don’t have time, you are not my responsibility, and I have more important, more joyful, more enriching things to do with my life than to try to teach you the basic manners you should have learned long ago.
Got it?
Goodbye. (Third time’s a charm, right? Let’s hope so!)
Thank you for shedding light to this key topic, @cryptographic .
As actifit witness, we had our witness running previously on hetzner back in steem days, but as of hive, we moved it to privex team, their service is quite reliable.
We do not prefer running on our own home connection as the potential for Internet loss is at stake, so from a service continuity and reliability basis this is more optimal.
I ran a server at my apartment in the beginning. Until my provider was sold and the performance collapsed. In the meantime, my Internet crashes three times a day.
The current nodes are at Hetzner. One of them is only a few kilometers away from me and I know when I have to call them if something is wrong. Hetzner is located here in Nuremberg and several fellow students from my studies work there today.
You need luck here. If you use the former state-owned company as your Internet provider, you have a stable network, but it is relatively slow.
But if you live in a place where private companies have bought the network, you have to live with crashes and strong fluctuations, because they give every customer as much bandwidth as possible - but their infrastructure is not up to it.
"I firmly believe that all the resources we currently have available that are not dealing with mission critical issues should be immediately directed towards this urgent task of getting our witnesses set up on independently owned servers to which those witnesses have PHYSICAL ACCESS"
That doesn't make much sense.
There are pros and cons related to running a witness node "on-premises".
In most cases, for a top witness, running a node that way is might be a bad idea.
Are you currently running your witness node on hardware you personally own and have physical access to?
Yes. No. Maybe.
I have variety of nodes (some of them are witness capable, i.e. backup witness nodes):
I own them, I lease them, I rent them, I have them on-premises, I have them collocated in data centers, in some cases I have direct physical access, in some cases I can have a physical access with some time and effort, and in some cases I can't even get close to them.
My current scenarios:
Owned
On-premises
Physical access
0
0
0
0
0
1
0
1
1
1
0
0
1
0
1
1
1
1
None of my witness nodes run on AWS, GCP, or Azure (but it's nothing wrong to run them there).
And I don't touch Alibaba Cloud even with a poop-stick.
Thank you for your reply. Yes, I agree with you, or should I say, the consensus agrees with you, that a "yes, no, maybe" mix of things is probably the ideal. It's also worth noting that you are still in the minority, which is to say the majority (of respondents) are operating out of data centers exclusively and a sizeable number are running "off the shelf" products from the same providers (as I note in my "report" which followed this post). That's where we need to direct "all the resources we currently have available" - obviously not literal in meaning, but my opinion is that our available "human" resources should be directed at getting the rest of the witnesses up to your level (or at blocktrade's or arcange's, both with similar setups to yours), and certainly get everyone more widely distributed across providers and geographical areas. This depends entirely upon adequate documentation and assistance, which takes time, effort and knowledge to put together and share, hence the call for a temporary and urgent "reallocation of resources", again, metaphorical - the only resources needed here are the human experts to document and share their expertise . . . but that means dropping the coding, the programing of the next game, next release, etc., while that documentation, let's say knowledge base, is built . . . hence, "all our available resources", because, for all intents and purposes, that's what it would end up feeling like . . . I would think . . . if it were to be done in the urgent manner that events seem to suggest it be done in.
(As for the physical access, that was the ideal assumption I began with - that's what would be my preference if I were a witness - however, respondents' replies opened my eyes to the benefits of a mix, for practical reasons more than anything.)
I am preparing to run 1x Witness node and 1x RPC node from performance hardware (on-premises) connecting through Starlink (With secondary ISP on Optic Fiber for backup).
Hive.StarlinQ.Space
I would say I am within weeks/days to have it up and running. As I am following available documentations, I am still facing roadblocks, so I take note of what needs to be addressed for more clarity. Lack of documentation should not be a bottleneck for Hive's distribution. I am part of a team designing complex strategies and Hive has been identified as a key component of a Global Space Force strategy. Plans are under development to streamline Hive's evolution. To be continued...
Lack of documentation should not be a bottleneck for Hive's distribution.
You're absolutely correct and I agree 100%. That's the reason for my post, and the tone with which it was written, because documentation deficiencies are not uncommon in crypto, in fact, it's the opposite: it's almost a "cultural trait" to completely forget about all documentation that's not on GitHub, as if it wasn't important or didn't matter, and that just can't happen if you're serious about taking the project forward.
I'm running two witness servers (primary & backup) on PCs sitting right next to me as I write.
I'm in the process of spinning up another backup witness on an older MacBook Pro which will provide battery backup and the ability to switch to other internet (wifi / 4G) if my home fibre internet was to go down.
Once I get that working I'll convert my primary witness to an API node and Hive-Engine Node.
In terms of reliability my main issue has little kids pressing buttons and rebooting the PC but I've changed the settings now so the buttons don't do anything.
Since I got the backup working 2 weeks ago I've only missed one block.
Great to hear there is some progress on your side. Missing only 1 block sounds promising - it would be interesting to see how your setup would be performing on a higher (top 20?) load.
Let's just hope your kids do not unplug your machine ;)
!gif unplug
LOL.
I don't think they are capable of pulling a plug.
The flashing LEDs next to the buttons attracted them.
I've got automatic failover to the backup witness node so I shouldn't miss more than one block even if I was a top 20 witness.
If I get that high I'll run a third backup.
Via Tenor
When I saw the mention here I figured this was going somewhere else. That IPFS document was unacceptable to have been passed and promoted as quality by Jack from Twitter without further review. I understand how it happened but it still pisses me off. We can't undo it and I'm not about to cry over it publicly or go back to them with corrections. Hive has everything going for it.
Regarding the witness question: We've always been on our own various servers. During the HF and immediately after we were cycling through our 5 different backup witnesses + main (although having them all crash in sequence in one day was not fun). Now we're down to fewer due to the need to upgrade the servers. But they are all decentralized, hosted in different areas of the world. The reason for that is lessons learned during the blackout in the early 2000s that took out the power grid in our area and kept it off for about a week. We also realized the difference in datacenter upkeep and tech physical response time between countries and services with the pandemic.
Having physical access to the main witness is not a good idea for most people. That would mean it's running out of their house or out of their office or a local datacenter. We tried it (local datacenter) and it was sub-optimal, mainly due to sharing the same infrastructure issues as the datacenter. Backup machines, however, can run out of a house or some local building very well.
The issue with some witnesses is that running a node is challenging, paying for it is challenging, and some like to start off with the 'in the box' step by step guide or to buy a ready-made one from Privex (who does a great job managing them). Where some witnesses upgrade once they learn more others just become complacent and shift their attention elsewhere. None of this is a simple matter to go through.
(Just to be clear here; a witness is not one server. It's one main server and at least one backup server or secondary server, usually more than one. It takes a long time to replay. You need to manage snapshots. There's a lot involved there.)
Thanks for the great summary about the current situation here on HIVE. I am currently running my witness on a server I do NOT have physical access to. But it is running on a small data center and not on one owned by a big company. So I don't really see any shortcomings in relation to running it on a server I have physical access to. Additionally I have a backup witness running on my computer at home, in case the other server has problems, or if the other server should be shutdown for some reason. I am using the server as my main witness, because of the much higher reliability. Also if someone was to attack my witness server, I guess they could also be attacking it if it was running at my home. Once again - I don't really see any shortcomings running my witness node on a small data center owned by a small company, in relation to running it at home. What do you think?
!ENGAGE 50
And here I go. I said I wasn't going to comment, but since you asked . . . 🙂
I think you'd need to have an extremely high degree of confidence in that small data center - at the end of the day, it's still a third party you have to depend on. With your home backup, you're looking good. Please, don't get me wrong, everything can still go wrong even if you've got it all on your own premises, but the more autonomous the witnesses are, and the more spread out they are, the less importance any one failure has to the entire ecosystem (and, of course, that failure is recognized, accounted for, and recovered from). The key idea I want to impress on everyone is that the more we are setup in that direction, the better, and the more we depend on "cloud" infrastructure, the worse. It sounds like you are somewhere in between, leaning towards complete autonomy - I even get the impression that you have a personal face-to-face relationship with that small data center (for those who don't know, don't laugh, I've worked personally with small data centers - they do exist).
Thanks for the question, but hopefully no more comments from me if I can help it. I really prefer to have more knowledgeable and expert people responding to these kinds of questions, if possible.
Sorry for pushing you to reply to my comment ;)
Yes I am depending on a third party and I would like not to, but unfortunately my internet connection is not all that good. If is failing regularly with complete outages of a few minutes.
To make HIVE more decentralized and for more witnesses to join, I guess a larger portion of the rewards would have to be payed out to the >20 witnesses. For me the decision to become a witness was not because of the rewards, but because I wanted to support the ecosystem (I will most probably never reach top 20 - I am happy if I am break-even at some point). However I guess that a lot of people think the opposite or don't have the opportunity to do so. Maybe an increasing value of HIVE would facilitate that.
ENGAGEtokens.Thanks for the great informative post. I run my witness on a server through @privex And it has been amazingly reliable and I have nothing but confidence in their company. But in a just incase scenario I could start getting a back up witness running on the PC I am typing from. Once I get another PC I do plan on having a back up witness 24/7 with 1gb/1gb package. Currently I have been running almost a year on Hive with @privex so that's my plan for NOW. I am going to prepare myself through the information provided to get a back up, up and running.
I run 4 witness nodes in 4 separate locations/countries.
2 are hosted in 2 different Cloud Service Provider's datacenter, mainly because they provide reliability and high availability for power and connectivity, with SLA. This is a key factor when you are a top 20 witness.
The 2 others are hosted in different secured professional IT rooms, which I have physical access to, one of them being under my sole control and ownership.
Jeez, I just posted my "report". Let's see. I'll edit to get you in. 😊
Both internet connections I have access to are abysmal (10 mb down 3 mb up each) - therefore I can't host my server at either of those places. Fibre/faster internet is not available in either place either, so I can't do that.
My witness node is currently hosted on a Hetzner Dedicated Server which I do not have physical access to unfortunately (but at least it has decent internet)
There are a lot of witnesses using Hetzner and 75% of API nodes.
I am sure they are good but its a major single point of failure / attack at the moment.
While I agree 100% and this is something I referenced in my comment as well, they do have a lot of data centers, even in the same area.
A lot use Privex as well (as they can't/won't maintain their witness) which also uses them but also uses other servers as well.
As I stated elsewhere, running the witness is pretty insignificant part of the equation, being there when there is a hardfork/emergency, knowing wtf you are doing when there is (beyond just following the 8 steps for Hive in a box), and actually providing intelligent insight for changes/issues, and perhaps even reviewing hardfork code. That is what a witness really is, but unfortunately it's really just boils down to running Hive in a box or renting a server from Privex for most of them.
I don't have physical access to my witness server, since my internet connection is not reliable and/or fast enough (I have a 20/5 Mbps connection with a 300 GB/month limit (it's capped at 5/1 Mbps after)) to run it at home.
If something extreme happened and I couldn't host my witness at any server provider, I could get a witness up and running on my personal PC in under 3 hours, and I could get my own hardware & better internet connection in about 1-2 weeks.
But unless something really extreme happens, I prefer renting servers since it's much faster to upgrade/switch to a new provider and they have more reliable power/internet than most households (and it's cheaper).
I won't be able to find reliable internet connection at homes in India and thus will not be possible for me to host the witness on own hardware.
Getting high availability (different from decentralization) is quite easy with available software as @@@blocktrades had mentioned. Find multiple locations, get a DDNS, get a heart beat tool (there are many) + get your domain name registered in say https://handshake.org/ and done.
Consider the scenario of running 2 nodes in 2 buildings, depending on which network interfaces are active, we can switch - but yea this is going to be very ugly hack and going to be better in the case of API nodes.
Hi, I decided to rent virtual server from @privex, where they hosts servers in different locations and I hope they can move VPS from one datacenter to another when needed.
I also have small server in my own office for testing and I can switch to it in short time if necessary. But I have only ADSL connection there and it is not enough for running witness node. But it's an option I have and I can bridge few days with it.
I have some plans for backup witness server and public node located in Czech Republic. I subsidize running of witness node by my own so first of all I have to work on financial security of the planned servers.
Let's see what we hear from our witnesses. Even though some are not running on decentralized physical servers they own, how can attackers take over them ? They would still need lot of hive stake to control, right ?
To take over as a witness, yes stake is required.
If the tech companies started to take hive nodes down because they are on their servers, like AWS did with Parlor, that is what the topic of the article is about.
i think the idea is to be able to host in few different countries. different laws, different interests... If Parlor was hosted somewhere outside the US it would most probably not happen to them.
Ahh, that explains. But I hope, we still have good amount of nodes running on their own servers, let's see.
^ this is more what I take from it as well.
A proper witness server only has SSH exposed, even that can be blocked by a firewall. The likelihood of a witness being attacked is slim to none if done properly, but most witnesses are just running Hive in a Box and really don't know much more than following the 8 steps to get it up and running or securing their server(s).
I think the real threat being presented here, is a data center or service provider telling you the take a hike. This happened with SSD Nodes, they ninja changed their service policy and said no one can use crypto projects on their server and cancelled anyone running a witness or anything crypto related without refund or notice. I had 8 servers there at the time they shut a few of them down without warning or refund and referenced a Terms of Service change that was made without notice and said it was my responsibility to read their terms of service daily.
Yes, the data center threat is political not technical.
While its obviously important to have some witnesses with a high level of technical skill able to help with hard forks, having lots of HIAB witnesses running their own equipment mitigates the political threat of Big Tech censorship.
Hive shouldn't be doing any hard forks at a time that it is under political attack. As the system matures hard forks should be much rarer than they have been in the past.
I have my own server sitting next to me that serves as a backup, and my main server is being hosted on Privex. Both work very well, but I keep paying the server bill because I like the freedom it affords me in handling things like forks and replays.
All due respect to Privex and people that use them, but it's concerning to me on their home page they are offering pre-packaged witness nodes. It begs the question: how many people such as yourself are using them?
A lot
Almost scared to ask, but... How many in the upper ranks?
I am not 100% sure but I would guess half.
Ironically this is the blockchain I've spent the least amount of technical time on. I'm assuming I could spin up a node and run a tcpdump on it and figure out all the IPs. Is it like other blockchains where there are a few pre-coded seed nodes and it just discovers the rest?
Yes. There is a seed node list which are usually not witness nodes. Technically you can get started with only one and pick up the others.
I have a variety of hosting providers that I use with servers available in multiple countries. I am not running on dedicated hardware that I control though.
great post and call to action!
@tipu curate
!ENGAGE 50
ENGAGEtokens.Yes
Although there are big advantages to running witness servers with experienced and reliable hosting providers: DDoS protection, network uptime, power uptime. As long as there is sufficient distribution of witness servers across hosting provider companies, I think the network is more robust using top tier professional hosting services.
This is the truth, but prefer to be berated based on my experience with the author recently.
Just saw this scrolling up. Please refer to my reply to your main comment below. And stop making a fool of yourself (that is actually good faith advice; now you've just got to think about it - it's still not too late for you to shut that trap, turn the page, and just move on with what dignity you have left and start rebuilding). Do you think everyone else is an idiot? Below I said goodbye to you again.
Okay, I'm going to scroll higher again. Sure hope you haven't trolled this entire thread, not that it would really matter though: everyone would see it for exactly what it is.
Ich habe eine solide Internetverbindung, so kann ich der Server zu Hause aufstellen , das einzige Problem zurzeit sind die Stromkosten ;)
And I really thought there were more of us who could say that, but it looks like the reality is that we're still very much in the dark ages in much of the world. I'm 5 kilometers from a small town of 70,000 and we've got the best fiber connectivity and stable electricity, man, from what I hear, I guess I could say "the best in the world!" And 55% of the world's population lives in urban areas! So I guess we have three possibilities here, either good broadband and power supply is much rarer than I thought, or the average HIVE witness is somewhat of an anomaly (in the sense that they are somewhat "off grid", so to speak, or they simply aren't representative of the typical urban dweller - neither of which is necessarily good or bad - it simply is what it is), or a combination of both (which is the most likely). Bottom line is: given that folks like us are the exception, expectations on this subject need to be revised. All up coming in my "report" that will be posted tomorrow. Thanks for the comment!
!ENGAGE 50
ENGAGEtokens.My witness nodes are running in datacenters where I don't have access... but what I have is a stable network and redundant electricity and these are major factors for running services 24/7.
I rent a server from Privex and have full faith that if something did happen there, @someguy123 and his team would be on point and handle it with discretion and swift action.
Hosting a node at home is not ideal, in my area broadband connections are crappy, we have several blackout a year due to weather conditions. Renting server in a data center in the country (Australia) is very expensive compared to EU or USA.
We just need more witnesses with more diversity of providers. But a backup witness is not paid well enough to attract new witnesses especially when going up the ranks is a slow process.
I kind of sort of agree more is better, but honestly only 21ish witnesses are actually getting used at any given time. API servers are a different beast, since you actually have to point your applications to them. I'm more worried about the 21 witnesses we have voted in not being able to sustain a DDoS, and how it would play out if multiple top witnesses are DoS'd. I whole-heartedly agree that any top witnesses running on underpowered or vulnerable hardware need to be weeded out. At some point they'll have to cross the threshold from hobbyist to institutional grade. We need the witness votes to age out to keep constant pressure on them to stay ahead of the curve.
The irony of me leaving this comment on a post talking about running a witness at home resulted in you throwing a fit and berating me about it.
Source: thread from hell
Yet, as you can see from most of the responses, it is in fact a big issue for most people, in fact the biggest usually (as I stated). One of the two responses that it isn't, runs both a primary and secondary on the same network, so if the Internet does go down so do both witnesses.
Either way, I disagree with your post, I have witness servers in four different countries. If one of them does get affected in the slim to none chance, I'll have another up somewhere else within the day.
The real problem is most everyone is using the same provider.
Thank you for the "summary" (as if you were the only qualified person to do so), and your continued innuendos (a "fit" for you is when someone insists on the same question when you do everything possible to avoid answering, even to the point of being obnoxious, offensive, and resorting to personal attacks). The "irony" of this comment is your acting out once again, announcing to the world that you are king, that you know best (because you say so, and reasons, explanations, and examples are beneath you), that you are so above the rest of us and that we need you to summarize and clarify for us, even before giving the original author time to do so him or herself (it's in the works, BTW, and it's a bit more comprehensive, which I know we couldn't ask of you). The "irony" of your comment is your need to show us all just how big a chip you have on your shoulder.
When someone says GOODBYE to you, as I did in that painful thread you call the thread from hell (thanks so much to your unwavering efforts to say a thousand unrelated words to hide having said nothing) where it took 20 comments to get a straight answer from you (not just from me by the way - the immediate reply to your godlike reply wasn't me, by the way – and, if anyone doubts my word - I'm not going to rehash it - it's reblogged on my blog!) and always finishing on a negative (as you do here, once again), you really should take the hint and go your own way (or at least be smart enough to not deliberately create problems with that person when in their presence - unless that's your modus operandi, in which case, I guess we're all screwed).
Don't you have any self-respect?
Please, get a clue. This is my third official "goodbye" to you. Don't bring your crap into my life, okay? Take it somewhere else. Is that clear? Do you understand? Again, as I said the first time, from my point of view, it’s nothing personal, it’s more about time and productivity. It’s just that I don’t have time, you are not my responsibility, and I have more important, more joyful, more enriching things to do with my life than to try to teach you the basic manners you should have learned long ago.
Got it?
Goodbye. (Third time’s a charm, right? Let’s hope so!)
You have a weird little world you live in.
If you don't like it, you've got an easy solution: stay away from it!
Thank you for shedding light to this key topic, @cryptographic .
As actifit witness, we had our witness running previously on hetzner back in steem days, but as of hive, we moved it to privex team, their service is quite reliable.
We do not prefer running on our own home connection as the potential for Internet loss is at stake, so from a service continuity and reliability basis this is more optimal.
Thank you for the reply. It's another valuable data point that needs to be included!
I ran a server at my apartment in the beginning. Until my provider was sold and the performance collapsed. In the meantime, my Internet crashes three times a day.
The current nodes are at Hetzner. One of them is only a few kilometers away from me and I know when I have to call them if something is wrong. Hetzner is located here in Nuremberg and several fellow students from my studies work there today.
Thanks for sharing that! From the looks of things, it looks like you are in the majority group!
On another note, I'm really surprised by what I've heard about broadband connectivity in Germany.
You need luck here. If you use the former state-owned company as your Internet provider, you have a stable network, but it is relatively slow.
But if you live in a place where private companies have bought the network, you have to live with crashes and strong fluctuations, because they give every customer as much bandwidth as possible - but their infrastructure is not up to it.
If someone is kind enough to direct me to instructions on how to run a node. I will gladly build one and run it.
I think there are lots of folks like you. That's the main reason for this post. Hopefully it'll be in the works soon!
That doesn't make much sense.
There are pros and cons related to running a witness node "on-premises".
In most cases, for a top witness, running a node that way is might be a bad idea.
Yes. No. Maybe.
I have variety of nodes (some of them are witness capable, i.e. backup witness nodes):
I own them, I lease them, I rent them, I have them on-premises, I have them collocated in data centers, in some cases I have direct physical access, in some cases I can have a physical access with some time and effort, and in some cases I can't even get close to them.
My current scenarios:
None of my witness nodes run on AWS, GCP, or Azure (but it's nothing wrong to run them there).
And I don't touch Alibaba Cloud even with a poop-stick.
Thank you for your reply. Yes, I agree with you, or should I say, the consensus agrees with you, that a "yes, no, maybe" mix of things is probably the ideal. It's also worth noting that you are still in the minority, which is to say the majority (of respondents) are operating out of data centers exclusively and a sizeable number are running "off the shelf" products from the same providers (as I note in my "report" which followed this post). That's where we need to direct "all the resources we currently have available" - obviously not literal in meaning, but my opinion is that our available "human" resources should be directed at getting the rest of the witnesses up to your level (or at blocktrade's or arcange's, both with similar setups to yours), and certainly get everyone more widely distributed across providers and geographical areas. This depends entirely upon adequate documentation and assistance, which takes time, effort and knowledge to put together and share, hence the call for a temporary and urgent "reallocation of resources", again, metaphorical - the only resources needed here are the human experts to document and share their expertise . . . but that means dropping the coding, the programing of the next game, next release, etc., while that documentation, let's say knowledge base, is built . . . hence, "all our available resources", because, for all intents and purposes, that's what it would end up feeling like . . . I would think . . . if it were to be done in the urgent manner that events seem to suggest it be done in.
(As for the physical access, that was the ideal assumption I began with - that's what would be my preference if I were a witness - however, respondents' replies opened my eyes to the benefits of a mix, for practical reasons more than anything.)
I am preparing to run 1x Witness node and 1x RPC node from performance hardware (on-premises) connecting through Starlink (With secondary ISP on Optic Fiber for backup).
Hive.StarlinQ.Space
I would say I am within weeks/days to have it up and running. As I am following available documentations, I am still facing roadblocks, so I take note of what needs to be addressed for more clarity. Lack of documentation should not be a bottleneck for Hive's distribution. I am part of a team designing complex strategies and Hive has been identified as a key component of a Global Space Force strategy. Plans are under development to streamline Hive's evolution. To be continued...
You're absolutely correct and I agree 100%. That's the reason for my post, and the tone with which it was written, because documentation deficiencies are not uncommon in crypto, in fact, it's the opposite: it's almost a "cultural trait" to completely forget about all documentation that's not on GitHub, as if it wasn't important or didn't matter, and that just can't happen if you're serious about taking the project forward.
Thanks for the reply!
An article worth passing along...
@newparadigmtt, @eyeofthestrom, @fulltimegeek