I don't think deathwing is more stable, per se. But I think his setup is probably closest to ours in terms of power and it's a lot less heavily loaded I think. Currently, I think api.hive.blog handles almost all the traffic from hive.blog, for instance.
I did publish some of our data for api.hive.blog: https://hive.blog/hive-139531/@blocktrades/hive-api-node-performance-measurement
I'd be interested to see the traffic on some of the other nodes, and if some other nodes would be willing to run the analyzer on their site, I could post an update with more recent data from the jussi traffic analyzer.
I was trying to get the traffic analyzer working at one point, but as far as I recall, you guys use a custom jussi structure. Which meant that my jussi wasn't really capable of working with the traffic analyzer. I'll maybe look into it later on.
I think what Rishi meant (as well as some other people I've heard from) is mainly revolving around the networking of hive.blog -- unfortunately, it seems like OVH's network has been relatively on fire (pun intended) for the last couple of months, causing dropouts from time to time or in the best-case scenario, sudden latency spikes. (usually lasting about 10-ish seconds, but it's enough for it to error out).
Unfortunately, it's really hard to get a "usage metric" per se with jussi. The best possible ways I found so far were -- link usage (per second), bandwidth use (per month) and active connections to the node.
I may write a post on it soon.
For benchmarking, I think @fullnodeupdate overall is doing a great job. While it is sometimes vague and counts things that may mess up the stats (block latency, for example, which isn't always accurate) it's a nice overlook to most nodes.
Check with @gtg, he can assist you in getting your logging correct. It's not really "custom jussi structure", just a configuration of the log settings. IMO, it shouldn't be difficult, with a little assistance, and I think it should work fine for a usage metric.
I don't think the issue is OVH (other than that one time, when there was an actual fire), but we have seen cases where cloudflare has blocked some locations (while most locations are simultaneously able to access it without a problem). I suspect it's related to their DDOS attack prevention system. There are occasional attacks launched against our jussi node.
I have my doubts about all the remote benchmarking tools I've seen. They mostly seem to report different results depending on where you run them from. That's why I think local data at the node is best, if you're trying to measure performance of the node itself, and not external network issues.
Hmm, might be. OVH's own DDOS protection might be dropping packets too.
I agree, That's why I think looking at active DL/UL rate and overall monthly bandwidth usage, as well as established netstat connections (for 443 should work, more or less) to be able to (at least partially) understand the average use on the node.
It seems like on average in April 2021 (so far)
api.deathwing.meused more than 2.5 TB BW (uploaded) with an average of 10 Mbps. Overall established connections stay at about ~650. I've seen it get close to ~1k whenapi.hive.bloghad an issue a couple of weeks ago. (though this just shows the active users at the time of checking it, so the majority are probably bots.)Even without any statistics and just looking at overall use on logs, I can honestly say that I would not be surprised if
api.deathwing.meis the second most used node on the chain.I was wondering if it would be possible to somewhat implement/upgrade jussi with some new features such as monitoring active connections, calls, chart data etc. In other words, log everything in an organised manner and make it easy so we don't have to scour through tens of gigabytes of logs :P (I know @good-karma was building a project, an upgraded version of jussi, but I don't know it's latest status.)
From my point of view, I'm more interested in the amount of each type of api traffic rather than raw amount as it is more useful for optimization work, that's why I'd like to get the jussi traffic analyzer data.
Would love to see more up to date stats. I remember talking about how one of the nodes in the clusters might be timing out and I think that probably got fixed by now since I haven't seen that in a while on your node.