The secret is IO =)
Doing a lot of experiments with ZFS lately... which have been proving that with much less RAM, performance grows quite fast and read access to RAM cache is very beneficial for slower IO subsystems.
But of course, if one has NVMe's for very big bursts, then it's like cloud9, and ZFS will eat all that... and it will be even faster. The problem I have been trying to understand is how to best manage ZFS memory usage (now understood) and the next harder part is tunning some parameters on the kernel extension. These are usually part of my daily job, but because I don't use ZFS at work, it's like finding a new fun toy!
...and I understand the pain when using services that don't provide enough visibility on what's below the infrastructure. It's usually the way to make money. Maybe that's why I never liked to run things on the cloud... 🤣 Bones of my own crypt I guess! - but in that regard, I am happy to provide some tricks for virtual providers.
In simple words is trying to use all the "VPS package" resources in the more intelligent way possible, with the right software.
Yeah that's the downside of using cloud hosting. You never really know for sure what you will get and resources may vary largely at various times. Happened to me a few times already that services kept stopping randomly, probably due to some resources being drawn from my particular cloud instance.
Additionally I've been having some weird networking / routing issues on my snapshoting server, which caused very slow downloads for some particular accesses. But all in all I am quite happy with the result / performance.
Until now I've never really looked into ZFS, as the speed of my servers suffices for now (using nvmes).
My new HF26 HIVE node will migrate to ZFS (from LVM+XFS). My raw benchmarks indicated at least 4x performance while synchronizing the chain (when read cache is enough for the workload), compared with ZFS for example (ext4 is even worse). And this is on DDR4 so, on DDR5 things are going to be even better.
Overall, feeling very positive about it... and still not using the latest ZFS version... (using what comes with ubuntu, which is v0.8.3) but it is so easy to manage and has so many features that it becomes a no-brainer.
I had started a post series on @atexoras account and then got so far away exploring that any of what I had planned made sense anymore LOL.. typical for my ways of exploring. Will come, later, when I have more time.
You might like talking to @deathwing, he's doing testing with ZFS as well.
👍