Update on BlockTrades Hive core coding

blocktrades (79)in HiveDevs • 3 years ago (edited)

We’ve spent the past week analyzing the performance of Eclipse hived/hivemind in our API service (api.hive.blog) and making improvements based on real-world loading patterns.

It’s been a very productive week and we’ve resolved most of the outstanding issues reported and performance problems we observed. We’ve also identified and fixed several longstanding issues in the hived mutex locking code that prevented peak performance and led to potential data corruption issues.

Hived work (blockchain node software)

We made a number of non-consensus changes to improve hived performance and fix old bugs:

We reviewed the locking used to protect the chainbase database and found several serious longstanding bugs that showed up under high stress API loading of a hived node. These bugs could lead to data corruption and non-optimal performance. Those fixes are incorporated into this merge request:
https://gitlab.syncad.com/hive/hive/-/merge_requests/143

After fixing these problems, we ran several high stress tests using the siege tool and were unable to “break” hived’s operation anymore. We were also able to get hived to fully utilize all the server’s cores (48 on our test system), demonstrating that the locking is fairly optimal for realistic loads.

As a performance optimization, we enabled filtering of operations returned by one of the most time-consuming API calls, get_account_history:
https://gitlab.syncad.com/hive/hive/-/merge_requests/136
https://gitlab.syncad.com/hive/hive/-/merge_requests/137
https://gitlab.syncad.com/hive/hive/-/merge_requests/139

We made several changes to API responses returned by hived, mostly in response to reports from apps developers:
https://gitlab.syncad.com/hive/hive/-/merge_requests/125
https://gitlab.syncad.com/hive/hive/-/merge_requests/126
https://gitlab.syncad.com/hive/hive/-/merge_requests/133

We also limited the max number of operation of operations to that can be fetched by this call to 1k operations (previously it allowed up to 10k operations) to optimize hived throughput: https://gitlab.syncad.com/hive/hive/-/commit/9a05f4d780fc98b891f51ab4b88ebec89fde467b

We corrected account-impacting for several virtual operations created recently:
https://gitlab.syncad.com/hive/hive/-/merge_requests/134

Hived status

We started a full replay and a full sync early this morning to check all the above changes (full replay takes around 18 hours). We don’t expect any issues, but there is a very small chance for issues due to the mutex locking changes, even though no consensus changes were made.

With these changes, I think hived’s performance and reliability has been substantially improved over its past condition and it seems quite solid in the face of very substantial API traffic loading.

One minor issue still to be resolved is the longstanding issue with hived crashing if it receives API traffic when starting up, so we may tackle that next, since it can be an annoyance when updating several hiveds deployed inside of a API server node configuration.

As this version contains substantial operational improvements, we’ve tagged this version as v1.24.5, and it’s a strongly recommended upgrade for all API server nodes. However, there’s no need for exchanges or consensus nodes to upgrade to this version, since the performance improvements mostly come into play when a node is handling significant API traffic.

Hivemind

We made numerous optimizations and bug fixes in hivemind this past week:

Hivemind status (2nd layer social media microservice)

By late tomorrow I expect we’ll have resolved all the remaining big issues with hivemind responses and performance, other than reputation support (which we’ve intentionally left out until all other issues were resolved).

Condenser + Condenser wallet (open-source code for hive.blog)

Merged in support for decentralized lists:
https://gitlab.syncad.com/hive/condenser/-/merge_requests/135
This functionality is still waiting one last issue to be resolved in hivemind, so it won’t be fully testable until tomorrow when we deploy a new version of hivemind.

Proposals page changes:
https://gitlab.syncad.com/hive/wallet/-/merge_requests/47
https://gitlab.syncad.com/hive/wallet/-/merge_requests/49
https://gitlab.syncad.com/hive/wallet/-/merge_requests/51

Condenser status

Condenser is looking pretty good at this point, but we’ll continue to test and look for any remaining problems.

Hive-js improvements

In order to make hive.blog faster, we modified hive-js to support the new option to filter the operations fetched by the get_account_history call:
https://gitlab.syncad.com/hive/hive-js/-/merge_requests/12
https://gitlab.syncad.com/hive/hive-js/-/merge_requests/13
https://gitlab.syncad.com/hive/hive-js/-/merge_requests/14

What’s next for the week?

Tomorrow we’ll hopefully be deploying a new hivemind with all the performance improvements and fixes we made in the last week, after some real world traffic testing on our API node.

We’ll also be testing condenser and the wallet and looking for fixes and optimizations we can make.

Assuming deployment of latest hivemind goes well tomorrow, we’ll likely still need to make a few more fixes as issues get reported, but I expect that by the latter part of the coming week we can begin to analyze feasibility for various previously suggested features for HF25 and also intermediate changes that aren’t consensus changes that can be released earlier.

#hive #blockchain #software #blocktrades

3 years ago in HiveDevs by blocktrades (79)

$49.56

Sort:

Trending

[-]

bashadow (69) 3 years ago

I'd like to thank you and the crew for all the hard work. I know there are a few of us that just can not wait for something to break so we can bitch and moan and complain, (guilty in the distant past of the internet), about things not working right or the way we want them to work, and never bothering to find out who to report the problems to, (because then we would not be able to bitch moan and complain).

So no bitching, moaning or complaining from me, just a thank you for helping make a system I enjoy work better faster smoother.

$0.12

2 votes

[-]

game.player (56) 3 years ago (edited)

Hey @blocktrades, hive.blog is in urgent need of a language filter(When someone who does not speak English enters the site and only sees posts in English, they do not return.) and change the trend algorithm. And instead of showing how much you earned in the post, show a message in profile like "You have x earnings this week, keep interacting to increase your earnings!"

$0.02

2 votes

[-]

blocktrades (79) 3 years ago

Ok, I'll give some consideration to these ideas and discuss with devs, thanks for feedback.

$0.08

4 votes

[-]

aiovo (62) 3 years ago

yo I like you @blocktrades and the team you guys work hard to keep exchange and other delegation thing ruing
and I don't get any bigs when exchanging so far good work

$0.00

1 vote

[-]

lupega (68) 3 years ago

You have and have done a lot of work as I read in your post, we users see the flaws, but we do not go into detail in the work that this whole system takes.
Thanks to you

$0.00

[-]

alfonsoj (61) 3 years ago

hard work friend @blocktrades we all have to thank you for your work for the proper functioning of the Hive platform. Solved the failure of the notifications that I mentioned earlier 🙏 Do not forget to support me in my Post. Thanks for everything.

$0.00

[-]

forykw (68) 3 years ago

Now I am seeing that fast development of things as we all wished when on the "cornered-platform".

$0.00

[-]

hiro-hive (74) 3 years ago

Thank you for doing all the hard works

$0.00

[-]

danieldedosd2 (70) 3 years ago

Cuando leo publicaciones como estas me siento muy mal porque quisiera saber como ayudar, nosotros los creadores a veces nos sentimos como unos parásitos, y escribo de esta manera porque pienso que no soy el único, he aquí mi pregunta ingenua, ¿Cómo podemos hacer, o más bien que esperan ustedes que nosotros los que solamente nos encargamos de subir contenido ya sea musical como es mi caso, Que esperan ustedes de nosotros? Quisiera poder ayudar más, quisiera que nos enseñaran a conocer más de la plataforma para sentirme digno de poder ayudar a mejorar el funcionamiento de hive, amo esta plataforma porque me mantiene con vida espiritual, y desde que estoy aquí me siento que mi trabajo musical mejora, quiero dar más, enseñen al usuario común de la plataforma a saber como meterle la mano a los problemas para sentirnos dignos de ser usuarios con herramientas para que esta maravilla que es la "Colmena" pueda ser mejor y avanzar hacia lo que buscamos todos que esta red social sea, "La mejor de todas las redes sociales, La más especial, La más digna y la que todos quieran usar cuando la conozcan" Disculpen la cadena... Gracias por traernos esta posibilidad de vivir a través de la blockchaing un mundo de pulcritud...

$0.00

[-]

urun (69) 3 years ago

Good work.

What is the plan next?

$0.00

[-]

good-karma (77) 3 years ago

Great work and thank you for tagging releases, adding changelog, similar approach in hivemind would be great! 🙇

$0.00

[-]

nuthman (74) 3 years ago

I am loving the detailed updates. Thanks!

$0.00

[-]

marki99 (61) 3 years ago

Well done. The amount of work was huge. You should take a week off or something before HF25 talks.

$0.00

[-]

omarcitorojas (77) 3 years ago

Thank you for informing and creating new strategies so that in this platform where we live, things work well and we can function as we all hope; keep up with your hard work that the fruits are showing

$0.00

[-]

themarkymark (80) 3 years ago

Amazing to see all these long standing issues get fixed. I really appreciate all the work you and your team has been contributing.

$0.00

[-]

macchiata (78) 3 years ago

Hello @blocktrades.

I have a question. What is the function of muting author in a community? from what I experience and some hivers, a muted author can still post in the community. Is this a bug from recent HF or has it always been this way? Will there be future update on this feature? as in, a muted author will no longer be able to post in a community unless the moderator/admin lift the ban.

$0.00

[-]

utopiaeducators (-8)(1) 3 years ago

$0.00

Reveal Comment