Below is a list of Hive-related programming issues worked on by BlockTrades team during the last week:
Hived work (blockchain node software)
We launched the first testnet of hived for hardfork 25 last Wednesday and have been experimenting with configuring tinman for it.
Tinman is a testnet management tool for hived that can create accounts and inject various forms of test data into a hived node.
During some of our testing, we found a problem with hived where it would eat up 100% CPU of a core if a hived was launched without a valid seed node because the blockchain code was stuck in a tight loop polling for transactions and blocks that never show up (because the node has no peers to get data from).
Even more problematically, no reasonable diagnostic information was reported in this situation, so it took a little while for us to identify the real problem. We’re currently working on a fix for this issue that lowers the CPU overhead in this case, and also adds warning messages to identify the problem.
We found a problem with beem (the python-based API library for Hive) used for running API tests on hived. @howo provided a quick patch and we’re working on a long term fix now (should be fixed tomorrow).
@howo also made updates to the recurrent transfers function based on specification change requests by our devs, and we are planning a final review of those changes tomorrow, with the plan to merge it into the develop branch for deployment in the next testnet launch.
Jussi caching optimization
Over the weekend, I observed that our hivemind was loaded much more heavily than normal (this was initially noticeable on hive.blog where it was taking longer to open a post to read).
Eventually I found that our node is getting hit with a dramatic increase in the number of bridge.get_discussion API calls (these calls are made when you navigate to a user's post on a site like peakd or hive.blog) and this led to a CPU bottleneck in the python code that process API calls.
It turns out there's a new site that allows for browsing Hive posts, but unfortunately it is currently coded to make this call every second or so (I suspect to detect new comments on the post) and this resulted in a large increase in the number of API calls our hivemind had to process.
The ultimate solution was fairly simple: we reconfigured our Jussi process to cache results of this call for 1 second (previous we had no caching configured for this call).
All API calls first go to Jussi before being passed to hivemind, and if Jussi has cached the answer to a previous API call, then it can just return the previous answer instead of asking hivemind again.
Up till now, we haven't enabled a lot of caching on api.hive.blog other than a few basic things so that we could identify which calls are expensive for hivemind to process, and then optimize the processing of those calls.
But now that optimization work is mostly complete, so we'll be taking a look soon at ways to optimize our Jussi caching to reduce load on hivemind and hived, as this should allow for a substantial scaling increase for our node.
Hivemind (2nd layer applications + social media middleware)
We isolated the cause of the memory leak in hivemind: it appears to be a case where python isn’t releasing memory from dictionaries when the data in the dictionary is cleared. We’ve added code to do a “deep clean” of those dictionaries. We'll likely have performance measurements late next week on how much memory this saves at the current block height.
As a side note, I believe current nodes can recover this memory prior to obtaining our fix by stopping and restarting their hivemind instance.
We’ve also been making some changes to hivemind tests based on the change in the way community muting is being implemented, and those changes will probably be merged into the develop branch tomorrow.
We’ve been running full hivemind syncs on several systems to benchmark performance under different hivemind configurations. We found a slowdown on one machine running with one new hivemind command-line option and we’re trying to analyze if it is due to the use of the new command-line option or some other configuration issue on that system (e.g. hardware or database configuration). At this point, we’re trying various experiments to isolate the root problem on that system.
Modular hivemind (framework for hive applications)
This week, we completed the first pass work on the fork resolution code for modular hivemind using a fully SQL-based solution relying on shadow tables that save changes that need to be undone in the case of a fork switch.
We’re currently creating an architecture document to describe how the fork resolution code works and how to use it to create a Hive-based application.
Updating hived testnet
We’re planning to launch an updated testnet on Thursday with the latest fixes to hived discussed above.
That will be followed by a launch of an API node configured to draw data from the testnet (probably on Friday, if all goes well).
This API node will allow Hive applications to begin adding code changes to support new features added by the hardfork, such as vote expiration reporting.
New hivemind release planned soon
In the next week we’ll continue testing the latest changes to hivemind: first more performance testing of hivemind sync, then real-world performance testing on api.hive.blog.
If all that testing goes well, we may be able to release a new version of hivemind to API node operators by the end of next week. This release would contain the various bug fixes and performance optimizations reported in previous posts.
SQL Account history plugin for hived (requirement for modular hivemind)
We planned to get to this task last week, but unfortunately it was delayed by other tasks. But we hope to complete the changes to the SQL account history plugin that pushes data to postgres (eliminating the need for hivemind to pull the data via RPC calls) in the new couple of days.
After the plugin changes are completed, we’ll run a simultaneous replay of hived with a full sync of hivemind to measure the speedup, and see if it matches our expected improvement in hivemind full sync time (we expect 2 days versus the current 4 days required).
Anticipated hardfork date
I believe we still need to do more testnet-based testing before notifying exchanges of the new version of the code, and since we want to give them approximately 30 days to update after we have a well-tested code release, I think we're still at least 1.5 months out from a possible hardfork date.
Interesting, with each report I learn something about the architecture of HIVE. For the HF25, better to take time to have something well tested, we need it to be robust to avoid any long downtime during the release (there are many more users now than at HF24). I know that these are banalities and you already don't care about the pressure of a few about a fast release but I wanted to remind it.
Do you plan to update the HIVE white paper after HF25 so that it no longer correlates with the STEEM white paper given the major differences?
Do you plan to create a graphical representation of the full architecture with the different layers to better understand how HIVE works? I think this would be useful for a better understanding and to avoid design errors when developing an application around HIVE, I'm thinking in particular of the different API calls coming from the different layers, some of which are very similar in the result and knowing which one is the most relevant for our use case is not obvious.
We'll definitely be updating architecture-level documentation in the future.
We'll also be creating docs for how to implement Hive apps using modular hivemind technology once it is near to deployment. Modular hivemind allows for a faster and much more streamlined form of Hive app, but the design of such apps will be quite different in terms of API usage than current Hive apps. This is because with a modular hivemind-based app, the app designer will be able to create custom APIs to suit their needs as well as being able to incorporate sets of modularized, standardized API functionality.
!PIZZA
@mintrawa! I sent you a slice of $PIZZA on behalf of @darkflame.
Learn more about $PIZZA Token at hive.pizza
It is so great to have you not only as technical experts helping with the development but doubling as supporters for the content providers here. Always loved your transparency and advocacy especially with your top-notch applications.
Leave the community and marketing to us and keep doing what you are doing!
What they said! ;)
Which site/app is aggressively calling? I might be able to give them a hand in scaling that back or migrating some of the load onto their own systems rather than your infrastructure. I hope it's not something I'm working on, if it is please let me know and I'll work to get the calls moved over to my boxes, in my early HIVE app building days I know I had a number of run-away scripts that would absolutely abuse the API node they were aimed at, today I use my own infrastructure wherever possible and will gladly help other fledgling developers building on HIVE to do the same.
It's the Dollar Vigilante's site, but don't worry about it, our caching solution completely took care of the problem.
Ahh. Ok. I was recently talking to some new to HIVE devs helping them get some stuff going like witness node and whatnot and was fearing I'd given instruction leading to API spamming code being created. phew.
Heh, Jeff Berwick's site eh? I'm not even sure I want to know.
I didn't realise that the hardfork was so close. For me, the new changes to curation are the most exciting thing. It will for sure give the bots a run for their money. Most of the rest flies a bit over my head but words like performance, optimisation and speedups all sound great too.
Just to be clear, the 1.5 months is very much a "best case" estimate. It could easily be a month or more later, if we found any significant issue during testnet testing.
It wouldn't even have to be some new issue, as we're also doing more testing of the longstanding functionality of the blockchain that has never been fully tested to my complete satisfaction.
A minor example of such a longstanding problem was the cpu loading I mentioned in the post that we found when our testing node wasn't connected to any peers. But that's a relatively minor issue that could be fixed later without a hardfork (and it is also a small enough issue that we can just fix it quickly). If we find any definitive consensus bugs that need fixing during our testing, they could easily delay the hardfork by several weeks.
I am always happy to wait until everything is running to everyone's satisfaction and wouldn't want to see things rushed out to hit deadlines. I just thought that we were much further away from the best case.
Thank you for writing stuff that I do too :D Looking forward to the review tomorrow.
So was the other solution ditched or is it still wip ?
The shadow tables solution was our preferred solution, because it doesn't require a C-based extension, and we found that bugs in a C-based extension can take down an entire postgres server.
So if performance of the SQL-based solution is as-good or better than the C-based extension, we'll go with the shadows table implementation. Currently, I expect that we will end up going with the shadow table solution.
I see, sounds like the better solution going forward even in maintainability. I doubt that many people are familiar with C based postgres extensions
Sort of off topic from this nicely illustrated dev update, but have you got a ball park on when your team will begin rolling out decentralized bytecode execution for testing and whatnot? Quite interested to build bridges between all of the different already existing and future side / sub chains here on HIVE. I think that when HIVE has the capability to emulate EVM in a secure and properly decentralized manner we're very likely going to see a massive influx of outside interest and capital currently being tossed around on crap like BnB as the digital gold rush of new crypto users seek out alternatives with stronger fundamentals.
I don't have any firm estimate on timing of smart contract processing yet, as we won't be diving into the design phase until we complete the modular hivemind work that it will be built on top of.
My best guess now is that we will have modular hivemind tech ready relatively soon (in a month or two). But I don't have any real estimate on the smart contract processing code itself yet, as that task won't be fully scoped until we start (and complete) the design phase. A lot depends on how much functionality we decide to incorporate into the initial release.
Oh man, bah. I was under the impression that you guys were getting along on a working prototype already.. 0_0 That's ok though I guess. Just means as a community we're a slight bit behind where I thought we were on getting industry standard smart contract execution going here on HIVE.
The modular / plug and play style hivemind will be cool, a bit too hardware hungry for my budget sadly (I have the server, just not the storage) but for those with existing setups that step of allowing modules to be dropped in as necessary should really bring hivemind into the greater communities awareness.
I guess one more quick question, are your guys going to be supporting multiple languages compiled into bytecode or WASM for your setup or will it be in C, solidity or some other language you support contracts wise?
Trying to figure out what holes I need to cover to have HIVE end up with complete coverage / ability to run any sort of smart contract out there between the small number of planned projects on the horizon, basically whatever you guys aren't doing I'll try to pick up and implement into mine so we're effectively able to offer the same contract capability as any current tech out there. In the next few months I think things should be looking extremely bullish for HIVE as these things start to roll out and we collectively start offering alternatives to the overly expensive or questionably decentralized systems out there.
Modular hivemind is being designed to be pretty light on hardware, that's one of the primary goals. The others are scalability and ease-of-development.
Ah, this is music to my ears. Thank you for filling me in on it and answering my questions man.
very exciting, and thank you to every one for the hard work you are doing.
Thanks for the inovation, hardwork always brings success.
Congratulations @blocktrades! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s) :
Your next target is to reach 250 posts.
Your next target is to reach 90000 upvotes.
Your next payout target is 1125000 HP.
The unit is Hive Power equivalent because your rewards can be split into HP and HBD
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPCheck out the last post from @hivebuzz:
Support the HiveBuzz project. Vote for our proposal!
The later the hardfork, the more hive gets burnt :)
😇 How convinent, eh😉
Nice debugging. It is cool to hear about the spotted problems , it shows transparency...also I don"t see how else we would know if not shared. Personally, it makes me more confident. Good to see that it is not "short range". Hive on 🙂
Thanks for the transparency and hard work ⭐⭐⭐⭐⭐
Thank you for your dedication to the hive community ❤️
Luar biasa