Latest update on HF24 progress

in #hive2 months ago

This is just a quick update on some of our work on HF24 this week. As always, this is a fairly detailed report, so you can skip to the end if you just are interested in our updated expectations for the hardfork date.

Hived/Hivemind progress

Last week we continued testing and fixing discrepancies between output of the old hivemind and the eclipse version of hivemind.

One of the more time-consuming fixes required us to modify hived itself to report some post rewards information as virtual operations so that the hivemind indexer could store this information into its database.

We also found during testing that we needed to add an additional virtual operation to hived to inform hivemind when a comment was deleted. The related hived and hivemind changes for this are expected to be completed in the next couple of days.

Another “discovery” we made is that we now need to implement hot and trending tracking directly in hivemind (previously hivemind obtained these lists via realtime calls to hived). Our first attempt at this had some performance problems, but we’re planning to implement a faster solution tomorrow.

Until now, we’ve been working with a relatively small number of tests for hivemind (60 tests currently), which were sufficient to keep us busy fixing bugs initially, but we just recently added two new programmers to our hivemind team that will focus on creating a more comprehensive set of tests.

Investigating solutions to SQL view issue

In my previous post, I mentioned a performance problem that arose when we used a query that referenced a SQL view instead of inlining joins directly in the query. We traced this slowdown to a default limit in Postgres on how “deep” the Postgres planner will go when looking for an optimal query plan. Introducing the intermediary view exceeded the default join_collapse_limit. By increasing this value from the default value of 8 to 16, we were able to obtain the originally expected performance. So at this point we’re just investigating the best way to make this change.

Initial hivemind sync performance testing

We’ve started measuring the time it takes for an eclipse hivemind to sync. Our automated test system already tests this for 5 million blocks, and it takes around 40 minutes there. But when we tried this on another similarly configured machine that we were planning to use to measure the time to sync to the chain’s current head block, it took nearly twice as long to sync to 5 million blocks, so we need to figure out what’s causing the discrepancy.

At this point, we have a couple of possible reasons for the discrepancy: 1) the database instances are “tuned” differently, 2) the slower system may be overheating causing the CPU to be throttled, or 3) the slow system is configured with nvme drives rather than SSD drives (which should be faster, but possibly there is some driver problem).

Plans for upcoming week

While doing more testing of the new snapshot feature that allows fast saving/reloading of state information, we found that we weren’t storing the state information from plugins (we were only storing the state information in the core code), so we’ll be adding a mechanism for signaling plugins to save off their state data and implementing this functionality for the account history rocksdb plugin.

As previously mentioned, we’ll also be adding virtual operations for deletion of comments and creating more hivemind tests.

Micro-fork handling in hivemind

While working in hivemind, we found that hivemind’s current handling of forks is less than ideal. The forked blocks themselves are discarded from the hivemind database, but the effects of the forked-out blocks are retained in the database. With the movement of more data from hived to hivemind, this “solution” becomes ever more problematic since hived’s internal fork recovery mechanism would previously mask some of these problems, so we’re planning to look at a more robust method of handling forks in hivemind.

After a couple of days of discussion, we have at least two viable methods to investigate: 1) storing additional data into the tables to allow the effects of blocks to be rolled back up to the last irreversible block or 2) using a 2nd copy of the database that’s filled up to the last irreversible block that we can switch to in the case of a fork, and then play the fork blocks onto this second database (the mechanics of this are a bit more complicated, but that’s the general idea). We’re generally favoring method 1 right now, since it seems likely to be faster, and we currently think it can be done without too much difficulty and without requiring ongoing code maintenance to support it (the solution is generic to the types of data being saved and updated in the database).

We don’t plan to work on improved micro-fork handling until after we’ve released a fully tested hivemind, since it’s not a feature that will be needed by third party apps during their own testing work.

Updated release estimates (TL;DR)

I’d hope we would have a release candidate for hived out this last week, but we had several hivemind bugs that required hived changes AND we still need to make the fix to the snapshot feature, so we’re pushing out the expected date for the release candidate by another week (next Monday). We’re planning to release hivemind within a few days thereafter, barring any further major bugs.

As mentioned in my previous report, we plan to schedule the hardfork itself for 3 weeks after the date we publish the release candidate itself.

Sort:  

Early to mid September sounds like a good release time.

For several reasons, I think it would be great to get some updates from the community perspective on these things - a "what it means/What to expect" kind of thing - especially if it has any impact on the end user layer or experience. The reason I think it is important is that firstly, it creates a lot of discussion and while sometimes messy, I believe this is vital for "buy-in". Secondly, it allows for users who aren't necessarily following all of this to find the information that concerns them through their network here.

Thanks for all your work and these updates :)

As far as impacts on the end-user layer goes, there's been several prior posts that discuss the visible changes of HF24 (and some of the less visible ones), but I suppose it can't hurt to summarize those changes again when we publish the release candidate of hived (hopefully next week). These interim posts are only intended to record problems and low-level design decisions we've made during implementation of HF24 features.

The closer it gets the more apprehension some users will feel and the communication (even repetitive) goes a long way. It doesn't have to be you if course, but it would be nice to get some of the witness views on where it is headed and some "predictions" of outcomes. It is a great opportunity for engagement and the development of interest in Hive from the less technical side.

Last I heard, I think an announcement of that sort was planned using @hiveio account.

Let’s keep on BUIDLing and improving our Hive Ecosystem

Love your fast exchange 💱 thank you for this💗

This is gonna sound crazy but I honestly love Blocktrades, super easy to use and I'm so glad you are here on HIVE!

Doesn't sound crazy to me, and I'm glad you like our service. Cheers!

Excellent my friend, I take this opportunity to thank you for your support in my Post's. I hope one day to be as good the Hive as you 🙏

Incredible what you do.

Need help! I could not publish my post on iPhone6 S plus. There are still problems with the title section as I could not see the whole sentence on my screen.

The tag section also has a problem, typed tags kept disappearing once I press on Save post!

Thank you very much.

Cheers.

Thank you very much. I forgot to say that this was happening when using the new Ecency dapp.

Thank you very much.

Ah, that's a very important correction. Please contact ecency developer directly about the bug or file an issue in their repository, and I'll remove the issue from condenser/hive.blog repo.

I'm surprised to read Hivemind requires a new VOP for deleted comments as there is already a delete_comment operation. Is this not enough?

No, because this operation doesn't always succeed. The new vop was to report cases where the operation fails. For more info, you can look at code here: https://gitlab.syncad.com/hive/hive/-/commit/6b944337a007e7738e518961d77822806156e738

Congratulations @blocktrades! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s) :

You received more than 80000 upvotes. Your next target is to reach 85000 upvotes.

You can view your badges on your board And compare to others on the Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP

To support your work, I also upvoted your post!

Keep up the good work @blocktrades, that is all that I cans say, we just needed a robust blockchain that is capable for foing many things for the service of the end users:D

i am having trouble writing posts on my phone after updating the top and bottom view of the review. I have previously asked about this on discord.

image.png

image.png

This is what it looks like when I open the submit post page on my cellphone. I am using a samsung j brand 5. is there a solution. thank you

Yes, it's a known issue, we should have a fix for it deployed soon.

Is it also known that if one is writing a post in HTML and later tries to edit it, the original HTML code isn't shown in the editor anymore (also not if one chooses "View HTML source": for example then there are added many "<br>", so that the post looks completely different after editing it)?
That means to edit posts written in HTML I always use PeakD (which shows the code exactly as it was).

I just tested it with my newest post again, and it still doesn't work correctly.

Another little thing is that in the description of communities embedded HTML links don't work (in PeakD it works fine!).
You can compare for example "https://hive.blog/created/hive-101587" with "https://peakd.com/c/hive-101587/created" to see the difference.

I wasn't aware of it. You can file an issue here for it: https://gitlab.syncad.com/hive/condenser/-/issues

Hivemind sounds very complicated. I thought it simply was a second layer database, which doesn't sound like something that generates tons of bugs.

But then again, I am no dev.

Hivemind is responsible for responding to most of the API calls used by 2nd layer apps, so it's a fairly substantial sized service. And it has gained considerably more responsibilities as part of the eclipse upgrade. Calling it simply a database is probably the wrong way to look at it. It "has" a database, but then so do most web apps. The complexity is better measured in terms of the amount of functionality it provides.

I also don't want to imply that hivemind's design itself is "complicated" in a "difficult to understand" sense, it's not. It's just that there's a lot of queries and changes to the schema often requires those queries to be update. And those queries were also updated to make them faster. Between these two changes, it's a lot of changes to test.

Thank you very much for the detailed report and for the effort!

Sounds like continuous scope increase. :)

Would it make sense to leave out anything non-essential in Hivemind for after the initial release? My understanding is that non-critical bugs, optimizations and such can be addressed at any point in time - potentially after the hardfork, as continuous iterative improvements (something which hasn't been feasible to do for the blockchain code).

I am concerned that as these optimizations are being made, we are losing out greatly by not spending that time on adding Hivemind features which could make a tremendous difference for the platform. Here are some examples of such community-building features, but I'm sure ideas can be sourced from existing communities to see what will help them grow and offer incredible value to people currently using other platforms.

There's been no scope increase. It's more discovery of bugs in current implementation.