Quick project info
This software is a "mining" software for a family of cryptocurrencies based on so-called Proof-of-Capacity. The best information about that concept is available in info materials from the BURST coin. You can learn some technical bits from resources on their home page, for example this whitepaper.
Repository
GitHub: https://github.com/quetzalcoatl/blagominer/
New Features
This submission is related to my work on this project between Jul 15, 2019 - Jul 06, 2019. I've made several improvements and added a special mining mode for a new coin type, and released it all as a minor feature release v2.300002.0. (If you'd like to know why the version number looks so strange, I wrote a bit here about the reasons for that)
1. Faster Round Interrupt
Feature branch for this change is feature/fast-round-interrupt.
Some users, including me, observed that sometimes Blagominer acts as if it didn't notice that the current round has ended, and keeps further scanning plot files, instead of interrupting and starting over with new block height.
After analyzing log files, it turned out that in fact the "interrupt" signal was issued to worker threads, but was ignored.
In short, it turned out that the "interrupt" mechanism in worker.cpp : work_i() was too focused. In a multiply-netested loop that is responsible for visiting all plot directories and all plot files, then reading plot files in chunks, and so on - each worker thread only checked the interrupt flag in one specific moment at the end of a phase. If the worker were busy in preceding parts of its job, it could very well fail to notice the flag for a prolonged time. Also, for reasons yet unknown, for some users the worker threads seemed to simply skip that flag check and keep mining till end of input files. This seemed to happen especially on installations with many small plot files, or with very slow plot reading speeds.
Fortunatelly, a simple change of adding more interrupt points at various places in the workflow seems to have resolved these issues for now. Users reported that their miners react to round change much faster and no longer mine at wrong block heights for up to few minutes, losing chances of sending DLs on the new round.
2. Faster Exit
Feature branch for this change is feature/fast-exit.
This is a minor improvement for rarely-used feature, but nevertheless it was irritating. Blagominer supports a few hotkeys, including 'Q' which was meant to simply close the application. However, in many cases, the user had to wait a long time before it actually happened and before the app decided to actually quit.
This was caused by an overlook - the application could be closed in several ways ('q', ctrl+c, close the window, etc), but only one of them actually was setting the exit_flag flag that causes all threads to quit. Fix was obvious: just set the flag in all exit handlers.
3. Fixed Memory Leak
Feature branch for this change is feature/improve-networking-buffers.
Solves issue #2.
The problem is described the best by that linked issue on GitHub. By observing the behavior and memory growth, I came to a conclusion that most probably the leak is located in networking operations, most probably in the updater thread, since the leak seemed to grow regardless of mining operations.
Anyways, since the whole code often used HeapAlloc()/HeapFree() directly with zero exception safely, I decided to review everything and make it safer, hence the name of the branch. I wrote a heap_allocator and changed all places using HeapAlloc/Free to use std::vector with heap_allocator instead. This guaranteed that no buffer can get away without being released, and in fact it fixed the leak.
While at it, I also found and corrected few other issues, like wrong initial size for MemoryStruct buffer for reading responses via CURL, or a hideous bug in the call back handling that response-reading. This one was nasty and hard to notice, since good old memcpy, as a void*-based function, gladly accepted a pointer to vector, instead of a pointer to the contents of that vector. I have no idea how lucky it was it didn't immediately crash. Magic and pure luck, seriously.
4. Fixed Configuration Reading Bug
Feature branch for this change is feature/fix-odd-config-reading-issues.
That was strange! One user reported on Discord that his miner.conf file, while looking totally fine, was rejected by the Blagominer on startup with it stating something about "JSON parse errors".
We've analyzed the config file thoroughly and found no issues, so we started randomly chaning the config file and testing out when the issue occurs and when it does not. It turned out that the problem is related to the filesize being higher/lower than certain threshold. However, other users were successfully using config files several times longer without any problems.
Suspecting issues in the JSON parsing library, I started live debugging - and it actually turned out to be two issues, not one.
First of all, the code opens the config tile in "rt" mode, well known for MS Windows developers for its tricky new line translation. Developer who wrote this code had good intentions, but failed to notice that preallocation buffer equal to filesize will not match the textsize when the newline translation kicks in. For example, file of 15200 bytes were read as 15030 characters, because 170 two-byte new-line markers were translated into 170 one-byte new-line markers.. However, this alone would not be an issue, since the developer used HeapAlloc'ed buffer with HEAP_ZERO_MEMORY flag, so any left over space should be zero'ed out
...or at least, theoretically, because it turned out that the API function used to read the file, fread_s, didn't care about exact memory transfers. In "rt" mode, it actually transferred more bytes than it should. Having read a chunk of a file, it squashed newline markers in-place, and then transferred the chole chunk, with no respect of the fact that the data was now shorter due to the transformation. To be honest, that's in fact not really relevant, as next chunks of data were written at correct places, and only the final chunk of data could habe any chance leaving any trailing trash. And that was exactly what was happening. Comparing original file and contents read by the fread_s:
...and again, that alone wouldn't be a problem, if only the implementor took some care and had checked what was the return value form fread_s. While that function indeed were dumping too many data to the output buffer, it actually was returning the correct number of actual data bytes written, different from the actual number of bytes written to the buffer. Taking that returned number and trimming the data in the output buffer at that point fixed the issue.
5. Added Support for DiskCoin Testnet
Feature branch for this change is feature/support-diskcoin-gensigs. However, it took a fair deal of research, traces of which can be found in rnd/disccoin/deadline-algo
DiskCoin is a new type of Proof-of-Capacity coin that is currently being actively developed and we might expect its release in upcoming months. For now what's available, is a closed testnet (you have to register and apply for participating), and it's already known that the basic miner application is a modded Blagominer, probably from JohnnyFFM, as it lacks even the dual-mining capabilities from andzno1 version.
Needless to say, some of the users tried to mine DiskCoin on their testnet using recent versions of Blagominer and .. failed. It turned out that DiskCoin's Blagominer produced different dealines than the normal Blagominers, old or new versions alike.
Apparently, authors of DiskCoin tampered with the deadline calculation algorithms, and the challenge was accepted :D
How their "algorithm" was discovered is a different story, and it's a longer one, since reverse-engineering takes time, but the main points and obvservations can be summed up as:
- input data from the wallet/pool was totally ordinary
DISKCOIN%-08dwas visible in the EXE file, pretty suspicious considering it shows up neither in the UI nor log files- their miner contains AES128 code that is not present in "original blago"
- their shabal routines and DL formulas seemed unchanged
- ...
- DiskCoin miner gets different deadlines because the miner silently alters the gen-sig received from the wallet/pool, it takes the gen-sig from the pool, builds a 'key' of
DISKCOIN%-08dsubstituting %d for current blockheight, then it decrypts the orignal gen-sig with this key with AES128, and obtains a "corrected gen-sig" in this way, which is then used as if it were the round's gen-sig. - Yes. Seriously. It decrypts it, not encrypts it. It doesn't matter anyway, since the gen-sig is NOT an encrypted message, and AES128 couldn't care less, it just spits transformed bytes out, but nevertheless I find it hilarious to decrypt the gensig that way :D
I have no idea if this "corrected gensig" algorithm is just temporary precaution so the testnet is "safer" from outsiders, and no idea if this scheme will be repeated in the main network. We will see.
Since the gen-sig transformation is clearly dedicated to DiskCoin only, you can enable it on per-coin basis via enableDiskcoinGensigs setting. It's OFF by default.
Also note that since the secret gen-sig transformation, once known, is pretty simple, your pool may automatically apply that transformation to enable users to use ANY classic miner application. If that's the case for your pool, keep enableDiskcoinGensigs turned OFF.
Great update!
Really like to read your contribution posts. These include some great details.
I think you manually merge and push your feature branches so every time I review your posts I need to do some /compare magic on your future branches. Did you consider using github UI for pull requests and merging? (Not questioning your way of working, but this is just what I'm used to see in general.)
Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post, click here.
Need help? Chat with us on Discord.
[utopian-moderator]
Thanks for your kind words, and for the suggestion about PRs. "Manually" - actually, yeah, that's exactly how I work - it's related to the fact that I somewhat like doing merges, it's a great occasion to do a last review and cleanup the code, etc. And I like octopuses :D
However, I know what you mean. Few hours after pushing the code, I wanted to check something, and while it would pretty easy to do with my git client, I thought "hey, I'll check how does that look on GitHub". I went to this post, clicked on a link to that branch...
...and I was totally shocked that I can't readily see what are the actual diffs of those branches. Well, the more I think about it, the more obvious is that GitHub doesn't have any way of guessing where does a branch start to make the diff, but I really expected that there is some default view showing changes versus default branch/etc..
I don't like the idea of PR'ing to my own repo, where I'm the only active person, I like working on one-branch-per-thing, I often reorganize branches while working on them, and PR'ing everything would add (much? some?) "dead" overhead. But I don't see ANY other way to make reviewing easier, so yeah, just for the sake of reviewer's time, I will probably start making PRs. Thanks for reinforcing my observations regarding that!
Thank you for your review, @emrebeyler! Keep up the good work!
Congratulations @ookamisuuhaisha! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :
You can view your badges on your Steem Board and compare to others on the Steem Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPTo support your work, I also upvoted your post!
Defended (26.91%)
Summoned by @ookamisuuhaisha
Sneaky Ninja supports @youarehope and @tarc with a percentage of all bids.
Everything You Need To Know About Sneaky Ninja
woosh
You got a 36.22% upvote from @spydo courtesy of @ookamisuuhaisha! We offer 100% Payout and Curation. Thank you.
Hi, @ookamisuuhaisha!
You just got a 0.04% upvote from SteemPlus!
To get higher upvotes, earn more SteemPlus Points (SPP). On your Steemit wallet, check your SPP balance and click on "How to earn SPP?" to find out all the ways to earn.
If you're not using SteemPlus yet, please check our last posts in here to see the many ways in which SteemPlus can improve your Steem experience on Steemit and Busy.
Hi @ookamisuuhaisha!
Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your post is eligible for our upvote, thanks to our collaboration with @utopian-io!
Feel free to join our @steem-ua Discord server
Hey, @ookamisuuhaisha!
Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!
Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!
Congratulations @ookamisuuhaisha! You received a personal award!
You can view your badges on your Steem Board and compare to others on the Steem Ranking