Scotbot crashed!
Dafuk is Scotbot?
Scotbot is the technology/server that was invented to run SteemEngine. When Steem was attacked and we were forced to create Hive, HiveEngine was born. Scotbot now runs both these services.
Rumor has it Scotbot ate shit yesterday because no one renewed the domain name "steem-engine.com". Hilarious right? Don't even care enough about Steem to renew the domain. Har har.
Yeah well, Scotbot crashing not only wrecks SteemEngine but HiveEngine as well. It's a classic case of centralization. It's cheaper for the Scotbot server to run both networks, but that gain comes at the loss of the obvious vulnerability vector we witnessed yesterday.
It also shouldn't have taken that long to fix, but apparently they refused to work on Sunday? Day of rest and all that. Not very professional if you ask me but that's what you get when dealing wish such small-scale operations.
So why does this affect LeoFinance.io?
Unfortunately, LEO is also a fully centralized tiny community, so building everything from scratch is somewhat out of the question at the moment. LEO servers connect to the Scotbot server and rely on it for a lot of information. A failure of Scotbot to communicate with LEO means leofinance.io will not operate correctly.
Math.random()
Even more concerning than all these dependences are the way that HiveEngine coins are "mined" using the mining tokens. I recently found out that this process actually is "random". Up until now I just assumed that miners generated a known set of pre-determined inflation. In reality, miners have a chance to "mine" coins just like real POW mining.
Why is this bad?
Because real mining actually is random and these fake miners are not. Scotbot is generating random numbers (probably in some super basic-bitch way like JavaScript's Math.random() function) to determine who wins which rewards.
The reason I assumed it did not work this way is because this is the stupidest fucking idea ever ever. Scotbot is closed-source, so we already don't know how things happen behind the scenes. Absolutely nothing stops admins from accessing that server, modifying the random number generator, and gifting themselves free tokens. Worst of all, this kind of cheating would go completely undetected and unchecked.
Theoretically, Scotbot admins could create any HiveEngine token out of thin air and gift it to themselves, but this kind of cheating is blatant and detectible. Meanwhile, virtual mining needlessly adds an absolutely devastating attack vector to the protocol that has no reason to exist. The only reason why it should exist this way is to make it possible to cheat, and perhaps to make it look a tiny bit more similar to actual POW mining (pointless).
If HiveEngine miners simply generated a pre-determined amount of inflation, this attack vector would not exist.
With actual POW mining, the networks know for a fact that no one can cheat. The only way to win is for mining equipment to find the winning lotto ticket hash that has the correct number of leading zeros. Imagine trying to roll a die twenty times in a row and have it roll the same number every time. It's exactly like that, except provable on a worldwide scale.
Meanwhile, Scotbot generates all these random numbers locally, and no one knows how they are generated. Super shady shit that should not exist and has no reason to exist.
Breakaway
Not to worry. The chance that this kind of cheating is actually going on is low. I wish I wasn't such a scrub and I had some nodes of my own running to see who owns which miners and the distribution of inflation... because in all likelihood the statics are probably well within acceptable limits. Maybe someone more important can attest to that.
Regardless, it's happenstances like this that slowly force LEOfinance to break away from the system in which it was created and forge its own path. Scotbot is simply a bootstrap solution until we come up with something better. That was the whole point, after all... get something up and running until we can swap over to SMTs and get these tokens validated by actual witnesses.
Until then we just have to continue the grind and hope for the best. This Scotbot crash is a pretty big setback, as it rudely points out how dependent we are on centralized services. As far as development is concerned, there are always important priority decisions to be made.
Priorities!
Do we continue pushing forward with outward facing development, trying to attract new users and investors, or do we take a step back and clean up the mess we've made behind us? There are arguments to be made on both sides, but when Scotbot crashes and everyone realizes LeoFinance.IO can crash for 24 hours with zero recourse... obviously that is not acceptable and pushes up the timetables to become more independent, away from Scotbot.
Apparently last night @themarkymark figured out some kind of solution to get his STEM token back up and running and was looking to get LEO up and running as well. Perhaps he'll be sharing with the class in the near future.
Posted Using LeoFinance Beta
As one of the people who designed Steem/Hive-Engine and the functionality behind Scotbot (which is a name I have never liked) I have a lot of comments on this post. First of all, I need to clarify that I'm no longer involved with any of these projects so I don't have any "inside information" or influence over them.
This is not accurate. Steem/Hive Engine is a fully open-source platform that runs smart contracts as a second layer on top of the Steem or Hive blockchains. Scotbot is just a service that runs on top of Steem/Hive Engine to process posting and voting rewards using Engine tokens.
There is really no reason that Scotbot should be a separate, closed-source service. The initial design of it (which I wrote) called for it to just be another smart contract (or set of them) that run on the Hive Engine platform. This way the code would be open source just like all Hive Engine contracts and anyone can run their own Hive Engine node and verify all of the results.
If I remember correctly, it was first built as a separate service instead of a smart contract because that was quicker and easier and we were trying to get something up and running asap to keep the Appics platform on Steem at the time. I always intended for it to be migrated to a proper smart contract at some point (and I believe a bit of work had gone into that), but I ended up parting ways with the project shortly after that and I guess that never happened.
As for the mining, I really have no idea how that works because it was all done after my time. However, if it's really not done in a transparent and provably verifiable manner that's pretty bad, especially because it should be pretty simple to do.
Overall, I'm really excited about the fact that theycallmedan is supposedly working on further decentralizing Hive Engine and Scotbot, and I think that would be hugely beneficial for Hive and platforms like LeoFinance.
I'm working on something separate from Hive Engine. I have hopes Blocktrades sidechain will be something we can utilize for the Speak token system. We just need a decentralized way to distribute tokens and I want that value to be brought back to Hive. There are at least a handful of people I know are working on similar solutions, I'm helping a few myself. So I'm confident a solution will be made and I'm happy to help in anyway I can.
I would love to be able to help out and be involved with these things but I really need to be 100% focused on Splinterlands since there are many players and investors who are counting on that. In any case, I'm really glad to see people like yourself and blocktrades working on these problems and I can't wait to see what gets created on Hive over the next year or two!
Cool info dan :)
Hope we get something that values the token and not extract value out of it.
I think a sidechain will extract more value out of hive as it brings in. Sure you don't need Hardforks to change something, but it also shows the weak points hive has at the moment.
In other words, current management hasn’t done anything right since you left 🤨
#bringmattback
I was rooting for (and still am rooting for) @theycallmedan to make decentralizing HE possible. After reading Aggroed’s post and seeing the request from the DHF he made, I was a lot less excited.
Through what Splinterlands alone accomplished on the second layer, I think it became obvious we need more solutions to aid development. In running LeoFinance, it’s become even more obvious to me that the focus should be building good, open-source and decentralized tooling that people like me can leverage. Not closed source MVPs that never get improved
Posted Using LeoFinance Beta
Thanks a lot for dropping by to make that comment!
Posted Using LeoFinance Beta
@yabapmatt do you work on your own project on hive or you do a break from it?
The random number generation for hive-engine is defined here:
https://github.com/hive-engine/steemsmartcontracts/blob/hive-engine/libs/SmartContracts.js#L134
const rng = seedrandom(`${prevRefHiveBlockId}${refHiveBlockId}${transactionId}`);the random number function is using rng:
https://github.com/hive-engine/steemsmartcontracts/blob/hive-engine/libs/SmartContracts.js#L180
The mining source code can be found here:
https://github.com/hive-engine/steemsmartcontracts/blob/hive-engine/contracts/mining.js
Ah so it does indeed depend on the block creation of a single witness.
Good to know.
Thanks for sharing this.
Although now I'm wondering which on-chain transactions are used to create inflation as it pertains to the passive mining process.
like the dice game someone abuses a time ago right?
That dice game was EXTREMELY foolish because it only used the signature from the person who was rolling the die. This RNG uses that in addition to the witness signature and blocknum_id and the previous blocknum_id as well.
Ok thanks :D
But is the same with engine miners right?
I'm going to have to recant some of the stuff I said in the OP because the RNG rules for miners are a lot better than I assumed.
OK, thanks.
It's good to know Hive has enough smart tech people to control code :)
“ Regardless, it's happenstances like this that slowly force LEOfinance to break away from the system in which it was created and forge its own path. Scotbot is simply a bootstrap solution until we come up with something better. That was the whole point, after all... get something up and running until we can swap over to SMTs and get these tokens validated by actual witnesses.”
Well said. This is yet another tally in the column for needing to get the hell off hive-engine as soon as possible. The fact that this happened simply because Aggroed didn’t renew the Steem-engine domain shows exactly how hive-engine is currently being managed: no management at all.
I’ve already said that by the end of 2021, Leo will be off hive engine and on a new solution. Some are skeptical that this is possible, I am not.
We’re starting to build our own APIs and other key infrastructure necessary to remove HE from our site entirely. Many will start to notice these infrastructures being put into place as the site gains dramatic boosts in terms of performance and displays.
Whether the LEO token lives on our own stand-alone second layer that we decide to develop or “SMTs” should they finally get released is up for debate throughout 2021. There are 3 primary groups im keeping an eye on for this development. If they don’t deliver by mid-next year, we’ll start working on our own token layer which will be open-source and decentralized.
HE provided us the opportunity to start small and experiment. Tokenized communities didn’t exist prior to Steem engine. Now that we know the model works, it’s time for a real solution to get built.
We need someone capable to build this solution, not hive-engine.
Posted Using LeoFinance Beta
Posted Using LeoFinance Beta
Really good step. But it also shows we never had on steem or hive something close to working Smts. It makes me sad, we don't have a solution onchain for it.
That would make the path for more "leo like" projects and also helps to grow the system.
100%
Waiting is never an option. And the current solution is simply not good enough, in terms of decentralization but also in terms of UX. If Ned had some decency, he would wait for the SMT release before cashing out and leaving us with half a product, i.e. communities without SMTs.
Closed source. Interesting.
As a thought experiment, what would happen if Scotbot went offline for good?
All the token balances could be calculated from transactions posted to Hive. Correct? Because all the tribes have their token issuance rules and whatnot documented in perfect detail?
So, someone would have to create a new bot like that to implement those rules? Preferably a network of such bots that run some kind of a consensus algorithm. Losing one bot for a reason like that wouldn't shut down the entire second layer. Then the correct state of token balances could be arrived at by replaying all the transactions posted to Hive?
How would the randomness of the virtual mining process affect things?
Posted Using LeoFinance Beta
Yeah that's the one decentralized thing about it.
In theory, balances could be recovered.
However the network can never be decentralized while these foolish miners are implemented using centralized random number generators. It's easy to see on-chain that Scotbot sent mining rewards to account at such&such time, but it's impossible to verify the legitimacy of the behind-the-scenes random number generator. The obvious solution is to eliminate the randomness of it and just pump out consistent ROI. Either that or you could use on-chain data as the seed for RNG.
In that case you could use witness signatures as a seed for RNG, but then that witness could cheat and give themselves the money. Or we could combine 21 blocks for a given seed so the only way to cheat is if all witnesses worked together to do it... which is obviously extremely unlikely.
I dislike the miners. There is no reason whatsoever for the randomness. Statistically, if you buy a large batch of miners, your yields should be quite predictable from nearly the start.
Random number generators are notoriously difficult.
Just cut out the randomness. No one really needs it and it makes restoring the balances impossible because random number generation is not a deterministic process by definition.
Posted Using LeoFinance Beta
On the same day Google crashes and the Treasury gets hacked.
Who wins the prices for dumbest?
Posted Using LeoFinance Beta
lol awesome.
Wasn't aware of Google.
Yeah they got zapped somehow too.
Maybe the Chinese were running a test.
Posted Using LeoFinance Beta
The conspiracy theorists whisper of the next major false-flag being a devastating digital attack. That way the elite has justification to lockdown the digital world and the physical world simultaneously. Watch out for that Digital Pandemic.
Of course that doesn't make a whole lot of sense because such an event would give DLT like x1000 more legitimacy... which is exactly what you don't want if trying to clamp down on freedom.
DLT and mesh networks.
Instant credibility.
Posted Using LeoFinance Beta
maybe google was running on hive-engine secretly??
Posted Using LeoFinance Beta
Don’t forget the DHS!
Posted Using LeoFinance Beta
I didnt see them. Missed that one.
At least Hive-Engine is in good company there.
Posted Using LeoFinance Beta
We only could wish that HE went down due to some similarly abstract or super complex flaw that was exposed by a government organization.
Instead, we went down because Agg's domain didn't renew... L.M.F.A.O....
Posted Using LeoFinance Beta
To be optimistic, I will say that wont happen again.
Posted Using LeoFinance Beta
Not for at least 2 years (or insert_renewal_time_here)
Posted Using LeoFinance Beta
Wait the treasury got hacked? What a fun day.
I was in contact with the dev fixing the problem and there was a temporary fix that was done to allow functionality while the domain was offline. I pointed Khal in the right direction so he can fix it himself as well.
As far as I know, STEMGeeks and LeoFinance are the only tribes that are self hosted, all others are managed by the Hive Engine team.
At least it gave me a chance to recharge.
Posted Using LeoFinance Beta
No rest for the wicked.
There is a phrase in my country that says: "The cheaper Ends up being more expensive" as far as scotbot is concerned that clearly applies
Posted Using LeoFinance Beta
The miner attack vector isn’t the only one. The system is completely centralized without even usable logging and auditing. Any transaction can be manipulated behind the scenes.
There is a shit ton of money in Hive Engine, Leo alone has a market cap of over $1M USD.
Even if HiveEngine was fully decentralized... if you only have one frontend to access the information then again you're trusting that frontend not to lie.
At least some of the information is on-chain and can be verified... but still... given how long this service has been up and running. Pretty sad.
Actually it isn’t. What is on chain is custom Json messages that cannot be trusted. Think of them as withdrawal or deposit slips. You can ask to withdraw $1M and fill out a slip but it doesn’t matter if your account doesn’t have the balance.
Yeah but if you are running your own node that knows the rules you can verify the custom JSONs as legit.
good point
exactly. that's why I never understood how splinterlands is considered a DAPP for example. there is not a single point of actual decentralization in it
Posted Using LeoFinance Beta
We use the term dApp very loosely in these parts.
This is a pretty shocking post... but I wonder why nobody is commenting on it. Does everyone else disagree with this assessment or something?
On my side I noticed the HIVE network in general has been a bit laggy in the last day or two. Is that also related to this issue?
Lol you'd need to give the comments more time, as my post was up less than ten minutes by the time you wrote this :D
Nah this lag is something else entirely.
Hive Nodes seem to be having a lot of trouble today.
Oh wow, that is crazy. I remember a while ago when the main domain for my company expired and I didn't realize it. Luckily no one else snatched it up, but it was a huge nightmare getting it all sorted out. I have a much better plan now and it shouldn't happen again. At least not while I am still working here. Outside of this issue, Scotbot seems to be slow in general. It would be cool to move to something that offers more speed and reliability.
Posted Using LeoFinance Beta
This didn't only affect Leofinance.io but every other preference like Sportstalksocial. It was frustrating that each time I try to get in and view contents, it knocks me out. Even until now it's still faulty, recent posts made can't be view by the creator on Leofinance.io preference except and only on Hive , XD :(
Posted Using LeoFinance Beta
This worries me now for a number of reasons. One of the main is I own a decent amount of these so called miners and I was tempted to start buying into them more.
From running many businesses of my own and starting up others and offering help my "vote" would be to fix the issue now since it's a known issue and it's an issue in which you have no control over. Its a massive failure point that should not exist and should be corrected. Downtime today might not be so bad but downtime a year from now for the same issue that you knew about but didn't want to spend the time to fix can be 10x worse or 1,000x worse when scaled with more users, data etc.
It might seem like a step back and slow down but in fact it's setting you up for success and scalable success you can now build on.
Posted Using LeoFinance Beta
The RNG for mining isn't as bad as imagined, as I admit in my next post.
All we can do is keep on developing and breach those hurdles along the way.
Scotbot has caused leo problems a few weeks ago as well. What I find odd is the UX changes that took place after the issue which are probably created to enhance our experience especially in answering comments, but in reality don't.
Posted Using LeoFinance Beta
Some of the features released today are still experimental. But yeah, coupled with the scotbot issues it is pretty annoying
Posted Using LeoFinance Beta
They will be fixed, I have no doubt.
Posted Using LeoFinance Beta
There's always a loophole for cheating, but then even till now we still have some glitches and the thought of Scotbot admins creating and gifting themselves HE token sounds really malicious. Anyways I'm guessing some of these outages will definitely come, we don't have a 100% flawless system, a little of centralisation here and there but then, I'm glad it's better now.
Posted Using LeoFinance Beta
Never assume :P
This scotbot thing...when it's down all tribes suffer...
Even though they could cheat, I think that they wouldn't take such a risk...
When SMT's?
Seems like that even if it feels like an ancient idea by now...they're still much needed
Posted Using LeoFinance Beta
Even the biggest companies have outages. We've seen that with Google today, but it's the length and the frequency of the outages that makes people walk away from your service.
Ideally you want as less dependencies possible that can cause outages, but all comes at a cost.
I do think it should be a priority to eliminate dependencies and SPOF's for as far it isn't already.
Posted Using LeoFinance Beta
I agree. It sucks and this is why I want some kind of smart contracts onchain. They don't need to be super complex or "smart". As long you can connect onchain Tokens to with some parameters. Then it's saved.
I play with the idea to build a community + token since I ever hear about SMTs. Because I believe this can be the future. And not all token mechanics need to be onchain in a smart contract.
The biggest problem I always see with hive engine, you build a business/community on it and it all depends on it. And in a case like leo, it can ruin the project.
In other words, it can end or be more expensive from one day to another. That's why I'm a big fan of onchain tokens.
With open-source front ends like ecency.com, hive has a good chance to expand. But we need the infrastructure first.
If less technical skills needed to host/install a front end with basic parameters + connect it with a token would be world-changing.
1 Token standard is needed for hive longterm. So exchanges can list them without a degree in Hive science.
thanks for the summing up, why I haven't invested in leo yet
Not a single LEO was lost during the outage. By not having invested in LEO you missed out big!
🦁
Posted Using LeoFinance Beta
who cares
understandable... your loss :D
I'm not here for some little risky profits
also I'm broke
Good!
Except the being broke part.
We just need to wait for Koinos blockchain.
Posted Using LeoFinance Beta
Don't even know whats going on over there.
Check andrarchy's podcast on hive.
Posted Using LeoFinance Beta
Don't get that much hope with Koinos. It will be great I'm sure and I'm buying a ton but it can't replace hive. Choosing to allow smart contracts makes it a completely different beast. What we need is decentralized SMTs and all the second layer stuff blocktrades plans to deliver.
Thanks for letting me know I thought there was a big problem but I know now that was team slacking off
Posted Using LeoFinance Beta
Hey guys I am facing problem commenting on your posts when I type and click on reply doesn't work then I have to copy all the texts and reload the page and then paste it and then I click on reply then its reply
Please tell me or let the team know @onealfa
Posted Using LeoFinance Beta
I can go on about If only SMT was released and working properly we wouldn't be facing these issues! But I also understand we need those scalability related improvements to Hive to keep being a witness profitable in these bear market times.
I'm still hyped to see the devs in community coning together and working to fix these stuff. I was also greeted to this cute thing:
Oh! It's a crypto we can earn $$$@khaleelkazi really gets the marketing part well. We need people from outside coming into LEO for reasons other than
Posted Using LeoFinance Beta
Bwahahahahaaa!
Posted Using LeoFinance Beta
Up until now I just assumed that miners generated a known set of pre-determined inflation. In reality, miners have a chance to "mine" coins just like real POW mining
Posted Using LeoFinance Beta
this, and this really needs to be addressed like yesterday.
Posted Using LeoFinance Beta
In the proverb "A chain is only as strong as its weakest link," that link in the "Leo-Chain" would be Hive-Engine, correct?
These concerns shouldn't be taken lightly by the staff at S/H-E.
Scotbot sucks! I hope it doesn't happen again.
Thanks for this update, @edicted.
Posted Using LeoFinance Beta