It was bound to happen...
Things were pretty smooth. A hiccup every now and again, but generally, we've been enjoying smooth sailing on the blockchain for the past few years of building on it.
Then...Last week happened!
We'll go over a few of the issues, some fixes that are in place and how we are going to grow and learn from this but let's start with the elephant in the room:
The Hive Engine dev team released an update for light nodes and domains (v 1.7.2 to 1.7.3), so we decided to be proactive and get our servers up to date with the changes. Things didn't work out that smoothly. And for the better part of the weekend up until a few hours ago, our stuff basically went offline.
Blain has been working around the clock while in the middle of a family move to try to get things sorted and as soon as we found the fix, other witness servers started mentioning there were issues. Hey let's just say we are ahead of the game...But the damage was done.
ListNerds Was Offline
This was a glaring issue that most people noticed when LISTNERDS rewards stopped being sent. There are a few days of rewards that should be sent out tonight or tomorrow and things will be back to normal soon. The good news, we were able to do work with the server to improve deliverability and speed of the queue. But still, the site just wasn't working properly over the weekend.
HivePay Swap Had Issues
This is the biggest issue. Because of the issues with our server, the SWAP feature of HivePay basically stopped working and people sent SWAP but did not get refunded immediately. And then once our server got back online, it actually double paid the refunds.
Which drained HivePay Swap's liquidity. We are sorting through all the data, but from what we can tell, from around block # 65413973 SWAP basically double refunded customers and the pool was drained. It's our bad, and we will make sure we get liquidity back, but we are going to tag everyone that SWAP refunded over the past few hours and will ask for anyone that may have been double refunded to send the SWAP or Hive back to @hivepayswap
We know this isn't ideal and we do apologize for this, but if we can recoup a chunk of the liquidity it will help Hive Pay Swap get back to normal and functioning again.
As for future issues, Blain has found it and put the fix in place so the double refunds don't happen again. Regardless of what happens, the CTP Team will make sure liquidity is back in the SWAP!
Apologies in advance for tagging so many people, but we just want to make sure you check to see if the funds got sent in error:
@hariton
@lanny0938
@aiovo
@forsakensushi
@gadrian
@pu77up
@swaps
@savvyplayer
@simhot
@damla
@marketroll
@iamalivechalleng
@uyobong
@ordet
@cryptokungfu
@colskr
@vxc
@matiflesh
@burlarj
@roger.remix
@diecinueve
@pixelfan
@simhot
@globetrottergcc
@mango-juice
@vxxxc
@eddiespino
@writeandearn
@h3m4n7
@ijat
@birdwatcher
@creodas
@mypathtofire
@matiflesh
@alexicp
@p30505seamew
Again, apologies for tagging everyone, but we hope to see if we can get some help to recover some of these funds.
So all in all....
Growing pains. And a massive inconvenience for our members this past weekend. Which we cannot apologize enough for. We are confident the fixes in place will make sure any updates by Hive Engine in the future will not hurt our services and platforms.
Thank you so much for your patience and understanding and we hope to put this behind us and continue to grow on the blockchain!
Jon, Blain and Eliana
The CTP SWARM
P.S. If you have any questions at all, or need clarification, be sure to join our Discord server here! We are also transitioning all our Telegram communications into Discord going forward, which will help us serve our community much better!
I feel with you. Why all these things happen at inopportune times and over the weekend is beyond me.
Hopefully, people will send the funds back.
Yes, backup node, hot spare with different upgrade policy,... We can all be smart after the fact.
@tipu curate
Upvoted 👌 (Mana: 15/55) Liquid rewards.
Hello it looks like I was sent some hive as a result of this, can you check to see if what came to me was correct or not, I have used hive pay swap so many times, and I often dont even check to see if it came correctly. I will hold onto what I have now until you get back to me, its about 190 hive.
!PIZZA
!LUV
!LOLZ
!PGM
BUY AND STAKE THE PGM TO SEND A LOT OF TOKENS!
The tokens that the command sends are: 0.1 PGM-0.1 LVL-2.5 BUDS-0.01 MOTA-0.05 DEC-15 SBT-1 STARBITS-[0.00000001 BTC (SWAP.BTC) only if you have 2500 PGM in stake or more ]
5000 PGM IN STAKE = 2x rewards!
Discord
Support the curation account @ pgm-curator with a delegation 10 HP - 50 HP - 100 HP - 500 HP - 1000 HP
Get potential votes from @ pgm-curator by paying in PGM, here is a guide
I'm a bot, if you want a hand ask @ zottone444
lolztoken.com
At the end of the night he took her back to her place.
Credit: reddit
@ctptalk, I sent you an $LOLZ on behalf of @forsakensushi
Use the !LOL or !LOLZ command to share a joke and an $LOLZ
(1/8)
Yes, that HIVE was sent in error.
Just sent back 194.138 hive
have a good day broski
Well, good luck.
Now that you're back up and running, this has the chance to be a really cool case study...
We know that alongside the balances in people's wallets, Hive accounts also have a social value tied to them based on their owner's actions and standing within the community.
While Hive's immutable accounts and decentralised network mean you can't force those that got double refunds paid into their account to send them back, irreversible damage could be done to their reputation if they choose not to.
So those users are now faced with 2 choices:
Yeah, science!
Posted Using LeoFinance Beta
That's a really good point. It actually happened once before but to one customer. They sent it back right away and completely understood the mix up. It'll be interesting to see how many send it back once they are aware.
Crossing fingers :)
What an interesting dilemma.
Posted Using LeoFinance Beta
Heard about these issues from another h-e witness.
Yeah man, just a messed up weekend. But we'll get it all sorted :)
Glad you got it fixed and are back and I hope the folks that got money back twice will man up and send the second payment back to you. All the best
I dont have any doubts that the Hive community will work with us :)
as always your response is top notch,
Thanks for the kind words :)
Thank you for the update :) You must have worked tirelessly. Hoping things will work smoothly after the fixes.
Posted Using LeoFinance Beta
Blain hasnt slept. He will make sure everything is 100%
Sorry to hear, hope he is okay despite not having slept.
Salute to you all for all the hard work you do behind the scenes :)
Posted Using LeoFinance Beta
Growing pains. We'd like to think we have run everything flawlessly, ALL the time... but it just doesn't tend to happen that way. Sometimes we simply have to accept that 99% right, 99% of the time is actually a pretty good score.
I, for one, am very grateful for the SWAP utility.
=^..^=
We had a pretty good track record of being online for the past few years. lol Not a bad run!!!
I will completely agree with @thehoneypot, lol - once again you show us all what a great team you are
Thanks man, appreciate the kind words
You all rock. Here is some !PIZZA for being awesome and building a great community.
Thanks bro, rough weekend, but we'll get through it!
These things happen, but I know it was a terrible time for you guys with your moves as well. Oh well, just Murphey's Law.
It is nice to see things looking more normal on @ListNerds this morning.
Thank you for the update, @jongolson, and for the over-the-weekend work from you and Eliana and Blain!
Hello! This is harsh to read, especially
Ouch!
As one of the top hive-engine witnesses (@locolombia), I am interested in figuring out exactly what error occurred and how it could be avoided in the future! Was this an error on the hive-engine side or on the interaction with the hive-engine protocol?
Feel free to reach out to me, if I or my team can do anything to help you guys even an independent code review, we are happy to help. God Bless you and keep building guys!
thanks for the update.
Good to hear that the issues related to witness server has been resolved and listnerds and hive is back to normal.
I am sure most of them will turn-up and refund the amounts done wrongly or errorneously by the system.
I was suprised to see messages on ecency of refunds but no transactions or amounts :-)
That may be a glitch in ecency notification system.
Happy week ahead.
Sorry about the issues guys. I sent 536 Hive to @hivepayswap back. Hope you'll get all your funds back soon. Feel free to reach out if there's anything else I can do.
Knew something was up and never batted an eye. @blainjones @elianaicgomes @jongolson have earned and generated that all important trust factor. I wasn't on the list, but it would have been a no-brainer to have given it back. More importantly, thank you for providing us with the all important Big Ugly Nasty Update and not bs'ing us about it.
!CTP
Glad to have you back!
!ALIVE !LUV !CTP
@ctptalk! You Are Alive so I just staked 0.1 $ALIVE to your account on behalf of @globetrottergcc. (2/10)
The tip has been paid for by the We Are Alive Tribe through the earnings on @alive.chat, feel free to swing by our daily chat any time you want.
I was kind of busy so I didn't realize there were so many issues. I hope you get back your liquidity for Hivepay back.
Posted Using LeoFinance Beta
Thanks for the update and the hard work behind the scene to get things back on track.
Good time to check it out again and see if I can make use of it!
Hi guys,
I just send back the 5 hive I received by mistake.
But...I don't understand why I received it in the first place because...I didn't use your service for weeks. (as far as I remember). So maybe your problem(bug) is larger than a double refund...
We have found the issue and are implementing updates to prevent it from happening again
Great to hear. So, next time you send me HIVE, I can keep it 😉😂
Joking of course. Keep up the good work 👍
hahaha, but hopefully this never happens again
@hivepayswap
do i have to use the website? I have send over hive-engine the swap.hive, but didn't got Hive ;( Thought i can send it on hive-engine.
Sorry about the issues, I saw I was refunded Hive and Swap.Hive. I made the refunds for both below. Let me know if there is anything else. Thanks for all your work guys, I know situations like this are tough.
Wow. That is a Big, Ugly, Nasty Update indeed. Glad you solved it and let's hope it will never happen again. I hope you'll manage to recover the funds.
onwards and upwards
Growing pains indeed....Wishing you the best of luck
I will check mine and hit you on discord
Hey I received only one refund TX (https://he.dtools.dev/tx/4c01a7118fd7d787090bf2724199c03be3aa81d7), I use hivepayswap/io quite a few times and not sure if this refund is by mistake (certainly not double as there is only 1 transaction)
Let me know if it is by mistake I will send the swap.hive back
Yes, those were sent in error
Returned the mistaken refund
Just to update on this. HivePayswap is currently not running until @blainjones finalizes the updates and pushes it out. Please do not try to swap at this time. I will push another update as a reply here as well as in the HivePay and CTP discord servers and the website will be functioning again when it is back up and running
Thanks for the hard work during such inconvenient times in real life. I don't know if it's an option for Listnerds as it grows much bigger, but I know top Hive witnesses have backup nodes to avoid such situations when they upgrade, or if one node goes down. Of course, these are extra costs for redundancy you will hardly ever need.
Posted using LeoFinance Mobile
PS: Returned 115.937 HIVE to @hivepayswap.
good points!👌
!ALIVE
@ctptalk! You Are Alive so I just staked 0.1 $ALIVE to your account on behalf of @sabajfa. (5/10)
The tip has been paid for by the We Are Alive Tribe through the earnings on @alive.chat, feel free to swing by our daily chat any time you want.
Your team are doing a great job. Thanks for all the fixes, effort and help (replies).
Also, I'm very happy about the heart-to-heart talk at the Hive community.
From my side, I'll be patient, open to help and a big thank you.
I have just noticed this mention, but I have already just refunded the balance in Hive that was sent to me by mistake.
By the way, this explains why the service has been disabled during these days, I hope they will soon fix everything and enable it again, this is one of the services I use the most.
Hi Jon and Blain. I totally understand and know how it goes. I'm glad it's almost ready for you to put behind. I knew what was going on. Get some rest.
Barb
PIZZA Holders sent $PIZZA tips in this post's comments:
dkid14 tipped ctptalk (x1)
@forsakensushi(1/10) tipped @ctptalk (x1)
Join us in Discord!
The rewards earned on this comment will go directly to the people(@jongolson) sharing the post on Twitter as long as they are registered with @poshtoken. Sign up at https://hiveposh.com.