Proposal to Make HiveSQL Free Again

in HiveDevs23 days ago (edited)

The goal of this proposal is to make HiveSQL free to use again in order to enable as many developers and users as possible to benefit from this valuable tool and develop applications and services for the Hive ecosystem.

In the following document, I may have to mention It Which Must Not Be Mentioned. In order to spare the faint of heart, I'll do my best to avoid it and replace it with Hive. In case that is not possible, I will just strikethrough.

A bit of history

HiveSQL has been one of my first major projects for the Steem Hive blockchain. I initiated it in 2016, at the very beginning of the Steem blockchain life.

At that time, HiveSQL was free to use! Yes, totally free... except for me ;)

Over time, with more and more users and applications relying on HiveSQL for their proper functioning, and with the volume of data steadily increasing, operating costs started to climb.

HiveSQL has regularly undergone several code reviews [1][2][3] and infrastructure upgrades [4][5] to be able to maintain the performance provided at the best level.

A few numbers for the techies

HiveSQL has been running uninterrupted for 4 years now and its database size is over 2TB.

Every day, HiveSQL processes around 11082 requests per minute (graph), daily servicing an average of 25 applications (graph). Among them, well known apps likes Hive Keychain, HiveDapps.com, 3Speak, ...

Hive SQL database indexes are extremely harnessed and are the key to the performances it delivers. Every minute, up to 2 millions index scans (graph) are performed.

Failing to stay free

Inevitably, being able to sustain such hammering had an impact on the costs and at the end of 2017, I launched a call for help and support in order to continue to provide this service with the same top quality level.

The call was not heard and the only solution I found to make HiveSQL survive was to switch to the actual subscription model. Unfortunately, DHF and its proposals system did not exist at that time.

Why a proposal?

My wish has always been to keep access to HiveSQL public and free to everyone. I'm a bit of a utopian and would like to try again to find a way to fund this project while remaining true to the idea of providing it for free.

Others reasons are:

1. To cover costs

Let's call a spade a spade: I also need to cover costs.

For more than 4 years, HiveSQL has proven to be reliable, responsive and highly available.

Providing high-quality services is not free. Setting up, monitoring and maintaining such an infrastructure requires a lot of attention. Maintaining and monitoring the processes that inject data from the blockchain into the database also requires its share of work.

Although costs are managed effectively, they keep growing.

I have tried to keep the subscription fees affordable, but they are no longer enough to cover these costs and increasing the price of subscriptions is not a viable solution.

It is slowly becoming unsustainable for all and I cannot continue to provide this service at loss.

2. Bring back Less fortunate devs and users left behind

It really pissed me off when HiveSQL moved to a subscription model because I saw a lot of developers and users who couldn't afford the subscription get left behind.

This led me to offer a daily subscription, but despite that, it was not enough to slow down the slaughter.

I would really like these people to be able to take advantage of the enormous potential that HiveSQL provides to them in terms of support for applications, analytics or any other use case that they have.

3. Grow the hive ecosystem

HiveSQL is a powerful, fast and efficient tool.

By making HiveSQL available to a maximum of people already immersed in Hive or, even better, to people from outside our ecosystem, I am convinced that we can make it grow.

Knowledge is power and data brings knowledge.

If we give users and application developers free access to the blockchain data, in a way they can easily retrieve and manipulate it, we are providing them with the perfect tool to develop their business and services and in turn to onboard more users.

Request for Funding

This proposal is not intended to reimburse the expenses that have been incurred so far. It is strictly to cover current and future costs in terms of infrastructure and human resources so that HiveSQL can continue to run and enable developers and users to use it for free.

Here are the costs that have been budgeted, taking into account the past experience of several years and my plans for the future:

1. Infrastructure and services ($1900/month)

  • SQL Server (including license)
  • Dedicated Hive private nodes (AH and HM)
  • Firewall
  • Security management and monitoring tools
  • Performances monitoring tools
  • Backup services
  • Language identification services

2. DevOps ($800/month)

  • Infrastructure maintenance and updates
  • Code maintenance
  • Support to users and developers
  • New features implementation (like Sentiment Analysis)

3. WebOps ($300/month)

  • HiveSQL.io website update and maintenance
  • Documentation creation and maintenance
  • Tutorials creation and maintenance

Considering the elements mentioned above, I'm asking for 100 HBD per day for the remaining days of 2020 and the next two years.

Make HiveSQL free again!

Support This Proposal:

Thank you for your support!

Sort:  

I heard a lot about hivesql, is it opensource, I mean the codebase to sync, indexing, etc.?

HiveSQL is more than a simple synchronizer.
It's made of multiple micro-services running simultaneously. It also relies on paid third-party services (ex: language analysis, performance optimizer).
Hard to "opensource" such an infrastructure.

Opensource is always a choice. Setup could be opensourced, documenting process to setup similar environment could be opensourced, paid 3rd party tools could be disclosed (names, why, how they are used, etc.) Micro-services could be explained, etc.

Yes, but all of the processes you describe here are very time consuming and useless if they are not regularly updated.
I didn't say it can't be done. I said it's "hard" to do.

You might be right, as I don't know your setup. Directly asking for 2+ years without first having things documented in case you shutdown or something happens to you, is quite risky to entire community. But if there is already documentation that proofs that anyone could start their own instances, doesn't mean they should do it (given the cost of infrastructure is high as you wrote). It would give a lot more confidence, imho.

I made the proposal a long term one to avoid having to campaign for its funding every N months.
This does not represent a risk to the community because the proposal is not funded in its entirety at one time.
Small payments are made on a regular basis and can be interrupted at any time if the work or services provided no longer meets the expectations of the community.

Your post has been voted as a part of Encouragement program. Keep up the good work!
Dear reader, follow and support this author, Install Android: https://android.ecency.com, iOS: https://ios.ecency.com mobile app or desktop app for Windows, Mac, Linux: https://desktop.ecency.com
Learn more: https://ecency.com
Join our discord: https://discord.me/ecency

Would love to support, just wanted to know if it's going to be open sourced?

Posted using Dapplr

I have already considered it, but as explained in my reply to @sanat, HiveSQL is more than a few lines of code.
Moreover, open-sourcing would also require rewriting it as it uses private libraries that should therefore be removed.
So this is not currently planned, but having the proposal funded could help.

Ok, I can understand. IMO, open sourcing this can be big and you don't need to do it all at once and complete. You can open source it in modules which the community can use and work upon.

Posted using Dapplr

Considering your experience/credentials, the quality of the service and its utility, the proposal is absolutely reasonable and in my opinion should be supported.

Thank you for your appreciation and support @guiltyparties

Happy to support it!
Btw the links didnt work for me.

Please retry. It should work now.

Deploying a proposal is a tricky process as you have to create the proposal post (without knowing the proposal id) then create the proposal, then update your post links with the correct id 😅

Thank you for your support @dalz

Thank you for your unfailing support @traciyork, really appreciated!

I was a subscriber back when it was SteemSQL and I can attest to the stability of what you offer. Furthermore, what you offer allows running some substantially powerful and long queries, the kind that you couldn't query the blockchain itself (not without some tricky engineering a whole lot of Hive Power).

I would love to see @blocktrades and other heavy hitters in the Hive ecosystem support HiveSQL because the possibilities it allows from a dApp perspective are severely underrated and worth paying for. Not having to manage hardware, servers and sync your own database is worth paying for alone. I will be disappointed and disheartened if the Hive community doesn't support this proposal and make it happen.

Just letting you know @arcange that the second link for the proposal on Hive.blog appears to be broken as well.

Thank you for your feedback about your experience with HiveSQL (formerly SteemSQL) @beggars 👍

PS: hive.blog link fixed. Thanks for notifying me.

Love HiveSQL, I've subscribed a few times for specific instances but could not keep up the sub, sorry! I think it's a fantastic tool that could be used by devs and others, like me. Totally supporting your proposal!

Thank you for sharing your feedback about your experience with HiveSQL.
Your support is much appreciated!

You bet. Glad I can help in any way. You do a hell of a lot for the Hive ecosystem. Would like to see you rewarded and supported ..!!!! AND I just saw your latest POST! FANTASTIC! 👍

Things on the Internet are either free or out of reach.

The problem with free is that free things tend to be abused.

I've asked myself: are there any queries that I could perform on HiveSQL that would be worth the $40 a month subscription. So, the sad truth is "No"

IMHO, HIVE would be a happier and more sustainable ecosystem if we had more middle ground structures that minnows and dolphins could afford. I don't think you have to aim at making HIVEsql free. There needs to be some ways for low value accounts to small amounts of computer time on the machine to make periodic queries.

As much as I'm concerned about the closed source nature of this, I think that the benefits it can provide are worth it because:

  • Business analysts can use their standard toolkits to help us make better-informed decisions
  • dApps can get off the ground quickly. If a dApp gains traction the owner can then either support HiveSQL monetarily, code out the dependency on HiveSQL or run their own hivemind nodes.

So, it's a yes for me.

Thank you for your unfailing support @eturnerx, really appreciated!

I support this. If you're doing ruby/rails development, you can access HiveSQL through this library, which wraps the tables in ActiveRecord:

https://github.com/inertia186/hive_sql

Thank you for your support and for the integration with Ruby

I will support this proposal to make it easy for all developers to access the information

Happy to support something usefull... you have my vote 👍

Do you mind me asking if you use the funds from your frequent posts to support HiveSQL costs?
I'm not sure if it's going to get voted through, but maybe you can achieve your goals and receive some reimbursement from these posts.

Also I suggest using some H-E tokens like Leo, Palnet or Neoxian to bootstrap the earnings.

Voted!

Posting to finance a project, it could sometimes work at the very beginning of Steem, when posts received payouts with several zeros.
It doesn't work anymore.

Anyway, thank you for your vote @sk1920.

We need to move out of stagnation, I hope this will help other developers to create some more commotion in our ecosystem.

Voted

I also hope this will inspire developers and users.
Thank you @jocieprosza

Thank you for your support @fbslo, really appreciated! 👍

I support this 100% and if I had more of the technical skills would be very interested in helping.

Thank you support and your help proposal @darkflame
Having HiveSQL free to use might be a good opportunity for you to play with and improve your tech skills 😉

Yes that sounds great I would like to do this. There are a few things I want to make my website do using the Hive blockchain, so I have many questions. Thank you for your replies!

If you have any question, feel free to use the support channel.

Voting for this. I'd really like to experiment more and I think this is a key part of that.

Thank you @brianoflondon. Hopefully you will be able to play with for free.

I'll support you for sure!

Posted using Dapplr

My pleasure! 🙏

Posted using Dapplr

I hope it goes through, I am pretty sure my vote proxy will be voting for it.

Thank you for your support @bashadow, appreciated!

Don't get me wrong but hivesql is a centralized closed-source database. You leave at any moment and hivesql will be gone too.

No problem. These are expected and sensible questions.

  1. Did I left ever left this platform?
  2. HiveSQL should survive me, at least enough time for people to remove their vote from the proposal and stop the funding.
  3. Creating a blockchain streamer and database injector is no rocket science.

Access to all data from the database is fully open, ask those who already use it.

The rest is closed secured for stability and security reasons.
When I first publicly opened it to the internet, some people called me crazy and predicted he wouldn't hold out for long.
They were right: you can't imagine the number of intrusion attempts that the infrastructure undergoes every day.

The real added value of HiveSQL is its design, infrastructure and the professional service that comes with it.

pixresteemer_incognito_angel_mini.png
Bang, I did it again... I just rehived your post!
Week 30 of my contest just started...you can now check the winners of the previous week!
!BEER
1


Hey @arcange, here is a little bit of BEER from @pixresteemer for you. Enjoy it!

Learn how to earn FREE BEER each day by staking your BEER.

Congratulations @arcange! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s) :

You got more than 31500 replies. Your next target is to reach 32000 replies.

You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP

Do not miss the last post from @hivebuzz:

Feedback from the November 1st Hive Power Up Day

You mentioned that you seen a lot of smaller users unable to use HiveSQL once it became a payed subscription. That makes me wonder a few things:

  1. How much growth did you account for with the service becoming free?
  2. Could more calls and larger user base affect the QoS for those whose apps rely on the service?
  3. If growth does start to hurt user experience, will you be seeking additional funding through the DHF or with priority subscriptions?

These are all sensible questions.

  1. Honestly, as I am not a diviner and cannot answer this question. I hope that as many apps and services as possible will benefit from it.
  2. In the past, more apps than currently used this service, without its QoS being affected. I carry out regular monitoring, both in terms of overall performance and at queries level. In case queries that had a negative impact on perfs, I have on several occasions already assisted users to improve their queries.
  3. The current infrastructure (which has been upgraded very recently) is able to support a much greater load and the expected future volume of data. So I do not think additional funds will be required. Having said that, I hope the growth of the Hive ecosystem will contradict me ;)

Sorry I missed this one - have just reblogged!