Tower: A REST API implementation on the top of Hivemind

in #utopian-io6 years ago (edited)

The official STEEM public node, api.steemit.com started routing some of the endpoints (follow api + tags api) to a Hivemind instance. It looks like the current migration performs well.

I have been running a public Hivemind node for a while. Current endpoint list of Hivemind is limited, however, it has a powerful database where you can filter and audit blockchain actions.

Hivemind doesn't sync every type of operation into its internal database. But, you can access

  • accounts
  • posts/comments
  • relationships (follows, reblogs, etc.)

with a direct SQL query.

Introducing Tower

Tower is a solution to expose the Hivemind's database as REST API interface. You can have a look at the documentation at tower.emrebeyler.me, and the base URL for the API is: https://tower.emrebeyler.me/api/v1/.

Example Calls (/api/v1/accounts)

  • Get the accounts in Germany, ordered by highest reputation

/api/v1/accounts/?location__contains=Germany&ordering=-reputation

  • Get the most followed accounts (Top10)

/api/v1/accounts/?ordering=-followers&limit=10

  • List the accounts selected @inertia as their witness voting proxy

/api/v1/accounts/?proxy=inertia

  • Get an account detail

api/v1/accounts/emrebeyler/

Example Calls (/api/v1/post_cache)

/api/v1/post_cache/?author=emrebeyler&ordering=-payout

  • Find @inertia's first post in the chain

api/v1/post_cache/?author=inertia&ordering=created_at&limit=1

Example Calls (/api/v1/posts/)

  • Find @emrebeyler's posts where the community was dpoll.

api/v1/posts/?author=emrebeyler&community=poll

  • Find an author's deleted posts/comments

/api/v1/posts/?is_deleted=True&limit=10&author=author


These are just example queries. Make sure to check the documentation for filters and ordering options.

Limitations

Available filters and ordering choices may subject to change in the future. Also, keep in mind that this is an experimental service and depending of the costs, my tower.emrebeyler.me may become private.

Technical Details

Tower is a Python3.6 and Django application. It uses DRF to expose database models as a REST interface.

If you want to run a Tower instance, follow these steps.

Installation

$ git clone https://github.com/emre/tower.git
$ python3.6 -m venv tower-env
$ source tower-env/bin/activate
$ cd tower
$ pip install -r requirements.txt

Configuration

$ vim tower/local_settings.py

Add database information of your Hivemind:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'db_name',
        'USER': 'db_user',
        'PASSWORD': 'db_password',
        'HOST': 'host',
        'PORT': 'port',
    }
}

Running

For development:

$ python manage.py runserver

For production:

$ gunicorn tower.wsgi

Roadmap

  • More filtering options on posts and posts_cache tables. These tables are huge and don't have many indexes. If I come up with a fast solution, I plan to add more filtering options.

  • Additional view sets for relationships (follow, reblog, etc.) These are already handled on the hive's internal RPC server, not a priority but good to have.

Post Updates

Vote for me as a witness

I do my best to support the blockchain with my skills. If you like what I do, consider casting a vote on via Steemconnect or on steemit.com.

Sort:  

Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.

To view those questions and the relevant answers related to your post, click here.


Need help? Chat with us on Discord.

[utopian-moderator]

Thank you for your review, @helo! Keep up the good work!

ǝɹǝɥ sɐʍ ɹoʇɐɹnƆ pɐW ǝɥ┴

What are your storage needs currently? I would like to do something similar, started off doing it in MySQL but getting all the data means gigs and gigs of space is needed.

postgres=# SELECT pg_size_pretty( pg_database_size('hive') );
 pg_size_pretty
----------------
 181 GB
(1 row)

181 GB for hivemind. But beware, Hive doesn't store all the data of the blockchain.

Thanks for this, I see it's a Postgres db in the back then.

I am interested in only a subset of the data. Is it possible to filter posts where the parent id is empty (i.e. only get back posts, not comments)?

Going to read up a bit more.

I am encouraged by the possibilities of what Hivemind can bring to communities and appreciate your efforts in development. I have given you my witness votes to assist you in gaining influence to continue to work towards the sustainability of the protocol!

Posted using Partiko iOS

Thank you! Much appreciated.

i want make a group exchange upvote ..

this is fantastic, nice work. Is adding global properties (steem supply, pricing, block number ....) on the cards at all?

Thanks. Yes, there is a hive_state table stores the dynamic global properties. I will add with the next iteration.

fantastic. Where can I find a full list of tables or structure of the data?

Here is an ER diagram for the current database structure.

Full resolution

brilliant, thank you

I have no idea what you're talking about, but still you had me at Hivemind :-) Does that mean we're close to Communities?

i want make a group exchange upvote ......

Source
When you copy/paste or repeatedly type the same comments you could be mistaken for a bot.

More information:
The Art of Commenting
Comment Classifications

Lol, hivemind topic is hot nowadays. :) But we're not close to communities yet judging by the activity on the github repository.

Hehe you bet! #hivemind will become THAT trending tag on Steem shortly.

I am unable to read what's published on github, I mean i can read it but don't comprehend - so thanks for the translation here :)

Very cool dude will check it out when I fire up a HiveMind instance.

✌️

so were still getting communities?

No.

Actually, the answer is yes 😀

See: Ned's recent post

Awesome work! This is amazing. I can't even imagine the types of things that will now be possible to develop thanks to this. This likely means that developers need to know less about the quirks of working with the blockchain and can treat it more like a standard web service.

Fantastic Work!

Django .. Postgresql .. REST .. Hivemind

Screen_Shot_2019-01-14_at_8.49.58_PM.png
(Yes, it still works :))

Excellent work! Please be aware we just made a huge batch of updates to bring hivemind out of alpha into beta: https://github.com/steemit/hivemind -- many bugs fixed and indexes tuned/added. The latest migrations might take a while.

Great updates @roadscape! and congratulations on the beta milestone.

Reviewing the changes while taking a backup on the primary database. Will try migrating in a cloned database first. :)

Upgraded to beta. 🎉

Hello @roadscape i did not know in what way you can help me on steemit either by steem delegation. I have been posting on steemit but having low upvotes on my post and my account is not growing. Out of the low steem i get, i try to rebid bot to upvote my post so as to get more visibility but still not i am not getting much visiblity to my post.

I would be happy if you can surely help.me out with some good amount of steem power to grow up my steem account on steemit and i would really he happy for that.

Thanks for your helping hand for me to grow on steemit

i want make a group exchange upvote ....

This story was recommended by Steeve to its users and upvoted by one or more of them.

Check @steeveapp to learn more about Steeve, an AI-powered Steem interface.

Selam bu platformda yeniyim ama burası için farklı bir platform tarzı var aklımda bunu projelendirme anlamında yardımcı olabilirmisiniz?

Your posts are really great:) keep it up @emrebeyler

Awesome work @embreyler , I'll experiment with this on my hivemind install today :)

I don't know what it means...but it "sounds" like something good....best of luck.

I wish I understiod all this technical stuff about the blockchain.

Calling @originalworks :)
img credz: pixabay.com
Nice, you got an awesome upgoat, thanks to @mahsumakbas
BuildTeam wishes everyone a bullish new Year!
Want a boost? Minnowbooster's got your back!

Yeah, unfortunately, if you try a wide ordering on posts or posts_cache it happens since it takes a good amount of time to get the results. Try limiting and narrowing the query for a better performance, for now.

What did you try?

Not with the api, yet :) I will add it if it returns in a feasible time to serve in the api.

@emrebeyler I added an index for it (took about 15 minutes) with

CREATE INDEX ON hive_posts_cache((json::json->>'app'));

and then I could query like this

SELECT json::json->>'app' as app, author, permlink, body, created_at from hive_posts_cache where json::json->>'app' like 'busy%' order by post_id desc limit 100

image.png

This is great @jrawsthorne!

How long does this example query take to complete after the index?

Muy buena información, aunque tuve que usar el traductor ;)

Too much info I do not understand

Posted using Partiko Android

No worries, its nerdy stuff.

Hi @emrebeyler!

Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your post is eligible for our upvote, thanks to our collaboration with @utopian-io!
Feel free to join our @steem-ua Discord server

Very nice @emrebeyler! I appreciate your great effort for the steem blockchain.
I have casted two votes for you : one for your poste above and another for you as a witness.
Good luck and best regards
@hasmez

Thank you for the witness vote! :)

Not at all :)

Classy stuff dude!

Very good but what programming system are you using? Is it reliable for all systems from Android windows and mac?

Fantastic

hello sir,I have watched your Post and it was very helpful .I followed all the steps as shown in the tutorial but then also i m getting " java.lang.ClassNotFoundException: org.glassfish.jersey.servlet.ServletContainer.class " error ,why so?

This is awesome!
So you think it is possible to have a search API?
Like searching for posts in specific tags, containing specific field/value in the json_metadata etc...?

Yes. It's possible. But I limited the options on posts and post_cache tables because these tables are huge. To make the queries efficient I need to add indexes targeted for them. For example, Tower will support json_metadata.app filters soon.

Do you have an idea when such search API will be available? With AskSteem being shutdown, I’m looking for a replacement for my @steemtelly project

Posted using Partiko iOS

Probably in the next couple of weeks. What kind of queries do you use? I am gathering this data to create additional indexes on the database side.

Great!
I use the same asksteem queries as on dtube website. They are still there despite the asksteem API returning an error, so you can see what they are exactly. But to summarise:

  • recent dtube posts for an author
  • related content, probably dtube posts that are using the same tag as the current post. The way you make sure it’s a dtube content that has a video is by checking that json_metadata contains the ‘video’ property
  • searching posts by keywords inside title and body
  • searching posts by tag
  • searching posts by author
  • combine the three above with a filter that look for the ‘video’ property in the json_metadata

I believe asksteem is using Solr or Elastic search and you can enter the lucene query in the URL query string

Posted using Partiko iOS

#.... This post is very important for you.............good.........good........good.........IMG_20181129_071229.png

Hey, @emrebeyler!

Thanks for contributing on Utopian.
Congratulations! Your contribution was Staff Picked to receive a maximum vote for the development category on Utopian for being of significant value to the project and the open source community.

We’re already looking forward to your next contribution!

Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).

Want to chat? Join us on Discord https://discord.gg/h52nFrV.

Vote for Utopian Witness!

I have followed you on Steemit, please be kind enough to follow me as well.

I would be happy to do it, but I never receive votes by witnesses...

DO UT DES...

Posted using Partiko Messaging

As always, awesome work @emrebeyler !

This post has been included in the latest edition of SOS Daily News - a digest of all you need to know about the State of Steem.



Congratulations @emrebeyler!
Your post was mentioned in the Steem Hit Parade in the following categories:

  • Comments - Ranked 6 with 72 comments
  • Pending payout - Ranked 1 with $ 203,68

Aweaomees

I want to impove my account plz help me

How would you be able to help Me, if you become a Witness...???

IN GOD WE "TRUST" THE PLAN... January 17, 2019...

How difficult is to allow querying post by multiple tags at once?

wow interesting article

Thanks 😊

Posted using Partiko Android

good please continue !!

great!, finally some cool news for steem... Hopefully this keeps reflecting on price :)

Good!)))

I think it will be good for everybody

This post should make $200 without any bot. Sorin is creating the best content on Steemit at the moment.

Thanks for the work!
I noticed that you was not on my witness list... but it's fixed now :)

Thanks for the information.

Posted using Partiko Android

Steemian great joy beat me with cash. Very great deal!!