The official STEEM public node, api.steemit.com started routing some of the endpoints (follow api + tags api) to a Hivemind instance. It looks like the current migration performs well.
I have been running a public Hivemind node for a while. Current endpoint list of Hivemind is limited, however, it has a powerful database where you can filter and audit blockchain actions.
Hivemind doesn't sync every type of operation into its internal database. But, you can access
- accounts
- posts/comments
- relationships (follows, reblogs, etc.)
with a direct SQL query.
Introducing Tower
Tower is a solution to expose the Hivemind's database as REST API interface. You can have a look at the documentation at tower.emrebeyler.me, and the base URL for the API is: https://tower.emrebeyler.me/api/v1/.
Example Calls (/api/v1/accounts)
- Get the accounts in Germany, ordered by highest reputation
/api/v1/accounts/?location__contains=Germany&ordering=-reputation
- Get the most followed accounts (Top10)
/api/v1/accounts/?ordering=-followers&limit=10
- List the accounts selected @inertia as their witness voting proxy
/api/v1/accounts/?proxy=inertia
- Get an account detail
Example Calls (/api/v1/post_cache)
- List @emrebeyler's posts ordered by highest payout
/api/v1/post_cache/?author=emrebeyler&ordering=-payout
- Find @inertia's first post in the chain
api/v1/post_cache/?author=inertia&ordering=created_at&limit=1
Example Calls (/api/v1/posts/)
- Find @emrebeyler's posts where the community was dpoll.
api/v1/posts/?author=emrebeyler&community=poll
- Find an author's deleted posts/comments
/api/v1/posts/?is_deleted=True&limit=10&author=author
These are just example queries. Make sure to check the documentation for filters and ordering options.
Limitations
Available filters and ordering choices may subject to change in the future. Also, keep in mind that this is an experimental service and depending of the costs, my tower.emrebeyler.me may become private.
Technical Details
Tower is a Python3.6 and Django application. It uses DRF to expose database models as a REST interface.
If you want to run a Tower instance, follow these steps.
Installation
$ git clone https://github.com/emre/tower.git
$ python3.6 -m venv tower-env
$ source tower-env/bin/activate
$ cd tower
$ pip install -r requirements.txt
Configuration
$ vim tower/local_settings.py
Add database information of your Hivemind:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'db_name',
'USER': 'db_user',
'PASSWORD': 'db_password',
'HOST': 'host',
'PORT': 'port',
}
}
Running
For development:
$ python manage.py runserver
For production:
$ gunicorn tower.wsgi
Roadmap
More filtering options on
postsandposts_cachetables. These tables are huge and don't have many indexes. If I come up with a fast solution, I plan to add more filtering options.Additional view sets for relationships (follow, reblog, etc.) These are already handled on the hive's internal RPC server, not a priority but good to have.
Post Updates
- @ausbitbank has a new tower node located at https://hivemind.steemviz.com. Thank you!
- @jrawsthorne has an alternative graphql implementation located at https://steem-graphql.jakerawsthorne.co.uk/.
Vote for me as a witness
I do my best to support the blockchain with my skills. If you like what I do, consider casting a vote on via Steemconnect or on steemit.com.
https://github.com/emre/tower/compare/dff87fad19876efb6dd14c253609c4e4610a63b7..883425100e0841691822e5ea9c4ad3ac7fb2e964
Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post, click here.
Need help? Chat with us on Discord.
[utopian-moderator]
Thank you for your review, @helo! Keep up the good work!
i want make a group exchange upvote
Wooo
Awesome stuff! I've spun up a tower install publicly available at https://hivemind.steemviz.com
no entendí me ayudas
Wow, great! :)
Great Stuff!
ǝɹǝɥ sɐʍ ɹoʇɐɹnƆ pɐW ǝɥ┴
What are your storage needs currently? I would like to do something similar, started off doing it in MySQL but getting all the data means gigs and gigs of space is needed.
postgres=# SELECT pg_size_pretty( pg_database_size('hive') ); pg_size_pretty ---------------- 181 GB (1 row)181 GB for hivemind. But beware, Hive doesn't store all the data of the blockchain.
Thanks for this, I see it's a Postgres db in the back then.
I am interested in only a subset of the data. Is it possible to filter posts where the parent id is empty (i.e. only get back posts, not comments)?
Going to read up a bit more.
I am encouraged by the possibilities of what Hivemind can bring to communities and appreciate your efforts in development. I have given you my witness votes to assist you in gaining influence to continue to work towards the sustainability of the protocol!
Posted using Partiko iOS
Thank you! Much appreciated.
i want make a group exchange upvote ..
this is fantastic, nice work. Is adding global properties (steem supply, pricing, block number ....) on the cards at all?
Thanks. Yes, there is a hive_state table stores the dynamic global properties. I will add with the next iteration.
fantastic. Where can I find a full list of tables or structure of the data?
Here is an ER diagram for the current database structure.
Full resolution
brilliant, thank you
I have no idea what you're talking about, but still you had me at Hivemind :-) Does that mean we're close to Communities?
i want make a group exchange upvote ......
Source
When you copy/paste or repeatedly type the same comments you could be mistaken for a bot.
More information:
The Art of Commenting
Comment Classifications
Lol, hivemind topic is hot nowadays. :) But we're not close to communities yet judging by the activity on the github repository.
Hehe you bet! #hivemind will become THAT trending tag on Steem shortly.
I am unable to read what's published on github, I mean i can read it but don't comprehend - so thanks for the translation here :)
Very cool dude will check it out when I fire up a HiveMind instance.
✌️
so were still getting communities?
No.
Actually, the answer is yes 😀
See: Ned's recent post
Awesome work! This is amazing. I can't even imagine the types of things that will now be possible to develop thanks to this. This likely means that developers need to know less about the quirks of working with the blockchain and can treat it more like a standard web service.
Fantastic Work!
Django .. Postgresql .. REST .. Hivemind
(Yes, it still works :))
Good
Excellent work! Please be aware we just made a huge batch of updates to bring hivemind out of alpha into beta: https://github.com/steemit/hivemind -- many bugs fixed and indexes tuned/added. The latest migrations might take a while.
Great updates @roadscape! and congratulations on the beta milestone.
Reviewing the changes while taking a backup on the primary database. Will try migrating in a cloned database first. :)
Upgraded to beta. 🎉
Hello @roadscape i did not know in what way you can help me on steemit either by steem delegation. I have been posting on steemit but having low upvotes on my post and my account is not growing. Out of the low steem i get, i try to rebid bot to upvote my post so as to get more visibility but still not i am not getting much visiblity to my post.
I would be happy if you can surely help.me out with some good amount of steem power to grow up my steem account on steemit and i would really he happy for that.
Thanks for your helping hand for me to grow on steemit
i want make a group exchange upvote ....
This story was recommended by Steeve to its users and upvoted by one or more of them.
Check @steeveapp to learn more about Steeve, an AI-powered Steem interface.
Selam bu platformda yeniyim ama burası için farklı bir platform tarzı var aklımda bunu projelendirme anlamında yardımcı olabilirmisiniz?
Your posts are really great:) keep it up @emrebeyler
Awesome work @embreyler , I'll experiment with this on my hivemind install today :)
I don't know what it means...but it "sounds" like something good....best of luck.
I wish I understiod all this technical stuff about the blockchain.
img credz: pixabay.com
Nice, you got an awesome upgoat, thanks to @mahsumakbas
BuildTeam wishes everyone a bullish new Year!
Want a boost? Minnowbooster's got your back!
Yeah, unfortunately, if you try a wide ordering on posts or posts_cache it happens since it takes a good amount of time to get the results. Try limiting and narrowing the query for a better performance, for now.
What did you try?
Not with the api, yet :) I will add it if it returns in a feasible time to serve in the api.
@emrebeyler I added an index for it (took about 15 minutes) with
CREATE INDEX ON hive_posts_cache((json::json->>'app'));and then I could query like this
SELECT json::json->>'app' as app, author, permlink, body, created_at from hive_posts_cache where json::json->>'app' like 'busy%' order by post_id desc limit 100This is great @jrawsthorne!
How long does this example query take to complete after the index?
145ms
Muy buena información, aunque tuve que usar el traductor ;)
Too much info I do not understand
Posted using Partiko Android
No worries, its nerdy stuff.
Hi @emrebeyler!
Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your post is eligible for our upvote, thanks to our collaboration with @utopian-io!
Feel free to join our @steem-ua Discord server
Very nice @emrebeyler! I appreciate your great effort for the steem blockchain.
I have casted two votes for you : one for your poste above and another for you as a witness.
Good luck and best regards
@hasmez
Thank you for the witness vote! :)
Not at all :)
Classy stuff dude!
Very good but what programming system are you using? Is it reliable for all systems from Android windows and mac?
Fantastic
hello sir,I have watched your Post and it was very helpful .I followed all the steps as shown in the tutorial but then also i m getting " java.lang.ClassNotFoundException: org.glassfish.jersey.servlet.ServletContainer.class " error ,why so?
This is awesome!
So you think it is possible to have a search API?
Like searching for posts in specific tags, containing specific field/value in the json_metadata etc...?
Yes. It's possible. But I limited the options on
postsandpost_cachetables because these tables are huge. To make the queries efficient I need to add indexes targeted for them. For example, Tower will support json_metadata.app filters soon.Do you have an idea when such search API will be available? With AskSteem being shutdown, I’m looking for a replacement for my @steemtelly project
Posted using Partiko iOS
Probably in the next couple of weeks. What kind of queries do you use? I am gathering this data to create additional indexes on the database side.
Great!
I use the same asksteem queries as on dtube website. They are still there despite the asksteem API returning an error, so you can see what they are exactly. But to summarise:
I believe asksteem is using Solr or Elastic search and you can enter the lucene query in the URL query string
Posted using Partiko iOS
#.... This post is very important for you.............good.........good........good.........
Hey, @emrebeyler!
Thanks for contributing on Utopian.
Congratulations! Your contribution was Staff Picked to receive a maximum vote for the development category on Utopian for being of significant value to the project and the open source community.
We’re already looking forward to your next contribution!
Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!
I have followed you on Steemit, please be kind enough to follow me as well.
I would be happy to do it, but I never receive votes by witnesses...
DO UT DES...
Posted using Partiko Messaging
nice
As always, awesome work @emrebeyler !
Thank you @mcfarhat! :)
This post has been included in the latest edition of SOS Daily News - a digest of all you need to know about the State of Steem.
Editor of the The State of Steem SoS Daily News.
Promoter of The State of Steem SoS Weekly Forums.
Editor of the weekly listing of steem radio shows, podcasts & social broadcasts.
Founder of the A Dollar A Day charitable giving project.
Congratulations @emrebeyler!
Your post was mentioned in the Steem Hit Parade in the following categories:
Aweaomees
I want to impove my account plz help me
How would you be able to help Me, if you become a Witness...???
IN GOD WE "TRUST" THE PLAN... January 17, 2019...
How difficult is to allow querying post by multiple tags at once?
wow interesting article
Thanks 😊
Posted using Partiko Android
good please continue !!
great!, finally some cool news for steem... Hopefully this keeps reflecting on price :)
Good!)))
I think it will be good for everybody
This post should make $200 without any bot. Sorin is creating the best content on Steemit at the moment.
Thanks for the work!
I noticed that you was not on my witness list... but it's fixed now :)
Thank you :)
Thanks for the information.
Posted using Partiko Android
Steemian great joy beat me with cash. Very great deal!!