For developers using Steem APIs, most resort to libraries that can interface between the language of their choice for their application (e.g. Javascript, Python) and the steem blockchain. Trying to do things yourself can lead to weird results, or worse, a poor implementation that does work -- but might break later without knowing why.
As an example of this weird, inconsistent API behaviour, consider the following JSON-RPC calls.
'{"id":"0","jsonrpc":"2.0","method":"call","params":["database_api","get_dynamic_global_properties",[]]}'
'{"id":"0","jsonrpc":"2.0","method":"database_api.get_dynamic_global_properties", "params": {}}'
'{"id":"0","jsonrpc":"2.0","method":"call","params":["database_api","get_dynamic_global_properties",{}]}'
'{"id":"0","jsonrpc":"2.0","method":"database_api.get_dynamic_global_properties"}'
'{"id":"0","jsonrpc":"2.0","method":"call","params":[0,"get_dynamic_global_properties",[]]}'
'{"id":"0","jsonrpc":"2.0","method":"call","params":[0,"get_dynamic_global_properties",{}]}'
All of the above are valid calls to Jussi -- the Steemit provided API middleware -- and all return the same result. I’m even being generous here, as there are a lot more valid calls; for example, the value for “id” can be anything, the json keys can be provided in any order, and there is also a condenser_api call that can also be used in many ways to obtain the same result -- with all of the this, there is a near limitless number of valid calls to achieve the exact same simple data.
Here's a fun question, why is this call not valid but the above are?
'{"id":"0","jsonrpc":"2.0","method":"database_api.get_dynamic_global_properties", "params": []}'
And, out of all the above example calls, only the following are actually valid direct calls to a steemd API node:
'{"id":"0","jsonrpc":"2.0","method":"database_api.get_dynamic_global_properties", "params": {}}'
'{"id":"0","jsonrpc":"2.0","method":"call","params":["database_api","get_dynamic_global_properties",{}]}'
'{"id":"0","jsonrpc":"2.0","method":"database_api.get_dynamic_global_properties"}'
Yes, the Steem API is a clusterfuck.
JSON-RPC vs REST
Even worse, the above calls are a JSON-RPC "POST" request. What this means is that the operator is sending (“POST”ing) information to the API: saying what method to execute, and what parameters to execute with. For many types of APIs this format can make a lot of sense; methods or parameters may be complex. However, for public facing APIs, the industry standard is not JSON-RPC, but rather what is known as “RESTful” requests. For these types of requests, a differentiation is made that the requested method and parameters are encoded in the URL itself.
As an example, take the following:
https://api.coinmarketcap.com/v1/ticker/bitcoin/
This API from CoinMarketCap gives you a result based on the ticker requested: in this case, bitcoin. When you compare this type of API to the one we currently have for steem, it’s pretty clear which one is more developer friendly.
Designing a RESTful API for Steem
As many people know, I’ve been playing around with steem RPCs and API nodes for a while now. I run a full API node at https://anyx.io , and serve as the main API provider many of the leading steem apps like Busy, Partiko, and Steemmonsters. Notably, this infrastructure is owned, hosted, and operated by me. You can read more about that here and here.
I’ve recently been working on a layer that will accept RESTful requests and convert them into JSON-RPC requests that go to a steemd node. The long-term idea will be able to replace the API in steemd itself, but this will be a “beta” version interpreter to speculate on how the new format would look like.
The layer I have created basically takes the traditional appbase APIs and listens like it was a REST implementation. Some things that aren't supported are the (sort-of deprecated) condenser_api, nested JSON requests, and actual POST data (like pushing signed transactions). I don’t plan to support condenser_api in this format, as honestly it needs to go. (TL;DR: it doesn’t differentiate between actual steemd plugins and gives different results based on which plugins are on the steemd node. Appbase differentiates requests based on plugins.)
How do requests work in this format?
In general, to convert from JSON-RPC format, we build a URL as follows: take the appbase plugin first, followed by the appbase method, then provide any required parameters in URL query format. That’s really all there is to it!
Non-parameter examples:
https://anyx.io/v1/database_api/get_active_witnesses
https://anyx.io/v1/database_api/get_config
https://anyx.io/v1/database_api/get_dynamic_global_properties
Parameters specified (of which you can specify multiple times to make an array request):
https://anyx.io/v1/database_api/find_accounts?accounts=anyx
https://anyx.io/v1/database_api/find_witnesses?owners=anyx&owners=jesta&owners=timcliff
Some fun examples of history:
https://anyx.io/v1/block_api/get_block?block_num=10000000
https://anyx.io/v1/account_history_api/get_ops_in_block?block_num=10000000&only_virtual=false
https://anyx.io/v1/account_history_api/get_account_history?account=anyx&start=-1&limit=10
Some more useful examples:
https://anyx.io/v1/follow_api/get_account_reputations?account_lower_bound=anyx&limit=1
https://anyx.io/v1/follow_api/get_blog?account=anyx&start_entry_id=0&limit=1
https://anyx.io/v1/follow_api/get_follow_count?account=anyx
https://anyx.io/v1/rc_api/find_rc_accounts?accounts=anyx
https://anyx.io/v1/tags_api/get_discussion?author=cheetah&permlink=faq-about-cheetah
And finally, lets bring things full circle. As an example of how much more sane it is for a developer, these two requests are equivalent:
As specified by the docs:
curl -s --data '{"jsonrpc":"2.0", "method":"database_api.get_dynamic_global_properties", "id":1}' https://api.steemit.com
And remember how many options there were for that data payload?
Using our REST layer, there’s only one simple way to get that data:
curl https://anyx.io/v1/database_api/get_dynamic_global_properties
Or hell, just click the link and your browser can display it.
Designing new Middleware?
Steemit has spent a lot of time designing around Steems’ API format. The reason why a custom solution was needed is that there's a need to both parse and understand the JSON RPC request first, in order to understand how to serve it. You couldn't simply scale out servers because you would need to read and interpret the request data before routing it to a server that can handle it.
Swapping over to the RESTful implementation would be a big change: there would no longer be a need for Jussi with this format, as an http server like NGINX can now correctly route based on location directives. Furthermore, traditional systems for caching RESTful requests can be used. No more custom solutions would be needed -- we can move completely over to using well-established open source solutions like NGINX and REDIS directly without writing any custom middleware.
Steps to move forward
Notably, this is not a 'release' announcement of this RESTful layer. The implementation is still very much in the design phase as a work in progress, and there are likely bugs or inconsistencies, and the format is subject to change. This post is instead targeted at curious developers who want to see what other solutions to Steems’ API could look like. In general, I’m looking at getting feedback from these devs.
Mostly, I'd like to know if:
- Do you like this format and design -- is it worth supporting/continuing developing?
- If you're an app dev, would you swap over to using this format compared to using the existing libraries?
- Any comments or suggestions on the format?
Furthermore, if you’re interested in toying around, there’s a few more things I’d be looking for:
- Can you break/crash things?
- Do certain API calls not work?
- Would you be interested in custom API requests for your application?
As an example of (c), I’ve been testing API requests like “given a timestamp, give me the block number” or “what was the original content / diff of this post”. These kinds of API calls can be done without indexing or changing steemd, but can rather be implemented in the API layer to perform some server-side logic to reduce application-side logic.
Personally, I love having logical URLs for requests, rather than having to use curl with POST requests to quickly get data I'm interested in. If I want to get the contents of a block, I can now do so without opening the command line, and rather simply use a browser. It makes so much more sense to be able to link to blockchain data that will be obtained directly from an API node.
I’m hoping that there will be enough interest in this format to push API’s in this direction.
Like what I'm doing for Steem? You can read more about my witness candidacy here:
https://steemit.com/witness/@anyx/updated-witness-application
Then please consider voting for me as a witness here!
https://steemit.com/~witnesses
I'm a part-time steem dev, but I've missed the last 10-15 years of advancements in the technologies you talk about (REST for eg). So I'm not really sure how they work. But I use steem-js and rely on the promise behaviour of that API. Can you achieve asynchronous promise behaviour with this implementation of yours?
So, this layer would actually sit in-between something like steem-js (which is a library for a specific language) and the
steemdAPI node chosen. If you're working at the library level -- you need not worry about the changes here, only whether or not steem-js would adapt to use this protocol rather than the json-rpc protocol it currently uses.At a library layer, one could absolutely implement the same functions and behaviour presented to the user -- like awaits or async promises.
Think about it this way: it would not need to change the way steem-js presents itself to a user, it would change the way steem-js could talk to the back-end that is serving up the results.
The layer I'm focusing on would make API nodes -- like api.steemit.com, or anyx.io -- much more effective at serving multiple applications and users all at the same time, regardless of if they use steem-js or steem-python or use an in-house solution. It's optimization in the middle.
I have a couple of Steem bots and yeah, a restful API just makes so much more sense. Load balancing and Caching Middleware is easier to add in and there's much better support.
For certain lite applications, not need to use a Steem sdk library at all. For example: I have a Google sheet that scrapes web pages on Steemd to get data. I can replace this with a rest call easy and less brittle.
tl;dr zOMG yaaaassss
Posted using Partiko Android
Boom, refreshable web query in excel. It's just that easy.
I agree with you :)
now I have this link in bookmarks to query some data.
Question: Can I use it to broadcast transactions?
Not yet, no. It's definitely still a work in progress, and I've only mapped the GET requests so far (and some I know don't work).
But given some dev time I'll have POSTs working too :)
This is brilliant, I hope that you will open source this to allow anyone to host their own version of this API.
Additionally, it would be great to have some documentation around this implementation even to just reiterate the Steem docs as they can be a bit confusing sometimes.
So, semantically you wouldn't really be "hosting an API" -- this more of a translation layer. But definitely plan on open sourcing it after I get some feedback and formalize the design, as my long term goal is to get everyone moved over to a format like this.
In regards to the documentation, its an interesting point. I actually find the steem api documentation here decent, and I find it quite intuitive to map the "Query Parameters JSON" to the REST format that I'm using. If you try it, it seems natural; for example
{"author": "", "permlink": ""}maps to?author=""&permlink=""I really, really like rest api and I will use it.
Will vote for you.
Thanks!
One big benefit of JSON-RPC is batch requests (https://www.jsonrpc.org/specification#batch). I'm not sure REST could match the efficiency of JSON-RPC in that respect. For example, with JSON-RPC you can query 100 requests in a single connection, while you'd need 100 http(s) connections to get the same results with REST.
Its not a benefit for public nodes, its a huge detriment. Its not possible to cache the large request, it must be parsed and unpacked. REST absolutely can match the "efficiency" by utilizing https/2 and async & parallel programming on the client side.
The result is cache friendly server behaviour and excellent client performance.
Further, rate limiting is much harder when someone tries to batch 100 fat account history requests. Its much easier to do this with parallel requests.
Congratulations @anyx! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :
You can view your badges on your Steem Board and compare to others on the Steem Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPTo support your work, I also upvoted your post!
Do not miss the last post from @steemitboard:
Vote for @Steemitboard as a witness to get one more award and increased upvotes!
Definitely something that could be really useful and really simplifies the requests. Looking forward to following the work on this project. Absolutely makes sense to have this layer.
Dear @anyx
I just visited your account to see if you published anything new only to realize that you seem to give up on Steemit? :(
Hope you're not done with this platform yet.
Yours,
Piotr
Hi @anyx
It's me again. I really wonder if you will even read that comment since you haven't been present on Steemit lately :( I really hope you're okey.
I've been just wondering if you joined PALnet airdrop (PALnet is a token build on Steem-engine).
If you didn't claim it then perhaps I could ask you for small favour. Would you consider claiming your tokens and selling or delegating it to me for few months? I would obviously pay you some STEEM for those tokens (I can even pay for delegation - I want to invest some time and money into that platform).
How to claim your token?
a) login in http://palnet.io/ (it's build on STEEM blockchain, so you will use same login details)
b) login in steem-engine.com
c) claim your tokens -> to do that you need to open steem-engine.com and go to "wallet". In left-top corner you will have PALnet (blue colour) logo. Click on it and claim your PAL tokens.
d) go to your palnet wallet and either transfer your PAL tokens to crypto.piotr or STAKE it and delegate to me (click STAKE and then "advanced" and simply type in my username).
Amount of received tokens depend mostly on your current level of SP (I'm not sure if delegations are being taken into consideration as well).
So based on my current experience, if you have 200 STEEM POWER then you should be able to receive something around 100-150 PALnet tokens.
I would pay you 10 STEEM for purchasing each 100 of those PAL tokens. So basically you can earn something and help me grow on this particular platform, making both of us happy :)
You would earn some extra STEEM and I would have higher STAKE in this platform.
I've only 500 STEEM I can spent on that purpose (plus airdrop will not last forwer) so HURRY UP :) :)
Cheers,
Piotr
Hi @anyx
It's me again. I really wonder if you will even read that comment since you haven't been present on Steemit lately :( I really hope you're okey.
I've been just wondering if you joined PALnet airdrop (PALnet is a token build on Steem-engine).
If you didn't claim it then perhaps I could ask you for small favour. Would you consider claiming your tokens and selling or delegating it to me for few months? I would obviously pay you some STEEM for those tokens (I can even pay for delegation - I want to invest some time and money into that platform).
How to claim your token?
a) login in http://palnet.io/ (it's build on STEEM blockchain, so you will use same login details)
b) login in steem-engine.com
c) claim your tokens -> to do that you need to open steem-engine.com and go to "wallet". In left-top corner you will have PALnet (blue colour) logo. Click on it and claim your PAL tokens.
d) go to your palnet wallet and either transfer your PAL tokens to crypto.piotr or STAKE it and delegate to me (click STAKE and then "advanced" and simply type in my username).
Amount of received tokens depend mostly on your current level of SP (I'm not sure if delegations are being taken into consideration as well).
So based on my current experience, if you have 200 STEEM POWER then you should be able to receive something around 100-150 PALnet tokens.
I would pay you 10 STEEM for purchasing each 100 of those PAL tokens. So basically you can earn something and help me grow on this particular platform, making both of us happy :)
You would earn some extra STEEM and I would have higher STAKE in this platform.
I've only 500 STEEM I can spent on that purpose (plus airdrop will not last forwer) so HURRY UP :) :)
Cheers,
Piotr
Congratulations @anyx! You received a personal award!
You can view your badges on your Steem Board and compare to others on the Steem Ranking
Vote for @Steemitboard as a witness to get one more award and increased upvotes!
I normally save all of my Voting Power (VP) for the Pimp Your Post Thursday (PYPT) Show hosted by @shadowspub. Nonetheless; I found this post worthy of a 100% upvote. Why do you ask?
As a user as well as a small investor in the Steem and Hive blockchains, I am curious about a great many things. While not one that is a coder, I am an old school DBASE programmer. I can sometimes look at code and actually understand what is going on.
When I first migrated to HIVE from STEEM (thank G-D I was not one of the folks that did the I am neutral and will vote for both sock puppet and duly elected witnesses folks) the first step I took was to ensure that I would use your dedicated node. In my experience, it has always been the most reliable one.
Thank you for sharing this information with the more techy types. I am sure they are grateful as well.
Hello
I want to review the operations performed on steem from 2016 to 2018, the steem api I was using was very time consuming and I had to send 26 million requests and collect block information. Now that I have read this post, can I use this link https://anyx.io/v1/account_history_api/get_ops_in_block?block_num=10000000&only_virtual=false
Use and get all the block information so that it is not time consuming and where can I store this information and analyze it? Can you please help me?
Very interesting! I was considering building something like this years ago, but never got around to it. I worked for like five years building the full REST (complete with HATEOAS) api for FoxyCart and know how difficult it can be. I love the idea of having something like this for Steem.
I hadn't heard of HATEOAS, but that's interesting!
So far I have focused on stateless GET requests, as the majority of the interaction with
steemdis actually that. The only real thing to support for POST is just push_transaction, which should be simple.Otherwise, so far everything seems to work pretty smoothly, which is exciting to me!