Ingestron is a tool for reading the steem blockchain really, really fast into a storm-formatted boltdb database. The next step is to schlep on an API directly, which will of course have native support for swagger, and therefore libs for god knows how many languages and frameworks.
Right now, it just reads and writes into the boltdb database.
I'm hungry for feedback on ingestron. It should make many a steem-project go much, much faster because it'll provide for the first time a fully native (eg: it does not depend on the one fully-developed API library, steem-js) Swagger API for steem. At that point, building front ends should become the trivial task that the good lord intended it to be, eg: "just a front end." (Today you cannot make "just a front end" for steem because it follows no known standards, even the ones it says it does)
Maybe I was a bit tired last night, but after looking at this again, I think I may see part of the problem, starting from when you wrote:
For example, you talk about "saving from latency" and a "fully native" API. From the added context, respectfully, I think those things may mean something very different for me (and for the steemit devs) than they do for you. For example, if you're concerned about latency, clean code, etc, why would you be thinking of adding extra layers of external dependencies and complexity such as grpc?
And if you're already doing pull requests on their code while thinking on a completely different level than they are, well, put it this way. I wouldn't be comfortable touching their code without being pretty darn sure I know it inside and out (short of some very obvious bug fixes).
From your own code, you see that you are already talking to steemd with a websocket! From my perspective, the steemd is already easily accessible from ANY language (and over the net via wss://). It uses json-rpc. It doesn't get much simpler than that! It would be nice, though, to be able to do something like:
to open a stream of order book updates, in json or even better, maybe as a protobuf (to save bandwidth).
In fact, this describes what I was saying perfectly, though I'm not sure why the decision was made to remove asynchronous notifications (which to me means subscriptions):
https://steemit.com/theoretical/@theoretical/how-to-use-the-steem-api
And regarding @officialfuzzy's "promises", I thought he meant this (but perhaps I missed an intended "double entendre" lol):
https://scotch.io/tutorials/javascript-promises-for-dummies
As for latency, you should also read @dantheman's recent post on how he's thinking about "latency":
https://steemit.com/blockchain/@dantheman/blockchains-should-be-designed-like-massively-multiplayer-games
Ingestron isn't here to save you from latency anymore. Ingestron is here to save you from API design these days. Originally ingestron used RethinkDB and was designed to enable real-time updates of page elements that had to be fetched from steemd.
No, fuzzy is talking about an application that i am shamefully overdue in delivering to him, for reasons that are my fault and reasons that aren't my fault, both.
Ingestron is a part of getting that done properly. Tell you what alex, I will make another post that explains how ingestron would change how people build apps using steem. So, stay tuned. It'll be complete with a diagram.
Dan's recent post was awesome. He's got a lot of brain in that brain pan of his, for sure!
also, AFAIK, steemd doesn't follow JSON-RPC 2.0, though it claims to. In fact, one can just look at the dependencies for ingestron to see this: Need to go into gorilla/rpc (a completley fine implementation of standard json-rpc 2.0.... and break it.
EG: Steem doesn't follow even the standards that it claims to follow. This sucks for devs, so an API that carefully implements a Swagger API will make the situation much better.
you really need to understand what's going on at a deeper level. if you're having problems with a missing "2.0" in the API, well, that goes to what I said about strict frameworks wanting everything a certain way. The real world's a lot more messy. There should be exception handling for that, and it could just make an assumption as to the version. If you've seen as many different APIs as I have, you'd better understand what I mean.
It may also help to spend some more time in chrome inspect and better understand how both ends talk when you perform an action. Steemit posts feedback in the console for every action it takes, which makes it much easier for people to understand how an actual implementation works.
Hello
I need to analyze blockchain data blockchain data and when from https://hive.hivesigner.com/swagger.json
I use a lot of receiving time. Can you help me use this API?
will it be streaming too so we don't have to keep polling the price feeds every few seconds for updates? lol
Maybe?
Tell me more of what you'd like?
I haven't written a streaming API yet in go. Do you know if libs exist for this?
Put it in user-story format?
this is the way the current steem api works, which is highly inefficient. Just "inspect" the market in Chrome and you'll see what I mean:
The way it should be is more like a pub/sub, ie. sub_order_book(10); // subscribe to receive and update 10 top levels
Instead of repeatedly requesting updates, it feeds them to you automatically as they come in. If it helps, here are a bunch of golang examples:
https://www.google.com/search?q=golang+pubsub+websocket
I guess the real question is can the current websocket implementations already do any of this natively, as opposed to building a new API that polls, which then in turn converts that to a pub/sub style API. (ie. caveat: it's possible I don't fully understand all the features of the current API)
Does it need to use a websocket or can pubsub be achieved by other means?
Pls know I'm a relative virgin with anything involving javascript, and prefer dart or gopherjs due to static typing heaven.
websocket is just a layer over TCP, and you need it if you want a browser to interact with your API. It's how the current steemjs talks, which ironically means it should be able to stream pub/subs already. I would just imagine the front-end would already be using though if it were available. I built a websocket wrapper for delphi a while back. Maybe it'll help you understand better what it is: https://github.com/alexpmorris/WebSocketUpgrade
I know what websocket is, I just prefer to go with standards, and I know websocket is a standard, but.... have a look at this:
https://github.com/jcuga/golongpoll
Let me know. I can do this or websockets, or eventually both. I'm building this to get used, so the thing that is easiest for yall building front ends, is the thing that I'd like to include in ingestron.
Any idea what http long-polling is?
Generally I'd prefer to use http/2 instead of websockets, since well, ugh websockets.
if you're gonna bother doing this, you're really looking at a websockets / tcp / json / protobuf type solution. long polling basically means you poll with a complete HTTP call each time, except without closing the connection. Stay away. websockets are cake for golang, dart, etc. though. It's practically native. what I did for delphi would best be described as torturous by comparison!
https://github.com/gorilla/websocket
There are a number of libs for websockets in go. I will start looking into them and choose one or another.
I'm also totally interested in any excuse to use grpc-web, cause..... I mean.... it's grpc-web, eh?
grpc-web is almost exactly what I just described to you, though ironically it could add more complexity (because it's an extra framework that imposes the framework's way of doing things on you), but it should probably do everything we just talked about as well lol
http://www.grpc.io/blog/pubsub
or maybe not quite, since I just found this too:
https://github.com/tmc/grpc-websocket-proxy
Well, I've surely been looking for an excuse to put it to work. Are you also in the beta?
the beta of grpc? nah, just looked it up after you mentioned. I tend to build my own stuff from lower-level pieces (like the websocket interface on my github, that links directly to zlib to handle inflate/deflate, etc). Probably a bit too low level for most people's taste. lol
But it does give you a certain level of control over some things that you may lose otherwise. As always, depends on the usage case and what you're trying to accomplish.
Goal:
Make it absolutley, utterly, 100% easy AF for people to build front end systems that connect to the steem blockchain, whose github repository I am banned from
....cause.... "reasons!"
Cause "promises" is a better way to say it.
@originalfuzzy are you saying I am banned from steemit's github because I haven't been able to finish what you hired me to do?
Or are you just making idle conjecture?
....and why do you think ingestron exists?
It exists because the current interfaces are hostile to the creation of applications like the one you want built. So I built a new API for steem.
This post has been ranked within the top 25 most undervalued posts in the first half of Mar 27. We estimate that this post is undervalued by $9.63 as compared to a scenario in which every voter had an equal say.
See the full rankings and details in The Daily Tribune: Mar 27 - Part I. You can also read about some of our methodology, data analysis and technical details in our initial post.
If you are the author and would prefer not to receive these comments, simply reply "Stop" to this comment.
#steemdev a good tag for dev stuff.
https://steemit.com/active/steemdev
Thank you kindly! I will tag my next post accordingly.