Asynchonous code in Python is pretty powerfull. While Python, despite of some threading support, is fundamentally a single threaded language (CPU-intensive code tends to get slower if you add more threads), by making use of the language its async features, you can do much of the stuff that many languages throw threads at in a way more natural and quite efficient way. Instead of threads you have multiple tasks, and whenever a task is waiting for IO or doing an asynchonous sleep, other tasks will get awakened because of IO operations completing.
My aiohivebot library, that I'm currently actively developing does exactly this for HIVE. It starts a collection of tasks, each communicating with its own HIVE public API server (work on adding Hive-Engine nodes as well is on its way), and that's cool, but connecting different asynchonous python libraries can sometimes be a bit of a hastle.
Sometimes different async libs can be a bit of a chalange to fit together, and this can be especialy true when one of the async libs is an asynchonous web framework. The problem with web frameworks is that they often insist on being in the lead, and playing nice with other long runing tasks isn't always a standard feat.
With HIVE being a Web 3.0 ecosystem, web frameworks are pretty important, so I've spent a few days on trying to make aiohivebot play nice with a few top async web frameworks. I managed to get some generic helper functions added to aiohivebot, and I've added one framework specific helper function.
In this post, I'm going to give a simeple example of a completely useless aiohivebot script with a web server serving a JSON report, remember it's not about what the example does exactly, its about the fact that the bot and the web frontend play nice together and form a whole.
A simple bot
#!/usr/bin/env python3
"""Simple demo script that looks for votes on already paid out posts"""
import asyncio
import json
from aiohivebot import BaseBot
class MyBot(BaseBot):
"""Example of an aiohivebot python bot without real utility"""
async def node_api_support(self, node_uri, api_support):
print("NODE:", node_uri)
sup = {
key: ("published" if value["published"] else
("hiden" if value["available"]
else "disabled")) for key, value in api_support.items()}
for key, val in sup.items():
print(" -", key, ":", val)
bot = MyBot()
asyncio.run(bot.run())
This is a dumb bot that processes the per API node reports on the sub-API support of the different nodes. The aiohivebot lib will scan this info a few times an hour and use the info internaly, but its also available to the user defined node_api_support method, and this bot uses the supplied info to print a simple report like this.
NODE: hive-api.arcange.eu
- reputation_api : published
- jsonrpc : published
- condenser_api : published
- transaction_status_api : published
- block_api : published
- account_history_api : hiden
- rc_api : published
- network_broadcast_api : published
- account_by_key_api : published
- market_history_api : published
- database_api : published
- bridge : hiden
- follow_api : hiden
- wallet_bridge_api : disabled
The convenience methods
If we look at the last line of the code, we see:
asyncio.run(bot.run())
What this last line basicly does is that it creates and activates an event loop and within this event loop it invokes the asynchonous run method of the bot.
This is great if there is only one async lib with async tasks to worry about, but what if there were more? The first convenience method we defined for this purpose is the method wire_up_get_tasks()
Now imagine we have two libs like these, one with a HIVE bot and one with a FlureeDB bot with a similar API, how would we make them both run?
async def run_both(bot1, bot2):
tasks = bot1.wire_up_get_tasks() + bot2.wire_up_get_tasks()
await asyncio.gather(*tasks)
asyncio.run(run_both(hivebot, flureebot))
There is a bit of a caveat here, if one of the bots stops operation, the long running tasks in both bots will need to be aborted. This means the two bots will need to be aware of eachother in order for the two bots to be terminated correctly.
There is a second convenience method that could be used to decouple this a bit. The wire_up_get_runnables method returns objects that have both an async run method and a synchonour abort method. I'm not going any deeper into that one, but I hope it's clear that with those you have less convenient but maximaly flexible options.
But there is also a third and a fourth convenience method that we shall demonstrate with starlette.
Hooking up bot to starlette
Starlette, like basicaly every single async python web framework wants to be in control. But unlike a wide range of other frameworks, starlette has a pretty generic way of allowing other code to be hooked in on startup and shut down. The next two convenience methods were made precicely to fit on this pattern.
The wire_up_on_startup method creates the tasks and keeps an internal reference to them. The wire_up_on_shutdown_async method invokes abort to tell all running tasks to abort as soon as possible, and then calls asyncio.gather on the running tasks.
So how does this allow us to hook starlette to our bot. Let's change our code a tiny bit to integrate a dumb web server in our bot.
import json, typing
import asyncio
from aiohivebot import BaseBot
from starlette.applications import Starlette
from starlette.responses import Response
from starlette.routing import Route
class JsonResponse(Response):
media_type = "application/json"
def render(self, content: typing.Any) -> bytes:
return json.dumps(
content,
ensure_ascii=False,
allow_nan=False,
indent=2,
separators=(", ", ": "),
).encode("utf-8")
class MyBot(BaseBot):
"""Example of an aiohivebot python bot without real utility"""
def __init__(self):
super().__init__()
self.supportmap = {}
async def node_api_support(self, node_uri, api_support):
sup = {
key: ("published" if value["published"] else
("hiden" if value["available"]
else "disabled")) for key, value in api_support.items()}
self.supportmap[node_uri] = sup
bot = MyBot()
async def node_api_support(request):
return JsonResponse(bot.supportmap)
app = Starlette(debug=True, routes=[
Route('/', node_api_support)],
on_startup=[bot.wire_up_on_startup],
on_shutdown=[bot.wire_up_on_shutdown_async]
)
Notice how we can just set the on_startup and on_shutdown to hook the two libraries together?
Also notice that there is no run?
This is because starlette code is started through uvicorn or a similar server.
uvicorn server_starlette:app
If we start the bot now, and let it run and press ctrl-C to stop it, we see something like this.
It still takes a few seconds for the bot to start, but everything works as it should.
When we visit the URL, we see some dencently formated JSON like this:
I think the starlette code is pretty much self explanatory, and this isn't a starlette tutorial, so let's leave it as this for now.
hooking up a bot to aiohttp
Starlette is a pretty decent option to use for a async web framework, but if you want something a little closer to the metal, aiohttp is a great alternative. There are many more alternatives, and opinions may differ, but I personaly feel these two are the best options out of the eight of them that I've tried.
For aiohttp we've added its own convenience method to aiohivebot. The wire_up_aiohttp method.
Let's look how it makes our bot look:
#!/usr/bin/env python3
import asyncio
from aiohivebot import BaseBot
from aiohttp import web
class MyBot(BaseBot):
"""Example of an aiohivebot python bot without real utility"""
def __init__(self):
super().__init__()
self.supportmap = {}
async def node_api_support(self, node_uri, api_support):
sup = {
key: ("published" if value["published"] else
("hiden" if value["available"]
else "disabled")) for key, value in api_support.items()}
self.supportmap[node_uri] = sup
async def node_api_support(request):
return web.json_response(bot.supportmap)
bot = MyBot()
app = web.Application()
app.add_routes([web.get('/', node_api_support)])
bot.wire_up_aiohttp(app)
web.run_app(app)
See the code is even simpler than for starlette, but overall pretty similar.
The output when started (as a refular python script) is pretty bare:
And the output of the JSON from the web port isn't as slick because we arent using a pretty print option, but usually this wouldn't be relevant.
What's next?
I've started working on hive-engine support for aiohivebot. It's not complete yet, but its quickly approaching something working. Hive-engine support should be the last feature before the big missing feature, signed operations, is coming. I think that just like the web framework connectivity, hive-engine support is absolutely crucial.
Right now my todo list for aiohivebot looks as follows:
- Add hive-engine JSON-RPC support
- Make layer-2 support modular
- Add signed operations support
- Port the Hive Archeology bot to aiohivebot
- Add an example (plus personal use) frontend to the Hive Archeology bot
- Make a new Hive Archeology docker image
- Integrate coinZdense and coinZdense piggyback mode
- Explore and add support for other decentralized L2
- Look at supporting image hoster signed uploads
- Focus exclusively on coinZdense features and base maintenance and bug fixes.
Available for projects
If you think my skills and knowledge could be usefull for your project, I will be available for contract work again for up to 20 hours a week starting next January.
My hourly rate depends on the type of activity (Python dev, C++ dev or data analysis), wether the project at hand will be open source or not, and if you want to sponsor my pet project coinZdense that aims to create a multi-language programming library for post-quantum signing and least authority subkey management.
| Activity | Hourly rate | Open source discount | Minimal hours | Maximum hours |
|---|---|---|---|---|
| C++ development | 150 $HBD | 30 $HBD | 4 | - |
| Python development | 140 $HBD | 30 $HBD | 4 | - |
| Data analysis (python/pandas) | 120 $HBD | - | 2 | - |
| Sponsored coinZdense work | 50 $HBD | - | 0 | - |
| Paired up coinZdense work | 25 $HBD | - | 1 | 2x contract h |
Development work on open-source project get a 30 $HBD discount on my hourly rates.
Next to contract work, you can also become a sponsor of my coinZdense project.
Note that if you pair up to two coinZdense sponsor hours with a contract hour, you can sponsor twice the amount of hours to the coinZdense project.
If you wish to pay for my services or sponsor my project with other coins than $HBD, all rates are slightly higher (same rates, but in Euro or euro equivalent value at transaction time). I welcome payments in Euro (through paypall), $HIVE, $QRL $ZEC, $LTC, $DOGE, $BCH, $ETH or $BTC/lightning.
If instead of hourly rates you want a fixed project price, I'm not offering that, but there are options I can offer to attenuate cost risks if the specs are close to fully defined at project start. It's a +20/-20/-50/-80 setup. Let me give an example. Say you have a decently defined project that I estimate will take me 60 hours. I could for example offer you two options:
| hour | fixed rate | attenuated risk rate |
|---|---|---|
| 1 .. 50 | 100% | 120 % |
| 51 .. 75 | 100% | 80% |
| 75 .. 125 | 100% | 50% |
| > 125 | 100% | 20% |
Note that this is just an example, but it illustrates my take on providing you with the option to attenuate project cost risks. There is no fixed price, but still a pretty solid attenuation of your financial risk and a strong incentive for me to not go too much above planning.
Drop me a mail or DM me on X if you are interested in my services. I'm not on discord very often, but if need be, you can find me here on discors too.
Contact: coin<at>z-den.se
A lot of what you're doing goes right over my head, but I'm poring through it comprehensively anyway out of interest. Thanks for sharing and may I ask, what's the best way in your opinion for someone to learn programming from a place of not knowing very much at all? Was your experience one of learning a programming language by rote, or did you go straight for building something, learning what was needed as you went along?