When people ask me what their project should focus on, or how they could improve their project, I often point out how hard it is for an outsider to engage with their project (front end/app/game/whatever).
Motivation for wanting to process non-hive accounts
You may just not want to support them, as a dev who implemented them, it's a real pain, you have to engineer a lot of things (auth, signup, profile settings, making sure your whole app is compatible with multiple types of accounts etc) when it's usually just handled by the chain.
Here's why we did it:
Steempress is a wordpress plugin. Nowadays most of our efforts have been directed towards the comment section. One of the best ways to get hive to be known is not by catering to people that are already using hive. It's to try to get new people from the outside. That's really fit for a comment section powered by hive.
We can easily "hiveify" any website, they install the steempress plugin, and in an instant they get a hive powered comment section with everything that comes with it (decentralized, uncensorable, powered by rewards etc), and they become a new place where people can find out about hive.
you can view an example on this blog: https://www.blogrevenues.com/photography/how-i-look-at-art/
If you are a blog owner you don't care about your excuses like "oh sorry people can't post on your blog unless they have a hive account", they want their comment section to be as easy to sign up and comment on as any other comment section (usually disqus).
So assuming you don't support non-hive accounts, if someone wants a hive account his options are:
- buy an account ? No way, no matter how cheap, that's just not how people are used to use websites, especially not if it's to comment/like/whatever
- sign up for an account and wait for it to be approved: most people just won't bother to come back even in a day to put the comment they wanted to put
- Sign up for an account instantly but you have to give a ton of info (something with a phone number etc) that won't work either, few people will actually care enough to put that much info, especially if it takes more time to put all the info than to write a comment.
No, what people expect is give username + email + password -> confirm the email -> publish comment.(Ideally you wouldn't even confirm the email) or even better click "login with google/facebook/twitter) and instantly have an account to publish their account with.
And for that you have to handle non-hive accounts.
Architecture:
Sign up flow:
Our sign up flow is like this:
- You enter display name (username), email, password.
- You hit sign up, we tell you to check your emails to confirm your account
- You get an email with a confirmation link, you click on it and it redirects you to the blog. (improvement path here: you should be logged in when you are in the blog)
Or you can just click on sign up with google, and you'll be logged in.
The whole thing takes moments and it's very familiar to what people are used to, which is what we need to aim for: web 3.0 with the ease of 2.0
Actions flow:
Votes:
Those don't get carried over to the chain because there's no point, we simply have a db table that stores votes so we can render them on our comment section.
Comments:
We carry them over using a proxy account:
on hive.blog
(as a side note looks like a forgot to replace a "steem" for "hive" in the code)
on the blog
One limitation with proxying is the fact that an account can only comment once every 3 seconds, but it's quite easy to have a fleet of proxy accounts and make sure there is always one that's available for commenting.
The whole idea is that from the user perspective it's all transparent.
What to do with the rewards ?
We earmark all the rewards and whenever a user earns more than x hive, we let them "claim" a hive account, and send them all the rewards they earned so far. This is a great way to incentivize people to join hive and have an actual account.
let's get to the actual programming:
Database:
We have a local db used for storing non-hive user data with those tables
Use this as a reference sheet when reading the implementation logic.
authors:
Authors contains all the data related to the non-hive user:
most notable fields are
regular_author_uuid => a unique uuid to identify the user
display_name
email
password
profile_image => direct link to the image
about => about/website/location match the usual stuff you see on hive profiles
website
location
verify_id => the code that will be sent to the user to check his email
blog_origin => url of the page of the blog he signed up from (useful for redirecting)
type => how he signed up (email / google / twitter etc), it's important to know how to authenticate an user
(it's not called users for legacy reasons in our infra)
comments:
id <= auto increment id
author_uuid <= foreign key to author
permlink <= actual hive permlink
body <= the text of the comment
parent_author <= author he's replying to
parent_permlink <= permlink he's replying to
root_author <= the author post of he's putting his comment on
root_permlink <= the permlink of the post he's putting his comment on
date <= date
payout <= current payout
likes <= votes
is_paid_out <= boolean, useful for calculating rewards for the user
root_author/permlink is useful for rendering the comment section, I can just query all comments by root author/permlink to fetch all the non-hive comments related to that post.
votes:
author <= post/comment to vote on
permlink <= post/comment to vote on
voter_uuid <= foreign key to author
percent <= 1000 or -1000, it's set the same way as hive for easier rendering
root_author <= same as comments
root_permlink <= same as comments
date <= date
sessions:
used for managing returning users (so they don't have to enter their key for every single action basically) and providing a token that can be stored in the cookies where they can be stolen and it's relatively okay.
both user_uuid and type are set as primary key because a single user may be signed in using different methods on different devices
user_uuid (pk) <= uuid of the user
type (pk) <= type of the user
username <= hive username, display name for regular users
token <= auth token that we issue
private_key <= encrypted posting key of the user or hivesigner token, empty for regular users
expire <= date to define when the session expire in order to auto purge keys when they expire
Implementation logic:
FYI, I am only going over the happy path, and unless it's an important design decision, I am skipping over all of the sanity/security checks, those are usually obvious (always double check what the front end tells you, encrypt/hash stuff etc) and not worth going over.
sign up
sign up using email
Check if a user already exists with this email, take into account that an user might want to sign up with different methods but with the same email.
For instance I could register with [email protected] using my email, then do a google log in with the same google account, you should check for this in your flow, and set the email field as unique in the db.
As for the flow:
- We generate a regular_author_uuid using a uuid v4 (https://en.wikipedia.org/wiki/Universally_unique_identifier)
- Then generate a unique verification id
- We store the email, hash the password
- Set the user type as
regular(so we know how that user was created and how he should be authenticated) - Set the blog_origin as the exact page the user registered on, it's useful for redirecting and for statistics.
- We leave the optional data (profile_image, about, location, website) empty
Then we send an email with the verification id to the user, which is your basic "confirm your email" email.
When the user clicks on that link, we set the verification id field as null in the db, marking that the user has verified his email.
We could have went with a boolean or a status thing to mark if the user had verified his account or not. But we didn't see the use for now, so we saved the extra column by just setting the field as null. After all we have no use for it anymore.
We then redirect the user to the page on which he signed up using the blog_origin field.
Sign up using google
Since you don't "sign up" per se, we just sends data straight to our backed, signing in and login in with google is the same endpoint.
So I'll save the sign up process using google for later
log in
On top of what I usually return (described at the end of each authentication method), I return the user data in user json metadata:
for me that would be
{
"name": "",
"about": "Software engineer by day, Blockchain developer by night. SteemPress co-founder. @steempress witness",
"website": "https://brokencode.io",
"location": "",
"profile_image": "https://images.hive.blog/u/howo/avatar",
"type": "hive"
}
it's useful to do it that way because then it's less of pain for the front end if both regular authors and hive authors use the same format.
by the way, on hive there are two json metadata for a given user, one that is edited with the active key (json metadata) and one that is edited with the posting key, nowadays most apps use the posting one, but some people still use the active one (which is bad practice, please migrate if this is you).
So in order to support both here's a neat function:
function parseUserJsonMetadata(user) {
try {
return JSON.parse(user.posting_json_metadata).profile
} catch (e) {}
try {
return JSON.parse(user.json_metadata).profile
} catch (e) {}
return undefined
}
hive log in
check if username + type hive exists in the db, if it's not there it means it's a first login (or a relogin of an user that expired)
first login:
We just generate ids, and store everything in the sessions table
- Get the posting key and username
- Generate a user uuid
- generate a login token
- Set the type as "hive"
- store the hive username
- Encrypt the posting key and store it
- Set an expire date as now + 2 weeks
Send back the login token + user_uuid, it'll then be stored by the front end as cookies and used later on for auto auth
returning login:
fetch the user uuid and token, set the expire date as now + 2 weeks.
Send back the login token + user_uuid
It's important to keep the same token instead of generating a new one because otherwise if your user is loggin in from two different locations, he'll keep sign himself out by loggin in on another device:
- User logs on device A, gets token "AAA"
- token "AAA" is stored in cookies and user can interact with everything on his browser and the "AAA" token works
- User logs on device B, the token changes to "BBB"
- user tries to do things on device A, device A has the "AAA" token stored which gets refused, he has to login again which will create a new token "CCC" which invalidates the token on device B
This is an easy mistake to make and will make up for a bad experience for your users.
hivesigner login
pretty much the same, except we encrypt the hivesigner token instead of the private key
side note on hivesigner tokens: even though they are not the posting key. They should be treated as such, often they have almost the same priviledges, so if someone were to gain control of the hivesigner, he could benefit from the same powers as a posting key until it expires or gets revoked.
log in using email
- User sends us email + password
- We check if it matches our records of an user of type
regularand get the corresponding user_uuid - we generate a login token
- set the type as "regular"
- set an expire date as now + 2 weeks
- leave username, private_key as null
Send back the login token + user_uuid
log in using google
Most of the logic related to google itself is done on the front end, basically google opens a popup, user authenticates there, then it sends you back info on the user (email, name, profile picture etc) and a token to authenticate the user which we forward to the backend.
First we check the email to check if it exists in authors and if it has the google type (if not we tell them to login using said type)
if it didn't exist, it means we need to create a new author:
New google user:
- We generate a regular_author_uuid using a uuid v4 (https://en.wikipedia.org/wiki/Universally_unique_identifier)
- We store the email
- Set the user type as
google - Set the blog_origin as the exact page the user logged in on for statistics.
- We leave the rest as null (profile_image, about, location, website, password, display_name) empty
We leave the rest empty because the source of truth for those is google not us. You want to end up in a state where if the user changes his profile picture, his profile picture should change for your app as well.
then we proceed with the login flow, since we know it's a new user we create a new session:
- generate a login token
- set the type as "google"
- set an expire date as now + 2 weeks
- leave username, private_key as null
Send back the login token + user_uuid (which we just generated)
Returning google user:
Then we just refresh the expiration time and return the login token + user_uuid.
returning login
Via the sessions table, we can just authenticate a user using user_uuid + token and do actions based on it's type.
expiration
a simple cron job runs every 5 minutes to expire sessions if expire is in the past
displaying regular comments
We haven't implemented the fleet account logic yet because we don't process enough comments for it to matter (worst case if two people try to post at the same time, they get an error message to retry and then the comment goes through).
The logic is quite simple, when I get the request to get all the comments for a specific post, I fetch the comments (I'm skipping pagination logic, but it works the same way), then fetch all of the regular comments for that post using the root_author and root_permlink fields in the comments table then for every comment made by steempress-io (the proxy account making the regular comments on the chain), I find the corresponding db entry in comments using the permlink, then I replace the content, author etc and then the front end can do the work to display them correctly.
Keep in mind that it's important to mark them as regular comments and not hive comments so you can still reply/comment on it, so when you do actions on it you replace the author name with steempress-io because the chain is not aware of that.
more on the fleet accounts
First of all, this is a great problem to have, it means your app has a ton of engagement, hats off to you 👍
So if your front end ends up needing a fleet of accounts, here is how you should do it:
don't create a bunch of accounts manually and hope those will be enough to handle the load, do a specific program to do this:
have a fleet of accounts, record when they are in cooldown and when they are not, and have a runner check every x seconds the ratio of available accounts for commenting vs the amount of accounts that are in cooldown. If the ratio is too low for some time, (let's say less than 20% for 5 minutes) then it means you need to create more accounts and add them to the fleet. Keep a "mothership" account with a bunch of hive power and claimed accounts, to be able to create new accounts and delegate enough hive power to them so they can comment at least once every 3 seconds.
That way the fleet can auto expand and you won't end up in a situation where your app is just stuck just because you are asleep and your app blew up overnight.
Conclusion
That's it ! I didn't describe in details more than the authentication and sign up because I think that part is the most important and also the implementation starts to be use case specific so I just put a high level description on how we solved our specific problems, and you can probably guess using the database schema. Also I don't claim I have the best solution, this was bundled together during multiple dev iterations. This is really a "this is how we do it, use this to think of a better solution" that I wished I had when I started to architecture it. As there are a lot of things you can overlook when designing something like this.
If you are making a new app, please factor the fact that you will most likely need to support non-hive accounts in the future, so keep it in mind when designing everything. I didn't and had to redesign a big chunk of the code for this feature.
If multiple apps use some kind of regular author thing, I think it is becoming more and more important to have it in a common database (which is decentralized) like hivemind.
This is another subject entierely and I don't have time to talk about it, but at some point I think it'll be important to store the user info in the json_metadata of the comments like display_name, avatar, app, etc so the various front ends can display them (and perhaps with a little logo to know on which front end the regular commenter came from)
Sorry if this is quite long and for repeating myself a bit, sometimes it makes sense to explain something twice.
If you liked this writeup, please consider voting on our witness:
By the way, I know steempress may seem like an abandoned project between the lack of updates and me (the main dev) working on the core development team. But it really isn't, @fredrikaa and I are hard at work almost every single day. We are in the process of making an enormous update on top of the rebranding (don't want to have steem in the name for obvious reasons) and we want it to be perfect which is why we are taking so long to release it.
It seems a bit complex but I like the idea of the whole user flow. The only thing that I am concerned about is the scalability of this whole flow especially the proxy related stuff.
Personally, I think these complex stuff can be cut off if we are able to revamp the account creation flow. Or Maybe, DAO fund can be used for account creation. A lot of brainstorming is yet to be done around that but I can see the exciting future.
Along with that, Me as a part of the Dapplr team is so working on some sort of a POC to simplify the login and signup process. We will see about its outcomes.
Thank you for this wonderful post 🙌
Posted using Dapplr
Thanks !
I'm all in for a simpler solution, after all I am always looking for ways to improve it as well.
In the future, it will be a bit simpler, because with RC delegations and claimed accounts, assuming you got enough claimed accounts in advance, you could just allow anyone to create an account if they validate their email.
This can't really be done now because we are forced to delegate hive power to new accounts otherwise they would be almost unusable, and with even a little hive power comes potential abuse (We delegate 15 hive power per new account, an abuser could try to get 1000, now that starts to be significant enough to get some cool self votes)
This is a fantastic piece of work. The user journey seems pretty seamless. Using the googls or FB auth is a pretty sweet addition as many people use one or the other of these services. It will be a great marketing tool for hive as users become more exposed to the platform as they traverse "traditional" web and social channels.
I really like the idea of a "stored reward" section. It would be really neat to interact with the non hive user to let them know that they have X hive waiting for them on signup. I don't think I saw anything in the post about it.
All in all nice work!
I didn't expand on it, but it's definitely part of the flow, we send them emails remining them about it etc.
Perfect. Its that little worm so to say.
Thanks for breaking it down like this, @howo. Technical stuff aside, I know for a fact that having the two way comment thingy is attracting interest from my WordPress blogging friends, and there have been a few guest comments made on my blog. Oh, and have I said lately how wicked excited I am for the upcoming rebrand? 🤣
Haha thank you for your continuous support, it's always heartwarming :)
This is absolutely great stuff and with so huge and untapped potential. I am looking at tapping into that potential myself. :)
thanks !
It would be very interesting to see a video of everything Steempress can achieve. From setting up the plugging for our own website to someone using such website logging from a google account or the other way of signing up. it would also be interesting to see someone claiming their Hiveaccount.
Videos of different lengths could be produced to showcase all the features or only some of the features as a teaser.
Truly groundbreaking.
Thanks for your input. We've been looking to do something like that for quite some time, but haven't had the time to really dig into it. We might try to get someone from the community to do it. But it's pointless to do them now when the whole project will be rebranded very soon™️
As you know, I am an unashamed fan. And you have my witness!
Thanks :)
I love it and may create a Wordpress site because of this.
Since I'm not a programmer, did you make this painless for laypersons to add to their site?
Yep, it's just a regular plugin.
Beautiful! Thank you😀
Happy to be reading this and
Is Steempress now working and posting to hive ? Someone just told me few days back that its working for him and posting to hive !!!
Yes, we made the switch from steem to hive a few days after the chain launched, but we are still working on rebranding.
What is the difference between Steem and Hive? Sorry I'm new here..
Steem was the original version of Hive.
It was taken over.
People rebelled and started Hive instead.
Long amazing story!
Google steem hostile takeover.
Posted using Dapplr
Thank you for the information, I am curious about the story. I will search on google ;)
Looking forward to the update. Nice writeup
great work. are you planning on rebranding it to hive?
yes