The use of Custom JSON field on the STEEM Blockchain is becoming more and more popular. From discussion I have read, this field can do almost anything from creating Tokens to holding game information.
JSON data is a way of representing objects or arrays. It is easy to read, and it is easy to parse. Many API calls will return JSON format and many web apps use JSON to easily move information around the internet.
Its is therefore no surprise that developers are now using the JSON filed on steem to move data between an app and the blockchain.
Initially the Custom JSON field was used for holding follow information, where one account on the blockchain follows another. However, over time, this has expanded to holding information from multiple sources for multiple reasons.
With that in mind I wanted to take a quick look at the Txcustoms table in the STEEMSQL database to just get an idea of what and who is using this JSON field. This type of analysis is commonly called exploratory analysis.
Repository
https://github.com/steemit/steem
Aim of Exploratory Analysis
- Get a feel for the Txcustoms table in STEEMSQL.
- Establish if there was growth in the use of this field from 2018 to 2019
- Establish what app is using this most
- Get an idea of the type of data held
Analysis
The table below shows the custom JSONs for the first week of January 2018. None of the Custom JSONs required additional posting or other authorities. As we can see, most of the transitions using this field were Follows, however there were a few others too.
Pacman-live seems to be one of the first games to host data on the blockchain. Looking at this further the data sent to the blockchain was the username, the score, the game level and a timestamp.
Chainbb seem to have been using the JSON field to store details of Forum objects, such as forum configurations and forum posts.
Distribution is an interesting one and seems to be holding information about distributions from randowhale
And we can also see some testing of STEEMMonsters under TEST_sm_pre_sale_packs
Jumping on now to the first week of January 2019 and things are looking rather different. 980162 custom JSON fields were used, down 58% on the same time last year. However, this is not bad news because in 2018 99.8% of these related to follows and unfollows. With the downturn in activity on the blockchain, for the same period in 2019 follows and unfollows make up only 9.8% of the custom JSON transactions. The was a reduction of 95% in the actual number of follows and unfollows.
The table above shows the number of transitions of each time with custom JSONS. I have split this list in two. On the right we have all the STEEMMonsters custom JSONS. This makes up 88% of all custom JSONs. On the left we have others such as Follows, DiceHash and Dlux.
The table below shows a sample of the data contained within some of the JSONs.
Also changed considerably since 2018 is the user of authority to post and this is also recorded in the blockchain and the SQL database. Looking at this for STEEMMonsters in the table below, by doing a count of the required_posting_auths we can see that for this period there were 366,192 find match and by carrying out a distinct count of the same field, we can establish this was by 2379 different accounts.
Conclusion
Although there was a reduction in the number of custom JSONs used from Jan 18 to Jan 19, this is a reflection on the downturn in social activity on the block and not development activity.
The social activity, follows and unfollows is down 95%. However, in 2018 non-social activity recorded in JSON was only 0.02%, in 2018 this was 90.2%. This is an indication of development on the block and a sign in the shift of the platform from being a content/social platform to something more.
By looking at the data contained within the JSONs my eyes have been opened on how simple it can be to really make ‘ANY’ app a DApp on the STEEM Blockchain and I hope it gives you that sense too.
Obviously STEEMMonsters are well out there in terms of usage compared to other apps using JSONs but many apps are only being to test this feature. Some of this comes from the shift away from Steemit.Inc as the only developers on the block. STEEM is coded in C++ which is a skill many do not have. However, there is no shortage of App developers with JSON knowledge.
Now that I have a better understanding of how this table in the SQL database is kept and what data it contains, it opens up a new world of analysis.
So, who will be first to have a D+ site ready to replace G+ communities using our new friend JSON?
Data and Query
The data for this post was taken from the STEEMSQL database held and managed by @arcange.
The query used for 2019 was
select *from Txcustomswhere CONVERT(DATE,timestamp) BETWEEN '2019-01-01' AND '2019-01-07'and for 2018 was
select *from Txcustomswhere CONVERT(DATE,timestamp) BETWEEN '2018-01-01' AND '2018-01-07'As always, I used PowerBI to
- Connect to the data
- Transform the data
- Carry out calculations
- Visualise the data
If you want to learn more about using PowerBI or Excel do follow @theexcelclub
.
Thank you for your review, @crokkon! Keep up the good work!
Hello Paula,
Sorry to say it, but your analysis is (partially) invalid :/
There was a bug in SteemSQL Database Injector, which was not processing
required_authsandrequired_posting_auths.This problem was fixed in December 2018 and only new custom_json operations issued after the fix were recorded correctly.
Unfortunately, SteemSQL need to reprocess all blocks from the beginning of the blockchain, something I plan to do this month.
(upvoted for visibility)
Okay, thanks for letting us know. Good luck reprocessing, that will take ages.
lol love it, and thanks so much for adding extra value to this post in the comments, your a rock star
Cool.
Is there a record of who adds the custom JSON?
Any authorization to add a field that already exists and is being pulled for data?
Can anyone add anything, like any account add info for any JSON data and imitate an app, such as adding bogus data?
Could I just add followers to people who didn't really follow?
Can I add SM data just by adding a custom JSON object?
Thanks!
Yes and No. You can add any custom_json content, including similar objects as SM, but the author of the transaction won't match.
Check my comment to @paulag
Thanks. So if someone makes an app, they can just look for custom json where their app (or another) is the author of the data and only use that?
They need to look at the author of the custom_json, not "the app", unless you want to monitor all transactions written as "something you understand" (i.e. like how steemit.com works with "follow" custom_json).
Yes, the appname as the author, like busy, or partiko, etc. To get all real follows then, you need to search for all app accounts that add to
follow? If you don't look for busy or partiko in thefollow, you won't get all the followers, right?Here's how I'd explain it: anyone can create a custom_json transaction with an id. This could be something like
follow,sm_team_reveal, etc.When someone creates a custom_json
followtransaction with datakrnelthey express that they want to follow krnel. Then every Steem node sees that they expressed that they want to follow krnel, so they update their internal books that they followed krnel.Same with
sm_claim_reward. If someone broadcasts ansm_claim_reward, then everyone knows that they want to claim their reward, and it is considered that they have claimed their reward.OK, so it's per user account. Nothing to do with the app account, for follows. Are there fields that are validated through an app account? As a way to prevent anyone from just adding any data they want.
Say you have an app and create a field for data, how can you prevent just anyone from adding data to the field simply by signing/broadcasting it with their own account? Anyone could make fraudulent data claims on the field in that case. Is this a limitation of the custom JSON field, where it cant be used like any database?
Thanks!
Great questions
"Is there a record of who adds the custom JSON?"
In that table, not really, but there is posting auth recorded for apps that needs it. But it doesnt say which app it came from. Im sure if I mix this table with another one I could get the results. But I havent looked.
"Any authorization to add a field that already exists and is being pulled for data?"
Not sure I understand this question
Can anyone add anything, like any account add info for any JSON data and imitate an app, such as adding bogus data?
Hmm great question. I'm not a dev so I might not answer that correctly. But I am sure it requires some sort of conscious and authority, if not then it would be possible.
"Could I just add followers to people who didn't really follow? "
not unless you have the posting authority
Can I add SM data just by adding a custom JSON object? again you would need posting authority
Yes, that's why there are lot of invalid "follow" json in the blockchain and apps should double check both the author and the content
So when a new field is created, it is associated to the original posting authority that created it, and no other account can add to that field? I thought anyone can add a
followto the custom JSON. Similarly, anyone could add any field, and fabricate data. One needs to always check the author of the data added to a field I would think.As a non technical person I see this as a demonstration of the flexibility the blockchain has to use available resources to develop more on the ecosystem. All this activity while have the capacity to grow without impacting the spped or cost to transact! Seems like a powerful blockchain to me!
Posted using Partiko iOS
I'm not so technical myself. Most of the stuff I learned about the block I did so by exploring the data, I cant read code :-) so I too see this as a demonstration of the flexibility
Hi @paulag, happy new year and wish you and your family all the best, sorry it is too late to say it and celebrate it.
hay, how are you. Happy new year to you and your loved ones too xx
I am good, but not have more time online during this farming season, I am working in the farming land, planting veggies now. Thank you @paulag
really, thats rather interesting @wehmoen, thanks for sharing :-)
Hi @paulag!
Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your UA account score is currently 7.014 which ranks you at #90 across all Steem accounts.
Your rank has not changed in the last three days.
In our last Algorithmic Curation Round, consisting of 233 contributions, your post is ranked at #2. Congratulations!
Evaluation of your UA score:
Feel free to join our @steem-ua Discord server
This post has been included in the latest edition of SOS Daily News - a digest of all you need to know about the State of Steem.
Editor of the The State of Steem SoS Daily News.
Promoter of The State of Steem SoS Weekly Forums.
Editor of the weekly listing of steem radio shows, podcasts & social broadcasts.
Founder of the A Dollar A Day charitable giving project.
Hey, @paulag!
Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!
Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!