Steemit Followers - Data display and account details - Part 2

[shaunmza]

In the first part of this series ( https://steemit.com/steemdev/@shaunmza/steemit-followers-data-extraction-part-1 ) I laid out where and how I got the data for Steem Followers - https://steem.makerwannabe.com

I promised a follow up and here it is!

Getting the data out of the local database

As I mentioned before, I fetch the data from a Microsoft SQL database, then store it in a MySQL database locally. This allows me to save the data in a structure that lends itself to the kinds of queries I expect to run, as well as saving traffic and load on the STEEMSQL http://www.steemsql.com/ service kindly offered by @arcange

The page you see hat displays the data is a very simple PHP script. When the script runs it looks for parameters passed in, like the user name to search for, then runs a few queries to fetch the data from the database.

I am not going to put the whole query here, but if you are interested, drop me a comment, or gt hold of me on https://steemit.chat/direct/shaunmza.

Suffice to say, using the query I fetch the follow count, unfollow count and mute count for the user name, then group that by the date while summing all of the follows, unfollows and mutes separately.

This way I have 31 May 2017 - 1 follow - 0 unfollows - 0 mutes for every day that something changed on your follower data.

I run another query, which is similar, but that does not group the data, so that I can show each and every interaction in the list below the graph.

Data to graph

For the graph I use an open source library developed by https://keen.io/ which is a neat wrapper for Google charts.

This is one part that I struggled with immensely and am still not happy with the outcome.

I use the data that I retrieved in the PHP script, to build up a javascript object, which is then used to populate the graph.

I still want to add a trend line to the graph, so am in for a battle in future.

Your photo!

If you have added a profile picture, or any other information to your profile, then that gets displayed at the top of the page above the graph now too.

I implemented this a while back, but it was less than reliable. This has been tended to today though and I am hoping I have ironed out the bugs.

That information is fetched directly from the blockchain, using my library for the Steemd service, which is written in Go https://github.com/shaunmza/steemgo

I wrote an application that utilises this library. It runs on the same server, and is queried by the PHP script using curl.

Because I am not sure of the lag times, I added in a timeout, so that if the Go component takes too long to respond, for whatever reason, the PHP script carries on as normal.

Here is a code snippet that highlights that functionality;

<?php

function getSteemDetails($primary, $additional)
{
        //"{\"primary\": \"shaunmza\", \"additional\" : [\"gtg\"]}"
        $postString = json_encode(array('primary' => $primary, 'additional' => $additional));

        $ch = curl_init();
        $headers["Content-Length"] = strlen($postString);
        $headers["Content-Type"] = 'application/json';
        $headers["User-Agent"] = "Curl/1.0";

        curl_setopt($ch, CURLOPT_URL, '127.0.0.1');
        curl_setopt($ch, CURLOPT_PORT, 9876);
        curl_setopt($ch, CURLOPT_HEADER, false);
        curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
        curl_setopt($ch, CURLOPT_POSTFIELDS, $postString);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($ch, CURLOPT_TIMEOUT,10);
        curl_setopt($ch, CURLOPT_POST, 1);
        $response = curl_exec($ch);
        curl_close($ch);

        $json = json_decode($response);

        return $json;

}

Not a PHP developer? Let me know if you want an explanation, I would be happy to teach a bit of PHP 😊

The Go component has a LRU (least recently used) caching mechanism, which keeps a number of profiles details in memory, so that I do not call off to the steemd node each and every time a page is accessed.

Conclusion

So to bring you Steem Followers, I grab data from an external Microsoft SQL server, transform it and store it in a MySQL database on my server.
Then I serve it up using a mixture of PHP and Go.

All of this is made possible by services offered free of charge by others, as well as open source software which is available for use, again free of charge.

I am planning to extend the functionality, so if you have any suggestions please do let me know.

Also, I see the usage is rising, so now may be a good time to choose a real name and get a domain for it, anyone have any ideas for a name?!?