Python In The Shell: The STEEMIT Ecosystem – Post #103
BLOCKCHAIN AND DATABASE MANIPULATIONS USING PYTHON
In continuation as per the previous post here:
https://steemit.com/blockchain/@lightingmacsteem/2ke8de-i-t-spices-the-linux-way
We will finish our discussion on this topic with these last few lines:
1 ###RELAY THE LOGS TO THE CENTRAL LOG SERVER
2 ee = ('cat ' + logfile + ' >> /MARIO/BACKUPS/LOGS/insert-json-into-mysql_resume.logs')
3 os.system(ee)
4
5 ###CLOSE MARIADB CONNECTION AFTER
6 db.close()
7
8 ###CLEAN UP FOLDERS
9 shutil.rmtree(tempdir, ignore_errors=True)
Finishing Off And Cleaning Up
Lines 1 to 3 is making sure the logs are secured at a folder using once again the linux shell as called from python using the os module. A screenshot log sample is below:
Lines 4 to 5 is making sure the connection to the database will be terminated. Take note that we only connected once to the database at the very start of its manipulation.
After the 300 thousand lines of JSON blockchain data is processed, only then that we close the connection thru line 6.
Imagine connecting only once, then inserting lots of data, after which closing the connection once. Said behaviour of python using the pymysql module is one of the reasons that it is very fast as far as database manipulation of data is concerned.
Lines 7 to 9 is making sure that whatever temp folder is created at the start, it has to be deleted before the script exits, telling it to ignore the errors if there are.
That’s it, the python way to manipulate the steemit JSON-formatted blockchain data so that it can be saved into a database, MariaDB in our case.
All blogs up to this point regarding this topic will be good notes for the future generations to follow and be encouraged upon as they start embarking into the world of IT and Computers.
“Sky Is The Limit To One’s Quest…….”