Python In The Shell: The STEEMIT Ecosystem – Post #102
BLOCKCHAIN AND DATABASE MANIPULATIONS USING PYTHON
In continuation as per the previous post here:
https://steemit.com/blockchain/@lightingmacsteem/2hg6qa-i-t-spices-the-linux-way
The below lines are database insertion routines the python way:
1 print('\nI am processing block_id: ' + block_id)
2 ###PRINT COUNTER
3 print('COUNTER ' + str(counter))
4 ###EXECUTE FIRST LOOP SQL HERE
5 sql1 = ("INSERT INTO `data01`(counter, block_id, extensions, previous, signing_key, timestamp, transaction_ids, transaction_merkle_root, witness, witness_signature, operation_type, transactions, blockdata)VALUES(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)")
6 data1 = (int(counter), block_id, str(extensions), previous, signing_key, timestamp, str(transaction_ids), transaction_merkle_root, witness, witness_signature, operation_type, transactions, blockdata)
7 # Execute the SQL command
8 cursor.execute(sql1, (data1))
9 db.commit()
10 counter = int(counter) + 1
Blockchain Data As Inserted Into The Database
Line 1 to 3 is to print the block_id and the counter variable on the monitor screen. This will give us a lot of idea as to how and what the script is doing. A hint: if the displaying of messages here is slow, it gives us a hint that our script needs more polishing, but if it is fast, then surely we have nailed it. So this line serves a very important other purposes, not just displaying messages.
Line 5 is the very sql command that python needs to execute, we can observe it starts with an INSERT INTO, so self-explanatory what we want it to do, of course into means INTO THE DATABASE.
More on line 5, the table name is data01 and inside the parenthesis are the column names as defined on the database table, such as counter, block_id, extensions, etc etc. All these are defined as the sql1 variable.
Still in line 5, the VALUES with the multiple %s character simply defines the number of column data to be inserted, with each %s to be replaced by whatever data as given by the data1 variable in line 6. We can observe that the number of %s characters is the same as the number of data1 variable entries. This simply means a 1 to 1 replacement.
Lines 7 and 8 is the execution line, and python uses all the variable definitions in doing so. To add to the descriptions, this python command will have errors if the number of %s does not correspond to the number of data inside the data1 variable.
Line 9 is just to tell python that we meant business here, so go ahead, commit, insert these into the database table and never look back.
Line 10 adds one count to the counter variable to be able to be displayed to the screen, giving us an idea so that we won’t be fidget in waiting if something is really running as we wanted.
These insertions happen so fast even with all the python processes and preparations I had already described in previous blogs. I can not keep up with the block_id numbers as displayed, all I can see are numbers count as displayed by the counter variable, it increments one count at a time (so fast) by batches of 300 thousands.
Screenshot of the python script in action below:
Hmmmm… I can imagine 9x servers with mirror-striped SSD drives on ZFS, each server having a copy of the blockchain file but is processing a different set of blockchain lines. Multi-Gigabit network between each server and the clients, unexposed to the public.
Speed, stability, high-availability, unlimited storage. All to its maximum. Can you imagine what I just described?
Don’t worry, I know one day you will, just be sure you knew python when it comes.
“Hardware Is Nothing Without Software, And Vice Versa…… Both Are Useless Without You, The Mind Behind.”