Note: this witness update will be technical.
In order to effectively maintain a server you need a monitoring system to constantly check its health by looking at various stats such as resource consumption (disk, memory, CPU, network, etc...) and system logs.
Steemd, the blockchain software, is outputting its logs to the standard output so it is required to redirect it to a file. In my setup, it is redirecting it to /home/quochuy/steemd.log and I'm rotating and compressing that log file on a daily basis, keeping them only for the last 7 days using logrotated.
This is how a regular steemd.log looks like during normal operation:
Sumo Logic
In order to easily check and analyse the log output of steemd and setup some automated alerts, I've decided to setup a data collector that sends it to Sumo Logic, a log analytics service I'm already using for other personal projects and also at work, it has a free tier which I think should be enough for processing witness logs at my current rank. The collector is a little script that will stream the logs to Sumo Logic which in turn will parse and store it in their database. I can then access their web interface to do queries, analysis and other alerts.
After an initial data ingestion here is what the logs look like in Sumo Logic:
I can then apply a logreduce filter to it and Sumo Logic will try to categorise each of the log lines and group them by similarities and display the number of time they have occurred. This is very useful to reduce the noise and easily find an issue. In the screenshot below, you can see that Sumo Logic has managed to group block production logs by witness and removed slight variations from each line (number of transaction in a block, block offset etc...):
Now the interesting part is ability to spot errors from that logreduced result. From the screenshot below, noticed the unique types of errors and how many times they occurred (first number in blue).
Those are non-critical errors and exceptions (not sure why they are exception btw). But let suppose one of them is a critical error and I've noticed it freezes the app or makes me miss a block, then I could easily set an alert for when the same type of error is happening again.
I can also run a query that checks for popular keywords that are usually used in case of an error as seen below:
On a more fun side, I can also check how many times my witness has generated blocks too. The current data is less than 24 hours old so I have only two block shown in Sumo Logic:
Of course, I can also live tail the log file straight from Sumo Logic without the need of doing an SSH connection to my server. And I can also setup some field extraction rules, I can extract fields and their values from the log messages and then use that in a search or dashboard. Here for example, I'm extracting the number of transactions, the block number and the name of the witness who produced the block:
These extracted fields will allow me to generate this graph of transactions signed by each witness:
After creating multiple searches and graphs, I've came up with this simple dashboard that gives me visibility on my own number of generated blocks in the last 7 days, the number of transactions per witness in the last 24 hours and the number of transactions per block in the last 15 minutes:
I can obviously add more panels to this widget for monitoring errors and warnings.
Log rotation
For those interested in how I rotate my log file and compress them to save space like here:
All I do is create a file in /etc/logrotate.d/ called steemd with the following content:
/home/quochuy/steemd.log {
rotate 7
daily
compress
missingok
}
Replace /home/quochuy/steemd.log with the path to your own log file and also make sure you have logrotate installed and running. Also note, this won't work with Steem-in-a-box (aka Steem Docker by @someguy123), it's only for if you are running steemd natively. Update: They do have a collector for Docker so you can monitor the logs of a container, this could be configured to use with Steem Docker: https://github.com/SumoLogic/sumologic-collector-docker
Related content
- Witness Earnings Report - 2018-10-08
- Witness Earnings Report - 2018-10-01
- Witness Earnings Report - 2018-09-24
- Witness Update 2018-09-24 - Ready for HF20 (again)
Summary of some of my other contributions
| Project | Description | Link |
|---|---|---|
| Steemed Phish | Chrome browser extension to make external links more visible and alert you on Phishing/Scamming links | Steemed Phish |
| Dtube Community Support | This project by @evildido allows users to run their own IPFS node and pin videos in order to cache them for longer. Dtube has the tendency of deleting non-popular video after few days only. I helped Dockerizing the project for an easier deployment on almost any platform. I've also contributed by running two nodes, one in Australia and one in Canada to cache #teamaustralia and #teamnz videos. | Dockerizing Dtube Community Support |
| DiscoSteem Bot | This is a Discord/Steem bot project by @planetenamek. When I started Steemulant Bot for the Vietnamese community, I didn't know the existence of DiscoSteem Bot and wanted something written in NodeJS. I've ended up continuing my own project as it contains code and logic for personal use and for the Vietnamese community. But I've decided to share some useful features that can benefit the users of DiscoSteem Bot and will continue to add more functionalities as I go. | Dockerizing DiscoSteem Bot |
| A Dollar A Day | This is a project run by @pennsif and will donate 15 SBD every day to a charity | "A dollar a day" fundraiser - Karaoke Singing: Le chanteur malheureux |
| Team Vietnam | I started a new Vietnamese community on Discord and developed @teamvn, a community bot to help grow the Vietnamese community and help them create higher quality content. | |
| POCKET-JS | This is my Javascript (NodeJS) port of the POCKET confirmer by @biophil. POCKET is an experimental crypto token on top of STEEM. | POCKET-JS Confirmer |
| Skippy Bot | I've written this Discord bot for @teamaustralia, it has three main features: a level bot (like Tatsumaki but with Australian animal images), a chatter bot and a footer banner generator for both @teamaustralia, @teamnz and @steemph | Witness Update 2018-07-30 - Team Australia's Skippy Bot |
Credits
- The image at the top has been generated with the Canva app using my own photo.
This is really cool man, I hadn't heard of sumo logic before, might look into setting it up since there's a free tier
Posted using Partiko Android
The free tier has limited functionalities and reduced amount of data you can send them per day but should be enough for having fun still
Posted using Partiko iOS
I was thinking about trying datadog for handling the logs. This looks interesting too.
I've used data dog before (for their free trial tshirt :D) it was pretty useful but I didn't get into the logs part of it, only the webserver monitoring.
lol ..
You also have https://www.humio.com/ which also offers a free tier with 2GB of data per day and 7 days data retention. I was at their presentation at Yow! Conferences and the product looks good too. They also have a quite neat data compression strategy that chunks the data small enough that it can fit in L2 CPU cache in order to do decompression on the fly when searching a compressed DB.
this tool is awesome, I might give it a try soon!
How does your data collector script work? Might be interesting to code something in nodejs/python/whatever later on to handle the notifications and other fancy things!
The data collector is not mine, it's provided by Sumo Logic. When you use their setup wizard it asks you your OS etc... and gives you a command line to run to install the collector. But I believe you can also develop your own collector. Maybe check their GitHub: https://github.com/SumoLogic
Congratulations! Your post has been selected as a daily Steemit truffle! It is listed on rank 12 of all contributions awarded today. You can find the TOP DAILY TRUFFLE PICKS HERE.
I upvoted your contribution because to my mind your post is at least 11 SBD worth and should receive 184 votes. It's now up to the lovely Steemit community to make this come true.
I am
TrufflePig, an Artificial Intelligence Bot that helps minnows and content curators using Machine Learning. If you are curious how I select content, you can find an explanation here!Have a nice day and sincerely yours,
TrufflePigHi, @quochuy!
You just got a 0.6% upvote from SteemPlus!
To get higher upvotes, earn more SteemPlus Points (SPP). On your Steemit wallet, check your SPP balance and click on "How to earn SPP?" to find out all the ways to earn.
If you're not using SteemPlus yet, please check our last posts in here to see the many ways in which SteemPlus can improve your Steem experience on Steemit and Busy.
Hi @quochuy!
Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your UA account score is currently 5.673 which ranks you at #452 across all Steem accounts.
Your rank has improved 7 places in the last three days (old rank 459).
In our last Algorithmic Curation Round, consisting of 471 contributions, your post is ranked at #351.
Evaluation of your UA score:
Feel free to join our @steem-ua Discord server