How to use the logger class to filter out unwanted messages from beem

in #python6 years ago (edited)

When you run a python script which uses beem, you might have seen a lot of Retry RPC Call on node: https://steemd.minnowsupportproject.org (1/5)messages:
`
image.png

These messages indicate that the node was currently not available and the same node is retried again. Normally, it works then. The messages are only a problem when a lot of api calls are broadcasted, e.g.g when reading the complete account history of steembasicincome.

You can use the logger class to get rid of these messages by increasing the logger level just for the beemapi.node module.

Let's start with a simply script that uses the logger class but without any configuration:

#!/usr/bin/python
from beem.account import Account
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logging.basicConfig()


if __name__ == '__main__':
    account = Account("steembasicincome")
    n = 0
    logger.info("starting....")
    for op in account.history():
        n += 1
    
    logger.info("%s has %d history operations" % (account["name"], n))

After storing the script as use_logger1.py and running it by:

python use_logger1.py

I'm getting really a lot of useless logging messages:

image.png

Exclude logging messages from beemapi.node

It is possible to configure the logging class and increase the level of the beemapi.node module to the ERROR level:

#!/usr/bin/python
from beem.account import Account
import logging
import logging.config

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)
logging.config.dictConfig({
    'version': 1,
    'disable_existing_loggers': False,  # this fixes the problem
    'formatters': {
        'standard': {
            "format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
            "datefmt": "%Y-%m-%d %H:%M"
        },
    },
    'handlers': {
        'default': {
            'level':'INFO',
            "formatter": "standard",
            'class':'logging.StreamHandler',
        },
    },
    'loggers': {
        'beemapi.node': {
            "level": "ERROR",
            "handlers": ["default"],
            "propagate": False
        },
        '': {
            'handlers': ['default'],
            'level': 'INFO',
            'propagate': True
        }
    }
})


if __name__ == '__main__':
    account = Account("steembasicincome")
    n = 0
    logger.info("starting....")
    for op in account.history():
        n += 1
    
    logger.info("%s has %d history operations" % (account["name"], n))
    

After storing it as use_logger2.py and running it, the logging output contains only information that are useful:

image.png

Configuring the logging class in an external file

It is also possible to store the configuration in an external logging.json file:

{
    "version": 1,
    "disable_existing_loggers": false,
    "formatters": {
        "simple": {
            "format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
            "datefmt": "%Y-%m-%d %H:%M"
        }
    },

    "handlers": {
        "default": {
            "level":"INFO",
            "formatter": "simple",
            "class":"logging.StreamHandler"
        }
    },
    "loggers": {
        "beemapi.node": {
            "level": "ERROR",
            "handlers": ["default"],
            "propagate": false
        },
        "": {
            "handlers": ["default"],
            "level": "INFO"
        }
    }
}

and reading it in the python script:

#!/usr/bin/python
from beem.account import Account
import logging
import logging.config
import os
import json

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

def setup_logging(
    default_path='logging.json',
    default_level=logging.INFO
):
    """Setup logging configuration

    """
    path = default_path
    if os.path.exists(path):
        with open(path, 'rt') as f:
            config = json.load(f)
        logging.config.dictConfig(config)
    else:
        logging.basicConfig(level=default_level)


if __name__ == '__main__':
    setup_logging("./logging.json")
    account = Account("steembasicincome")
    n = 0
    logger.info("starting....")
    for op in account.history():
        n += 1
    
    logger.info("%s has %d history operations" % (account["name"], n))

This allows it to use the same configuration in multiple projects. After storing it under use_logging3.py in the same directory as the logging.json file, it can be started by python use_logging3.py and produces the following output:
image.png

Formation of the logging output

The logging output format was set by

            "format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s",
            "datefmt": "%Y-%m-%d %H:%M"

More variables can be found in the logger documentation. The different options for datefmt can be found here.

Sort:  

Hi @holger80!

Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your UA account score is currently 7.402 which ranks you at #59 across all Steem accounts.
Your rank has improved 1 places in the last three days (old rank 60).

In our last Algorithmic Curation Round, consisting of 234 contributions, your post is ranked at #1. Congratulations!

Evaluation of your UA score:
  • Your follower network is great!
  • The readers appreciate your great work!
  • Great user engagement! You rock!

Feel free to join our @steem-ua Discord server

This post has been just added as new item to timeline of beem on Steem Projects.

If you want to be notified about new updates from this project, register on Steem Projects and add beem to your favorite projects.

Hi, @holger80!

You just got a 2.79% upvote from SteemPlus!
To get higher upvotes, earn more SteemPlus Points (SPP). On your Steemit wallet, check your SPP balance and click on "How to earn SPP?" to find out all the ways to earn.
If you're not using SteemPlus yet, please check our last posts in here to see the many ways in which SteemPlus can improve your Steem experience on Steemit and Busy.

Thank you so much for participating in the Partiko Delegation Plan Round 1! We really appreciate your support! As part of the delegation benefits, we just gave you a 3.00% upvote! Together, let’s change the world!