Getting docker logs to Grafana via Loki

in #development5 months ago (edited)

Grafana is a a nifty tool and create dashboards based on various data sources. One of them is Loki which aggregates logs via various methods and Grafana can be used to display them.

image.png
[Back-end logs from a NextJS appkication with PostgreSQL as db]
--
Its handy to get development logs and if the application that you are working is open source and sharing the logs from local is allowed, then, using the Grafana Labs free tier is a great use case.

Sending logs from Docker

Many a times the application may not be supporting instrumentation for all use cases. For example NextJS seems to supporting Traces and not logs quite well at this point. One easy trick is send the logs via promtail.

Lets take the use case of the local application under development running in a dockerized manner. In such cases, we can use the following trick to send the logs to Loki.

In the docker-compose.yml find the services section if you have add labels


services:
  backend:
    container_name: backend
    image: backend
    restart: always
    build:
      context: ./backend
    depends_on:
      db:
        condition: service_healthy
    env_file: ./backend/.env
    volumes:
      - ./backend/migrations:/hasura-migrations
      - ./backend/metadata:/hasura-metadata
    ports:
      - "8080:8080"
    labels:
      # Add the key and service name labels
      service.name: bebe
      logging: "promtail"
      logging_jobname: "containerlogs"

Once you are setup, then start a promtail container as follows:

docker run -d --name promtail     --restart always     -v /var/run/docker.sock:/var/run/docker.sock     -v /var/lib/docker/containers:/var/lib/docker/containers:ro     grafana/promtail:latest     -config.file=promtail/config.yaml

The promtail config file can be:

# bash
root@adcf64feff04:/# vim /etc/promtail/config.yml server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: https://GrafanaLabsUserID:<Grafa-Loki-key-in-base64-format>@logs-prod-018.grafana.net/loki/api/v1/push

scrape_configs:
  - job_name: flog_scrape
    docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
filters:
- name: label
values: ["logging=promtail"]
    relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/(.*)'
target_label: 'container'
    pipeline_stages:
- cri: {}
- multiline:
firstline: ^\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2},\d{3}
max_wait_time: 3s
 # https://grafana.com/docs/loki/latest/clients/promtail/stages/json/
- json:
expressions:
level: level

How to find <Grafa-Loki-key-in-base64-format> ? : https://grafana.com/blog/2021/04/13/how-to-send-traces-to-grafana-clouds-tempo-service-with-opentelemetry-collector/

$ echo -n "<your user id>:<your api key>" | base64

This will make the listener to start looking for the logs & send to Grafana cloud.

Sort:  

This post has been manually curated by @steemflow from Indiaunited community. Join us on our Discord Server.

Do you know that you can earn a passive income by delegating to @indiaunited. We share more than 100 % of the curation rewards with the delegators in the form of IUC tokens. HP delegators and IUC token holders also get upto 20% additional vote weight.

Here are some handy links for delegations: 100HP, 250HP, 500HP, 1000HP.

image.png

100% of the rewards from this comment goes to the curator for their manual curation efforts. Please encourage the curator @steemflow by upvoting this comment and support the community by voting the posts made by @indiaunited..

This post received an extra 20.00% vote for delegating HP / holding IUC tokens.

Congratulations @bobinson! You have completed the following achievement on the Hive blockchain And have been rewarded with New badge(s)

You distributed more than 71000 upvotes.
Your next target is to reach 72000 upvotes.

You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP

Check out our last posts:

Be ready for the October edition of the Hive Power Up Month!
Hive Power Up Day - October 1st 2023

Congratulations @bobinson! You received a personal badge!

Happy Hive Birthday! You are on the Hive blockchain for 6 years!

You can view your badges on your board and compare yourself to others in the Ranking

Check out our last posts:

Our Hive Power Delegations to the September PUM Winners
Feedback from the October Hive Power Up Day
Hive Power Up Month Challenge - September 2023 Winners List