Monitoring notes with Grafana, Loki and Promtail - Configuration Examples

In the first part, we talked about monitoring certain services; the article got quite long, and adding the example configurations would have made it much longer, so in this second part we leave several configuration examples for each service, with a short description.

Before diving into each service, it is worth recalling the general flow. Promtail reads the log files and the systemd journal, assigns labels and sends everything to Loki, which stores and indexes them. Grafana queries Loki with LogQL to build dashboards. The examples in this part focus on two things:

• What each service must emit so that its logs are useful (log format, logging driver, paths).
• How Promtail collects and labels those logs so they can be filtered later in Grafana.

Important note: On Debian 13 (Trixie) rsyslog no longer exists by default, there is no /var/log/syslog or /var/log/messages. Everything from the system goes through systemd-journald, which is why several services are read with the journal type scraper.

List of services to monitor

1. SSH

SSH writes its events to the systemd journal through the ssh.service unit (on Debian the binary is sshd). There is no need to configure anything special in SSH to monitor it; it is enough to read the journal filtering by the corresponding unit.

To make sure that login attempts and source IPs are recorded with enough detail, it is worth reviewing the log level in /etc/ssh/sshd_config:

1# /etc/ssh/sshd_config
2LogLevel VERBOSE

With VERBOSE, the fingerprints of the keys used and more detail of each attempt are recorded. Apply the change:

1sudo systemctl restart ssh

To check what SSH is emitting to the journal:

1# Latest events from the ssh unit
2journalctl -u ssh -n 50 --no-pager
3
4# Follow failed attempts live
5journalctl -u ssh -f | grep -i "failed\|invalid\|accepted"

In Grafana you can run LogQL queries like these:

1# Failed SSH login attempts
2{unit="ssh.service"} |= "Failed password"
3
4# Successful logins
5{unit="ssh.service"} |= "Accepted"
6
7# Extract the source IP from each failed attempt
8{unit="ssh.service"} |= "Failed password"
9  | regexp "from (?P<ip>\\d+\\.\\d+\\.\\d+\\.\\d+)"

2. Nginx

For Nginx the ideal is to have a predictable log format and to split the logs per domain, so Promtail can label each file with its domain and tell access from error.

Define a clear log_format in /etc/nginx/nginx.conf inside the http block:

1# /etc/nginx/nginx.conf (http block)
2log_format monitor '$remote_addr - $remote_user [$time_local] '
3                   '"$request" $status $body_bytes_sent '
4                   '"$http_referer" "$http_user_agent" '
5                   'rt=$request_time';

Then, in each server block, point the logs to per-domain paths so the domain label comes out clean:

 1# /etc/nginx/sites-available/example.com
 2server {
 3    listen 80;
 4    server_name example.com;
 5
 6    access_log /var/log/nginx/example.com-access.log monitor;
 7    error_log  /var/log/nginx/example.com-error.log warn;
 8
 9    location / {
10        proxy_pass http://127.0.0.1:8080;
11        proxy_set_header Host $host;
12        proxy_set_header X-Real-IP $remote_addr;
13        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
14        proxy_set_header X-Forwarded-Proto $scheme;
15    }
16}

Validate and reload Nginx:

1sudo nginx -t
2sudo systemctl reload nginx

Example LogQL queries for the dashboard:

 1# Requests per second
 2sum(rate({job="nginx", log_type="access"}[1m]))
 3
 4# HTTP status code count
 5sum by (status) (count_over_time({job="nginx", log_type="access"}[5m]))
 6
 7# Only 5xx errors
 8{job="nginx", log_type="access"} | status =~ "5.."
 9
10# Nginx errors in real time
11{job="nginx", log_type="error"}

3. Fail2ban

Fail2ban writes its actions to /var/log/fail2ban.log, that is where bans and detections come from. It is worth making sure of the log level in its configuration.

In /etc/fail2ban/fail2ban.local (so as not to touch the .conf that gets overwritten on updates):

1# /etc/fail2ban/fail2ban.local
2[Definition]
3loglevel = INFO
4logtarget = /var/log/fail2ban.log

A minimal jail example for SSH in /etc/fail2ban/jail.local:

 1# /etc/fail2ban/jail.local
 2[DEFAULT]
 3bantime  = 1h
 4findtime = 10m
 5maxretry = 5
 6
 7[sshd]
 8enabled  = true
 9port     = ssh
10logpath  = %(sshd_log)s
11backend  = systemd

Restart the service:

1sudo systemctl restart fail2ban
2sudo fail2ban-client status sshd

LogQL queries for the defense dashboard:

1# Banned IPs
2{job="fail2ban"} |= "Ban" != "Unban"
3
4# Detected IPs (before the ban)
5{job="fail2ban"} |= "Found"
6
7# Ban trend over time
8sum(count_over_time({job="fail2ban"} |= "Ban" != "Unban" [1h]))

4. Docker

As seen in the first part, to monitor containers we ask Docker to write its logs to the systemd journal by changing the Logging Driver from json-file to journald.

Reminder of the /etc/docker/daemon.json file:

1sudo tee /etc/docker/daemon.json > /dev/null <<'EOF'
2{
3 "log-driver": "journald"
4}
5EOF
6sudo systemctl restart docker
7docker info --format '{{.LoggingDriver}}'   # Output: journald

With journald, Docker tags each line with metadata such as the container name and its image. Promtail exposes them as labels with relabel_configs on the same journal type scraper:

 1# Promtail scrape_configs (fragment) - Docker via journal
 2- job_name: docker
 3  journal:
 4    path: /var/log/journal
 5    max_age: 12h
 6    labels:
 7      job: docker
 8      host: vps-01
 9  relabel_configs:
10    # Only lines coming from Docker's journald driver
11    - source_labels: ['__journal_container_name']
12      target_label: container
13    - source_labels: ['__journal_image_name']
14      target_label: image
15    # Drop entries that are not from containers
16    - source_labels: ['__journal_container_name']
17      regex: '^$'
18      action: drop

LogQL queries for the containers dashboard:

1# Logs from a specific container
2{job="docker", container="my-app"}
3
4# Log volume per container
5sum by (container) (rate({job="docker"}[5m]))
6
7# Search for errors across all containers
8{job="docker"} |~ "(?i)error|fatal|panic"

Applying the Promtail configuration

Every time the scrape_configs are edited, you have to validate and restart the agent:

1# Restart Promtail
2sudo systemctl restart promtail
3
4# Check that there are no startup errors
5sudo journalctl -u promtail -n 50 --no-pager
6
7# Verify active targets from the Promtail API
8curl -s http://localhost:9080/targets | head

If Loki is receiving data, in Grafana → Explore the labels (job, unit, domain, container) should already appear for filtering.

Label summary per service

Conclusion

The key to good log-based monitoring is not only in the stack, but in each service emitting logs with a predictable format and in Promtail labeling them consistently. With well-defined labels (unit, domain, status, container) the LogQL queries become simple and the dashboards come together quickly.

In this part we covered the configuration examples for each monitored service. The configurations of the Grafana stack itself (Loki, grafana.ini and the full Promtail config.yml) I will leave on my side to complement these notes.

References

• Grafana
• Loki - Official Documentation
• Promtail - scrape_configs Configuration
• LogQL - Loki Query Language
• Nginx - log module
• Fail2ban - Official Documentation
• Docker - Logging Drivers

Translations:

• "Spanish" |