Centralized logging on FCOS?

Ask Fedora
1

Currently, there is no centralized logging package installed (e.g., rsyslog, syslog-ng). What do you use for this? And I think this will not be possible on toolbox and I don’t want to add new layer using rpm-ostree.

My use case is for host security like (sshd, sudo calls, custom firewall logging).

2

There is journald. I recommend fluentbit, personally.

Here’s the journald/systemd input for it to ship your FCOS los to a central logging system(s) of your choice:

This will also work for toolbox and other podman containers since logs end up in journald as well.

sudo dnf install fluent-bit to install it

3

I wan’t to run it like daemon without allocated tty more of like a systemd process. Ah yes fluentbit, I’ve tried it earlier but its unable to output the SSHD logs inside, it only outputs user journald logs. I am still trying to figure the config out for that.

4

I think it should possible to run any log forwarding daemon as a privileged container.

5

Yes, that’s what I’m doing but it seems there is a problem on my config for fluent-bit. It’s running on --privileged mode but I only get user journald logs.

podman run --rm -it --privileged \
  -v /etc/machine-id:/etc/machine-id:ro \
  -v /run/log/journal:/run/log/journal:ro \
  -v /var/log/journal:/var/log/journal:ro \
  docker.io/fluent/fluent-bit:latest \
  fluent-bit -i systemd \
    -p read_from_tail=on \
    -p systemd_filter="_SYSTEMD_UNIT=sshd.service" \
    -p tag=host.* \
    -o stdout

Found the solution as the journald gets ingested in fluent it renames the _SYSTEMD_UNIT to different session names so best is to debug it using a busybox or any container that has bash access. For sshd I used _COMM=sshd.

6

I take this to mean you figured out the problem?

2 Likes
7

Fluentbit absolutely can be ran as a service. I test it by running the command and when the output looks right, add it to the /etc/fluentbit config. The fluentbit docs give examples for both CLI and conf file syntax. If you really want to use a container instead of the native packaged fluentbit in Fedora (via rpm-ostree install fluent-bit), then you can use podman generate systemd to generate the systemd service for it as a podman container.

8

I have used quadlet instead of the generate systemd as that one is already, I think deprecated. Instead of going for fluent-bit I did go for timberio/datadog vector as log collector. As this also collects metrics.

9

TIL:

1 Like
10

My 2 cents : I’m using a containerized promtail service to collect logs from journald and send them to a loki instance, then, I can query logs with grafana

1 Like
11

This is the solution I have too. Works well.