Title: Systemd journald cheatsheet | |
Author: Solène | |
Date: 24 December 2024 | |
Tags: linux | |
Description: In this blog post, you will learn how to deal with logs | |
managed by systemd-journald | |
# Introduction | |
This blog post is part of a series that will be about Systemd | |
ecosystem, today's focus is on journaling. | |
Systemd got a regrettable reputation since its arrival mid 2010. I | |
think this is due to Systemd being radically different than traditional | |
tooling, and people got lost without a chance to be noticed beforehand | |
they would have to deal with it. The transition was maybe rushed a bit | |
with a half-baked product, in addition to the fact users had to learn | |
new paradigms and tooling to operate their computer. | |
Nowadays, Systemd is working well, and there are serious non-Systemd | |
alternatives, so everyone should be happy. :) | |
# Introduction to journald | |
Journald is the logging system that was created as part of Systemd. It | |
handles logs created by all Systemd units. A huge difference compared | |
to the traditional logs is that there is a single journal file acting | |
as a database to store all the data. If you want to read logs, you | |
need to use `journalctl` command to extract data from the database as | |
it is not plain text. | |
Most of the time journald logs data from units by reading their | |
standard error and output, but it is possible to send data to journald | |
directly. | |
On the command line, you can use `systemd-cat` to run a program or pipe | |
data to it to send them to logs. | |
systemd-cat man page | |
# Journalctl 101 | |
Here is a list of the most common cases you will encounter: | |
* View new logs live: `journalctl -f` | |
* View last 2000 lines of logs: `journalctl -n 2000` | |
* Restrict logs to a given unit: `journalctl -u nginx.service` | |
* Pattern matching: `journalctl -g somepattern` | |
* Filter by date (since): `journalctl --since="10 minutes ago"` or | |
`journalctl --since="1 hour ago"` or `journalctl --since=2024-12-01` | |
* Filter by date (range): `journalctl --since="today" --until="1 hour | |
ago"` or `journalctl --since="2024-12-01 12:30:00" --until="2024-12-01 | |
16:00:00"` | |
* Filter logs since boot: `journalctl -b` | |
* Filter logs to previous (n-1) boot: `journalctl -b -1` | |
* Switch date time output to UTC: `journalctl --utc` | |
You can use multiple parameters at the same time: | |
* Last 200 lines of logs of nginx since current boot: `journalctl -n | |
200 -u nginx -b` | |
* Live display of nginx logs files matching "wp-content": `journalctl | |
-f -g wg-content -u nginx` | |
journalctl man page | |
# Send logs to syslog | |
If you want to bypass journald and send all messages to syslog to | |
handle your logs with it, you can edit the file | |
`/etc/systemd/journald.conf` to add the line `ForwardToSyslog=Yes`. | |
This will make journald relay all incoming messages to syslog, so you | |
can process your logs as you want. | |
Restart journald service: `systemctl restart systemd-journal.service` | |
systemd-journald man page | |
journald.conf man page | |
# Journald entries metadata | |
Journalctl contains a lot more information than just the log line (raw | |
content). Traditional syslog files contain the date and time, maybe | |
the hostname, and the log message. | |
This is just for information, only system administrators will ever need | |
to dig through this, it is important to know it exists in case you need | |
it. | |
## Example | |
Here is what journald stores for each line (pretty printed from json | |
output), using samba server as an example. | |
``` | |
# journalctl -u smbd -o json -n 1 | jq | |
{ | |
"_EXE": "/usr/libexec/samba/rpcd_winreg", | |
"_CMDLINE": "/usr/libexec/samba/rpcd_winreg --configfile=/etc/samba/smb.conf … | |
"_RUNTIME_SCOPE": "system", | |
"__MONOTONIC_TIMESTAMP": "749298223244", | |
"_SYSTEMD_SLICE": "system.slice", | |
"MESSAGE": " Copyright Andrew Tridgell and the Samba Team 1992-2023", | |
"_MACHINE_ID": "f23c6ba22f8e02aaa8a9722df464cae3", | |
"_SYSTEMD_INVOCATION_ID": "86f0f618c0b7dedee832aef6b28156e7", | |
"_BOOT_ID": "42d47e1b9a109551eaf1bc82bd242aef", | |
"_GID": "0", | |
"PRIORITY": "5", | |
"SYSLOG_IDENTIFIER": "rpcd_winreg", | |
"SYSLOG_TIMESTAMP": "Dec 19 11:00:03 ", | |
"SYSLOG_RAW": "<29>Dec 19 11:00:03 rpcd_winreg[4142801]: Copyright Andrew T… | |
"_CAP_EFFECTIVE": "1ffffffffff", | |
"_SYSTEMD_UNIT": "smbd.service", | |
"_PID": "4142801", | |
"_HOSTNAME": "pelleteuse", | |
"_SYSTEMD_CGROUP": "/system.slice/smbd.service", | |
"_UID": "0", | |
"SYSLOG_PID": "4142801", | |
"_TRANSPORT": "syslog", | |
"__REALTIME_TIMESTAMP": "1734606003126791", | |
"__CURSOR": "s=1ab47d484c31144909c90b4b97f3061d;i=bcdb43;b=42d47e1b9a109551ea… | |
"_SOURCE_REALTIME_TIMESTAMP": "1734606003126496", | |
"SYSLOG_FACILITY": "3", | |
"__SEQNUM": "12376899", | |
"_COMM": "rpcd_winreg", | |
"__SEQNUM_ID": "1ab47d484c31144909c90b4b97f3061d", | |
"_SELINUX_CONTEXT": "unconfined\n" | |
} | |
``` | |
The "real" log line is the value of `SYSLOG_RAW`, everything else is | |
created by journald when it receives the information. | |
## Filter | |
As the logs can be extracted in JSON format, it becomes easy to parse | |
them properly using any programming language able to deserialize JSON | |
data, this is far more robust than piping lines to AWK / grep, although | |
it can work "most of the time" (until it does not due to a weird | |
input). | |
On the command line, you can query/filter such logs using `jq` which is | |
a bit the awk of JSON. For instance, if I output all the logs of | |
"today" to filter lines generated by the binary `/usr/sbin/sshd`, I can | |
use this: | |
``` | |
journalctl --since="today" -o json | jq -s '.[] | select(._EXE == "/usr/sbin/ss… | |
``` | |
This command line will report each line of logs where "_EXE" field is | |
exactly "/usr/sbin/sshd" and all the metadata. This kind of data can | |
be useful when you need to filter tightly for a problem or a security | |
incident. | |
The example above was made easy as it is a bit silly in its form: | |
filtering on SSH server can be done with `journalctl -u sshd.service | |
--since=today`. | |
# Conclusion | |
Journald is a powerful logging system, journalctl provides a single | |
entry point to extract and filter logs in a unified system. | |
With journald, it became easy to read logs of multiple services over a | |
time range, and log rotation is now a problem of the past for me. |