Title: Systemd journald cheatsheet

	Title: Systemd journald cheatsheet
	Author: Solène
	Date: 24 December 2024
	Tags: linux
	Description: In this blog post, you will learn how to deal with logs
	managed by systemd-journald

	# Introduction

	This blog post is part of a series that will be about Systemd
	ecosystem, today's focus is on journaling.

	Systemd got a regrettable reputation since its arrival mid 2010. I
	think this is due to Systemd being radically different than traditional
	tooling, and people got lost without a chance to be noticed beforehand
	they would have to deal with it. The transition was maybe rushed a bit
	with a half-baked product, in addition to the fact users had to learn
	new paradigms and tooling to operate their computer.

	Nowadays, Systemd is working well, and there are serious non-Systemd
	alternatives, so everyone should be happy. :)

	# Introduction to journald

	Journald is the logging system that was created as part of Systemd. It
	handles logs created by all Systemd units. A huge difference compared
	to the traditional logs is that there is a single journal file acting
	as a database to store all the data. If you want to read logs, you
	need to use `journalctl` command to extract data from the database as
	it is not plain text.

	Most of the time journald logs data from units by reading their
	standard error and output, but it is possible to send data to journald
	directly.

	On the command line, you can use `systemd-cat` to run a program or pipe
	data to it to send them to logs.

	systemd-cat man page

	# Journalctl 101

	Here is a list of the most common cases you will encounter:

	* View new logs live: `journalctl -f`
	* View last 2000 lines of logs: `journalctl -n 2000`
	* Restrict logs to a given unit: `journalctl -u nginx.service`
	* Pattern matching: `journalctl -g somepattern`
	* Filter by date (since): `journalctl --since="10 minutes ago"` or
	`journalctl --since="1 hour ago"` or `journalctl --since=2024-12-01`
	* Filter by date (range): `journalctl --since="today" --until="1 hour
	ago"` or `journalctl --since="2024-12-01 12:30:00" --until="2024-12-01
	16:00:00"`
	* Filter logs since boot: `journalctl -b`
	* Filter logs to previous (n-1) boot: `journalctl -b -1`
	* Switch date time output to UTC: `journalctl --utc`

	You can use multiple parameters at the same time:

	* Last 200 lines of logs of nginx since current boot: `journalctl -n
	200 -u nginx -b`
	* Live display of nginx logs files matching "wp-content": `journalctl
	-f -g wg-content -u nginx`

	journalctl man page

	# Send logs to syslog

	If you want to bypass journald and send all messages to syslog to
	handle your logs with it, you can edit the file
	`/etc/systemd/journald.conf` to add the line `ForwardToSyslog=Yes`.

	This will make journald relay all incoming messages to syslog, so you
	can process your logs as you want.

	Restart journald service: `systemctl restart systemd-journal.service`

	systemd-journald man page
	journald.conf man page

	# Journald entries metadata

	Journalctl contains a lot more information than just the log line (raw
	content). Traditional syslog files contain the date and time, maybe
	the hostname, and the log message.

	This is just for information, only system administrators will ever need
	to dig through this, it is important to know it exists in case you need
	it.

	## Example

	Here is what journald stores for each line (pretty printed from json
	output), using samba server as an example.

	```
	# journalctl -u smbd -o json -n 1 \| jq
	{
	"_EXE": "/usr/libexec/samba/rpcd_winreg",
	"_CMDLINE": "/usr/libexec/samba/rpcd_winreg --configfile=/etc/samba/smb.conf …
	"_RUNTIME_SCOPE": "system",
	"__MONOTONIC_TIMESTAMP": "749298223244",
	"_SYSTEMD_SLICE": "system.slice",
	"MESSAGE": " Copyright Andrew Tridgell and the Samba Team 1992-2023",
	"_MACHINE_ID": "f23c6ba22f8e02aaa8a9722df464cae3",
	"_SYSTEMD_INVOCATION_ID": "86f0f618c0b7dedee832aef6b28156e7",
	"_BOOT_ID": "42d47e1b9a109551eaf1bc82bd242aef",
	"_GID": "0",
	"PRIORITY": "5",
	"SYSLOG_IDENTIFIER": "rpcd_winreg",
	"SYSLOG_TIMESTAMP": "Dec 19 11:00:03 ",
	"SYSLOG_RAW": "<29>Dec 19 11:00:03 rpcd_winreg[4142801]: Copyright Andrew T…
	"_CAP_EFFECTIVE": "1ffffffffff",
	"_SYSTEMD_UNIT": "smbd.service",
	"_PID": "4142801",
	"_HOSTNAME": "pelleteuse",
	"_SYSTEMD_CGROUP": "/system.slice/smbd.service",
	"_UID": "0",
	"SYSLOG_PID": "4142801",
	"_TRANSPORT": "syslog",
	"__REALTIME_TIMESTAMP": "1734606003126791",
	"__CURSOR": "s=1ab47d484c31144909c90b4b97f3061d;i=bcdb43;b=42d47e1b9a109551ea…
	"_SOURCE_REALTIME_TIMESTAMP": "1734606003126496",
	"SYSLOG_FACILITY": "3",
	"__SEQNUM": "12376899",
	"_COMM": "rpcd_winreg",
	"__SEQNUM_ID": "1ab47d484c31144909c90b4b97f3061d",
	"_SELINUX_CONTEXT": "unconfined\n"
	}
	```

	The "real" log line is the value of `SYSLOG_RAW`, everything else is
	created by journald when it receives the information.

	## Filter

	As the logs can be extracted in JSON format, it becomes easy to parse
	them properly using any programming language able to deserialize JSON
	data, this is far more robust than piping lines to AWK / grep, although
	it can work "most of the time" (until it does not due to a weird
	input).

	On the command line, you can query/filter such logs using `jq` which is
	a bit the awk of JSON. For instance, if I output all the logs of
	"today" to filter lines generated by the binary `/usr/sbin/sshd`, I can
	use this:

	```
	journalctl --since="today" -o json \| jq -s '.[] \| select(._EXE == "/usr/sbin/ss…
	```

	This command line will report each line of logs where "_EXE" field is
	exactly "/usr/sbin/sshd" and all the metadata. This kind of data can
	be useful when you need to filter tightly for a problem or a security
	incident.

	The example above was made easy as it is a bit silly in its form:
	filtering on SSH server can be done with `journalctl -u sshd.service
	--since=today`.

	# Conclusion

	Journald is a powerful logging system, journalctl provides a single
	entry point to extract and filter logs in a unified system.

	With journald, it became easy to read logs of multiple services over a
	time range, and log rotation is now a problem of the past for me.