In this guide, we’ll look at how you can persist the Systemd Journals logs on your Server. Preserving the System Journal can be useful for troubleshooting your services when things keep breaking. At the end of this article, you should be able to configure your system journal to preserve the record of events when a server is rebooted.
Similar articles: How To Forward Logs to Grafana Loki using Promtail
The System journals Basics
Systemd, by default, stores the system journals in the /run/log/journal directory. As discussed in our Understanding the Linux File System Hierarchy everything in /run directory is cleared and contents are recreated on reboot. This means the journals are cleared when the system reboots.
We can adjust the configuration settings of the systemd-journald service in the
/etc/systemd/journald.conf file to make the journals persist across reboot. Open the file to view its contents:
$ sudo vim /etc/systemd/journald.conf
These are the common options which can be configured – Most will be commented out in most systems for you to adjust accordingly.
[Journal] #Storage=auto #Compress=yes #Seal=yes #SplitMode=uid #SyncIntervalSec=5m #RateLimitInterval=30s #RateLimitBurst=1000 #SystemMaxUse= #SystemKeepFree= #SystemMaxFileSize= #RuntimeMaxUse= #RuntimeKeepFree= #RuntimeMaxFileSize= #MaxRetentionSec= #MaxFileSec=1month #ForwardToSyslog=yes #ForwardToKMsg=no #ForwardToConsole=no #ForwardToWall=yes #TTYPath=/dev/console #MaxLevelStore=debug #MaxLevelSyslog=debug #MaxLevelKMsg=notice #MaxLevelConsole=info #MaxLevelWall=emerg #LineMax=48K
Configuring Persistent System Journals
To configure systemd-journald service to preserve system journals persistently across reboot, you need to set Storage to persistent:
Other values that can be set for Storage parameter are:
persistent: Stores journals in the
/var/log/journaldirectory which persists across reboots. Directory is created if doesn’t exist
volatile: Stores journals in the volatile
/run/log/journaldirectory. This doesn’t persist system reboots.
auto: The rsyslog will determine whether to use persistent or volatile storage. If the
/var/log/journaldirectory exists, then rsyslog uses persistent storage, otherwise it uses volatile storage.
For persistent journals storage, set it to:
Once the changes are committed, restart systemd-journald service to bring the configuration changes into effect.
sudo systemctl restart systemd-journald
Confirm the service is restarted and running:
$ systemctl status systemd-journald ● systemd-journald.service - Journal Service Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled) Active: active (running) since Sun 2020-02-09 10:59:36 UTC; 8s ago Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 5299 (systemd-journal) Status: "Processing requests..." CGroup: /system.slice/systemd-journald.service └─5299 /usr/lib/systemd/systemd-journald Feb 09 10:59:36 cent7.novalocal systemd-journal: Permanent journal is using 8.0M (max allowed 1.9G, trying to leave 2.9G free of 16.9G a…it 1.9G). Feb 09 10:59:36 cent7.novalocal systemd-journal: Permanent journal is using 8.0M (max allowed 1.9G, trying to leave 2.9G free of 16.9G a…it 1.9G). Feb 09 10:59:36 cent7.novalocal systemd-journal: Time spent on flushing to /var is 68.506ms for 7962 entries. Feb 09 10:59:36 cent7.novalocal systemd-journal: Journal started Hint: Some lines were ellipsized, use -l to show in full.
The /var/log/journal directory should be created.
$ ls /var/log/journal 65113b1a0d2f6087d515e6a8cd0ee7ef
The subdirectories under
/var/log/journal have hexadecimal characters in their long names and contain
$ ls /var/log/journal/65113b1a0d2f6087d515e6a8cd0ee7ef/ system.journal
*.journalfiles are the binary files that store the structured and indexed journal entries.
Adjusting Maximum storage usage for Journals
You can set the maximum size of the persistent journal by uncommenting and changing the following:
The default size limit is set to a value of 10% of the size of the underlying file system but capped at 4 GiB.
sudo systemctl restart systemd-journald