Systemd journal collects logs from the kernel, services, and applications into a structured binary format. By default on most RHEL-family distributions, journal data lives in /run/log/journal/, a volatile tmpfs mount that gets wiped on every reboot. Ubuntu 24.04 and Debian 13 ship with persistent storage enabled out of the box, but without size limits, which means journals grow unchecked until you configure retention. Either way, you need to configure journald.conf for production use.
Persistent journal storage writes logs to /var/log/journal/ on disk, surviving reboots and giving you a complete timeline of system events. This guide covers enabling persistent systemd journal storage, configuring retention and size limits in journald.conf, rotating and vacuuming old logs, querying journals with journalctl, and setting up centralized logging with systemd-journal-remote. All steps work on RHEL 10/9, Rocky Linux 10/9, AlmaLinux 10/9, Ubuntu 24.04/22.04, and Debian 13/12. Full configuration reference is available in the official journald.conf documentation.
Verified working: March 2026 on Ubuntu 24.04 LTS (systemd 255) and Rocky Linux 10.1 (systemd 256)
Prerequisites
- A Linux server running any systemd-based distribution (RHEL/Rocky/AlmaLinux 9+, Ubuntu 22.04+, Debian 12+)
- Root or sudo access
- Sufficient disk space on
/varfor log retention (at least 1-2 GB recommended)
Step 1: Check Current Journal Storage
Before changing anything, confirm how your system currently stores journal data. The journalctl --header command shows where journals are written and the storage type in use.
journalctl --header | head -20Look at the file path in the output. If it shows /run/log/journal/, your journals are volatile and do not survive reboots:
File path: /var/log/journal/c062bd9d.../system.journal
File ID: 349eff25fb0e4ad49f00cb0dd7ddf735
Machine ID: c062bd9deb7c533216c9fa60191f76f3
Boot ID: c3a07bbbfd5f48a6b5bc83d1b983aa37
State: ONLINE
Compatible flags: TAIL_ENTRY_BOOT_ID
Incompatible flags: COMPRESSED-ZSTD KEYED-HASH COMPACTCheck disk usage of current journal files to understand how much space logs consume:
journalctl --disk-usageThe output shows the total size of archived and active journal files:
Archived and active journals take up 16.0M in the file system.Step 2: Enable Persistent Systemd Journal Storage
Persistent storage requires the /var/log/journal/ directory to exist. When this directory is present and Storage is set to auto (the default) or persistent, systemd-journald writes logs to disk instead of the volatile tmpfs.
Create the persistent journal directory with the correct ownership and permissions:
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journalThe systemd-tmpfiles command sets proper ownership (root:systemd-journal) and permissions on the directory. Verify the directory exists with correct ownership:
ls -ld /var/log/journalThe output should show the systemd-journal group:
drwxr-sr-x+ 3 root systemd-journal 4096 Mar 22 10:00 /var/log/journalRestart systemd-journald to pick up the new storage location:
sudo systemctl restart systemd-journaldVerify that journals are now written to persistent storage:
ls -la /var/log/journal/You should see a machine-ID subdirectory containing journal files. If the directory is populated, persistent storage is active.
Step 3: Configure journald.conf for Persistent Journal Storage
The main configuration file for systemd-journald is /etc/systemd/journald.conf. Key settings control where logs are stored, how large the journal can grow, how long logs are kept, and whether compression is enabled. Without limits, persistent journals can fill your disk over time, especially on busy servers.
Open the configuration file:
sudo vi /etc/systemd/journald.confUpdate the [Journal] section with these settings. Adjust SystemMaxUse and MaxRetentionSec based on your available disk space and compliance requirements:
[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=2G
SystemMaxFileSize=128M
SystemKeepFree=1G
MaxRetentionSec=6month
MaxFileSec=1week
ForwardToSyslog=no
RateLimitIntervalSec=30s
RateLimitBurst=10000Here is what each option does:
- Storage=persistent – explicitly write journals to
/var/log/journal/and create the directory if it does not exist - Compress=yes – compress journal data objects larger than 512 bytes, reducing disk usage significantly
- SystemMaxUse=2G – maximum total disk space journals can use (default is 10% of filesystem)
- SystemMaxFileSize=128M – maximum size of individual journal files before rotation
- SystemKeepFree=1G – always keep at least 1 GB free on the filesystem
- MaxRetentionSec=6month – delete journal entries older than 6 months
- MaxFileSec=1week – rotate journal files every week regardless of size
- ForwardToSyslog=no – disable forwarding to rsyslog to avoid duplicate logs (set to
yesif you use rsyslog for centralized logging) - RateLimitIntervalSec=30s and RateLimitBurst=10000 – allow up to 10,000 messages per 30-second interval per service before throttling kicks in
Restart systemd-journald to apply the new configuration:
sudo systemctl restart systemd-journaldConfirm the service is running with the updated settings:
systemctl status systemd-journaldThe service should show active (running) with no errors about the configuration file.
Step 4: Rotate and Vacuum Old Journal Logs
Journal rotation happens automatically based on the SystemMaxFileSize and MaxFileSec settings. You can also force rotation and clean up old entries manually when needed, for example after changing retention settings or when reclaiming disk space.
Force an immediate rotation of active journal files:
sudo journalctl --rotateAfter rotation, vacuum old journal files to enforce your size and time limits. You can vacuum by size, time, or number of files.
Remove journal data until total size is under 500 MB:
sudo journalctl --vacuum-size=500MRemove journal entries older than 30 days:
sudo journalctl --vacuum-time=30dKeep only the 5 most recent journal files:
sudo journalctl --vacuum-files=5Verify the disk usage after vacuuming to confirm space was reclaimed:
journalctl --disk-usageThe output should show a reduced total size compared to before the vacuum operation.
Step 5: Query Journals with journalctl
With persistent storage enabled, you can query logs across reboots and filter by time, service, or severity. Persistent journals are especially valuable for troubleshooting because you can trace issues back days or weeks instead of losing everything at reboot.
View Logs from Previous Boots
List all recorded boots to see how far back your journal history goes:
journalctl --list-bootsEach line shows a boot ID with the time range. The current boot is index 0, previous boot is -1, and so on:
IDX BOOT ID FIRST ENTRY LAST ENTRY
-1 a8f2e1... Mon 2026-03-24 08:12:01 UTC Mon 2026-03-24 23:59:59 UTC
0 c3a07bbb... Tue 2026-03-25 11:28:23 UTC Tue 2026-03-25 11:29:17 UTCView logs from the previous boot:
journalctl -b -1Filter by Time Range
Query logs within a specific time window using --since and --until:
journalctl --since "2026-03-21 08:00" --until "2026-03-21 18:00"You can also use relative time expressions:
journalctl --since "1 hour ago"
journalctl --since yesterday --until todayFilter by Service Unit
View logs for a specific systemd service using the -u flag:
journalctl -u nginx.service --no-pager -n 50This shows the last 50 log entries from the nginx service. Combine with time filters for targeted troubleshooting:
journalctl -u sshd.service --since "2026-03-22 00:00" --priority=errFilter by Log Priority
Syslog priority levels range from 0 (emergency) to 7 (debug). Filter for error-level and above to find problems fast:
journalctl -p errThis shows all entries at priority err (3) and above, including crit, alert, and emerg. Available priority names are: emerg, alert, crit, err, warning, notice, info, debug.
Follow logs in real time similar to tail -f:
journalctl -f -u nginx.serviceStep 6: Export and Forward Journal Logs
Systemd journal supports several output formats for integration with external log management tools. You can export structured data for analysis or forward logs to a traditional syslog server.
JSON Output
Export journal entries in JSON format for processing with tools like jq, Elasticsearch, or custom scripts:
journalctl -u sshd.service -n 5 -o json-prettyEach entry includes structured fields like _SYSTEMD_UNIT, MESSAGE, PRIORITY, _PID, and timestamps in microseconds. Other useful output formats include json (compact, one entry per line), short-iso (human-readable with ISO timestamps), and verbose (all fields).
Export to a File
Export journal data in binary format for archival or transfer to another system:
journalctl --since "2026-03-01" --until "2026-03-22" -o export > /tmp/journal-march.exportImport the exported journal on another system using systemd-journal-remote:
sudo /usr/lib/systemd/systemd-journal-remote --output=/var/log/journal/remote/ /tmp/journal-march.exportForward to Syslog
If you run a centralized rsyslog server, enable forwarding in journald.conf by setting ForwardToSyslog=yes. Rsyslog then receives journal messages through /run/systemd/journal/syslog and can forward them to a remote log collector.
Step 7: Monitor Journal Disk Usage
Persistent journals grow over time and can consume significant disk space on busy servers. Regular monitoring prevents unexpected disk-full situations. The built-in disk usage command gives a quick summary:
journalctl --disk-usageFor more granular monitoring, check the actual size of the journal directory:
sudo du -sh /var/log/journal/The output shows the total disk space consumed by all journal files:
19M /var/log/journal/To verify that journald respects your configured limits, check the active settings:
systemd-analyze cat-config systemd/journald.confThis command merges the main config with any drop-in overrides in /etc/systemd/journald.conf.d/ and shows the effective configuration. You can also set up a simple cron job or systemd timer to alert when journal usage exceeds a threshold. For full system monitoring that includes disk usage tracking, consider tools like Grafana with Telegraf.
Step 8: Centralized Logging with systemd-journal-remote
For multi-server environments, systemd-journal-remote provides native journal forwarding over HTTPS without needing an external log aggregator. The sending server runs systemd-journal-upload, and the receiving server runs systemd-journal-remote. Full details are in the systemd-journal-remote documentation.
Configure the Receiving Server
Install the journal remote package on the central log server. On RHEL/Rocky/AlmaLinux:
sudo dnf install systemd-journal-remoteOn Ubuntu/Debian:
sudo apt install systemd-journal-remoteEnable and start the receiving service:
sudo systemctl enable --now systemd-journal-remote.socketBy default, systemd-journal-remote listens on port 19532/tcp for incoming journal streams. Open this port in the firewall. On RHEL-family systems:
sudo firewall-cmd --permanent --add-port=19532/tcp
sudo firewall-cmd --reloadOn Ubuntu/Debian with UFW:
sudo ufw allow 19532/tcpConfigure the remote journal receiver by editing its configuration:
sudo vi /etc/systemd/journal-remote.confSet the output directory and split journals by host:
[Remote]
SplitMode=host
ServerKeyFile=/etc/ssl/private/journal-remote.key
ServerCertificateFile=/etc/ssl/certs/journal-remote.crt
TrustedCertificateFile=/etc/ssl/ca/trusted.crtThe SplitMode=host setting creates separate journal files for each sending host, making it simple to query logs per server. For production use, configure proper TLS certificates. For testing, you can use HTTP by passing --listen-http=-3 to the service (not recommended for production).
Configure the Sending Server
Install the upload component on each server that should forward logs:
sudo dnf install systemd-journal-remoteEdit the upload configuration to point to the central log server:
sudo vi /etc/systemd/journal-upload.confSet the target URL to your central log server:
[Upload]
URL=https://logserver.example.com:19532Enable and start the upload service:
sudo systemctl enable --now systemd-journal-upload.serviceVerify the upload service is running and connected:
systemctl status systemd-journal-upload.serviceOn the central server, confirm that remote journal files are being created:
ls -la /var/log/journal/remote/You should see journal files named after the remote host’s machine ID. Query remote journals the same way as local ones by pointing journalctl at the remote directory.
Reference: journald.conf Configuration Options
This table summarizes the key journald.conf options for controlling persistent journal behavior:
| Option | Description | Default |
|---|---|---|
Storage | Where to store journals: volatile, persistent, auto, or none | auto |
Compress | Compress data objects in journal files | yes |
SystemMaxUse | Maximum disk space for persistent journals | 10% of filesystem |
SystemKeepFree | Minimum free space to leave on the filesystem | 15% of filesystem |
SystemMaxFileSize | Maximum size of individual journal files | 1/8 of SystemMaxUse |
MaxRetentionSec | Maximum time to store journal entries | 0 (disabled) |
MaxFileSec | Time interval for rotating journal files | 1month |
ForwardToSyslog | Forward journal messages to syslog socket | no |
RateLimitIntervalSec | Time window for rate limiting per service | 30s |
RateLimitBurst | Maximum messages per interval before throttling | 10000 |
RuntimeMaxUse | Maximum disk space for volatile journals (in /run) | 10% of /run |
Conclusion
Persistent systemd journal storage gives you reliable log history across reboots, structured queries with journalctl, and native log forwarding between servers. With proper size and retention limits in journald.conf, journals stay manageable even on busy production systems.
For production environments, configure TLS certificates on systemd-journal-remote connections, set up log rotation policies that match your compliance requirements, and monitor /var/log/journal/ disk usage alongside your regular infrastructure monitoring. If you need full-text search or visualization across many servers, forward journals to a dedicated log platform like Graylog or Elasticsearch. To automate log maintenance tasks on a schedule, see our guide on configuring systemd timers.
