Logs on any Linux system are critical for analyzing and troubleshooting any issues related to system and applications. With the help of tools like Graylog, you can easily ship these logs to a centralized platform for easy visualization. In this guide, we will look at how to Configure Rsyslog Centralized Log Server on Ubuntu 18.04 LTS.
On Linux, by default, all log files are located under /var/log directory. There are several types of log files storing varying messages, which can be cron, kernel, security, events, users e.t.c. Mostly these logs file are controlled by rsyslog service.
On recent systems with systemd, some logs are managed by journald daemon and they are written binary format. These logs are volatile since they are written to RAM and doesn’t withstand system reboot. They are often found on.
/run/log/journal/ But note that journald can also be configured to permanently store log messages by writing to file.
Configure Rsyslog as central Log Server on Ubuntu 18.04
We’re going to configure rsyslog server as central Log management system. This follows the client-server model where rsyslog service will listen on either udp/tcp port. The default port used by rsyslog is 514. On the client system, rsyslog will collect and ship logs to a central rsyslog server over the network via UDP or TCP ports.
When working with syslog messages, there is a priority/severity level that characterizes a log file. Namely:
- emerg, panic (Emergency ): Level 0 – This is the lowest log level. system is unusable
- alert (Alerts): Level 1 – action must be taken immediately
- err (Errors): Level 3 – critical conditions
- warn (Warnings): Level 4 – warning conditions
- notice (Notification): Level 5 – normal but significant condition
- info (Information): Level 6 – informational messages
- debug (Debugging): Level 7 – This is the highest level – debug-level messages
Rsyslog is installed by default on a freshly installed Ubuntu 18.04 system. If for any reason the package is not installed, you can install it by running:
sudo apt-get install rsyslog
When installed, check service to see if it is running:
$ sudo systemctl status rsyslog ● rsyslog.service - System Logging Service Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2018-07-18 11:30:10 UTC; 4 days ago Main PID: 903 (rsyslogd) Tasks: 4 (limit: 4704) CGroup: /system.slice/rsyslog.service └─903 /usr/sbin/rsyslogd -n
Configure rsyslog to run in Server Mode
Now configure rsyslog service to run in server mode:
sudo vim /etc/rsyslog.conf
Uncomment the lines for udp and tcp port binding:
module(load="imudp") input(type="imudp" port="514") # provides TCP syslog reception module(load="imtcp") input(type="imtcp" port="514")
If you would like to limit access from to specific subnet, IP or domain, add like below:
$AllowedSender TCP, 127.0.0.1, 192.168.10.0/24, *.example.com
You can add above line after
input(type="imtcp" port="514") line. Remember to substitute given values with correct ones
Create a new template for receiving remote messages
Let’s create a template that will instruct rsyslog server how to store incoming syslog messages. Add the template just before
GLOBAL DIRECTIVES section:
$template remote-incoming-logs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log" *.* ?remote-incoming-logs & ~
The received logs will be parsed using the template above and stored inside directory
/var/log/. The file naming follows the convention:
%HOSTNAME% and %PROGRAMNAME% variables, i.e the client hostname, and client facility that produced the log message.
The & ~ instructs rsyslog daemon to store the log message only to a specified file.
Other variables that can be used include:
%syslogseverity%, %syslogfacility%, %timegenerated%, %HOSTNAME%, %syslogtag%, %msg%, %FROMHOST-IP%, %PRI%, %MSGID%, %APP-NAME%, %TIMESTAMP%, %$year%, %$month%, %$day%
Restart rsyslog service for the changes to take effect:
sudo systemctl restart rsyslog
Confirm if the service is listening on configured ports:
$ ss -tunelp | grep 514 udp UNCONN 0 0 0.0.0.0:514 0.0.0.0:* users:(("rsyslogd",pid=15220,fd=5)) ino:441816 sk:2 <-> udp UNCONN 0 0 [::]:514 [::]:* users:(("rsyslogd",pid=15220,fd=6)) ino:441817 sk:5 v6only:1 <-> tcp LISTEN 0 25 0.0.0.0:514 0.0.0.0:* users:(("rsyslogd",pid=15220,fd=7)) ino:441820 sk:a <-> tcp LISTEN 0 25 [::]:514 [::]:* users:(("rsyslogd",pid=15220,fd=8)) ino:441821 sk:11 v6only:1 <->
Configure Rsyslog firewall
If you have ufw firewall service running, allow rsyslog firewall ports:
sudo ufw allow 514/tcp
sudo ufw allow 514/udp
Configure Rsyslog as a Client
Once you’re done configuring rsyslog server, head over to your rsyslog client machines and configure them to send logs to remote rsyslog server.
sudo vim /etc/rsyslog.conf
Allow preservation of FQDN:
Add remote rsyslog server at the end:
You can also use FQDN instead of Server IP Address:
The above line will enable sending of logs over UDP, for tcp use
@@ instead of a single
Also add the following for when rsyslog server will be down:
$ActionQueueFileName queue $ActionQueueMaxDiskSpace 1g $ActionQueueSaveOnShutdown on $ActionQueueType LinkedList $ActionResumeRetryCount -1
Then restart rsyslog service
sudo systemtcl restart rsyslog