How to Setup Rsyslog Remote Logging on Linux (Central Log Server)

Every Linux distribution have some kind of logging mechanism that records all the system activities. A while back we provided a list of 20 log files that are stored under /var/log that you might be helpful during troubleshooting. These logs are very critical for sysadmin for troubleshooting purpose.

The following are the three common methods to log a message:

1. Logging on the same server: Messages get written into the local hard drive/local database
2. Logging on a remote server: Many systems forward their logs over the network to a central log server. On the central log server, the messages from various systems are written to the local hard drive/database.
3. Relay logging: Branch ‘A’ and Branch ‘B’ logs the messages on 2 different servers. These server in-turn logs the message to the ‘Head Office’.

Rsyslog is the default logging program on several Linux distributions including Debian and Red Hat based systems. Apart from implementing the syslog protocol, rsyslog adds additional features such as content-based filtering. This also uses TCP for transporting, and provides lot of configuration options.

This article explains how to implement the method 2 mentioned above. i.e This explains how to setup a central logging server, and send logs from individual servers to the central logging server.

This setup will help you to analyze the log files of all the servers in your infrastructure from a central log server.

Installation

Rsyslog comes as the default logging program in Debian Distribution and Red Hat based systems. If you system doesn’t have rsyslog, install it as shown below depending on your distro.

apt-get install rsyslog rsyslog-doc
(or)
yum install rsyslog rsyslog-doc

Rsyslog configurations are stored in /etc/ryslog.conf file and the files under /etc/rsyslog.d/ directory.

Configuration Structure

Before understanding how to setup the central logging sever, it is good to understand the configuration structure of rsyslog.

Rsyslog configuration files are structed in the following manner

1. Modules
2. Configuration Directives
3. Rule line

Modules

Rsyslog has a modular architecture. It enables functionalities to be added dynamically through these modules. The modules are categorized as:

• Input Modules – Used to gather messages from various sources
• Output Modules – Used to write the messages to various places ( file, socket etc.. )
• Parser Modules – Used to parse the message content

Please note that there are also other categories of modules available. This is to give an overview of what modules can do.

Configuration Directives

All configuration directives must be specified one per line and must start with dollar sign ($). It affects the rules.

Rule line

Every rule line consists of two fields, a ‘selector field’ and an ‘action field’. The selector field is divided into two, ‘facilities & priorities’. Action specifies what action must be taken for the matched rule.

A Sample Configuration

######################
	MODULES
######################

$ModLoad imuxsock
$ModLoad imklog

######################
	Directives
######################
# Set the default permissions for all log files. 

$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755

######################
	RULES
######################
mail.info	/var/log/mail.info
mail.warn	/var/log/mail.warn
mail.err	/var/log/mail.err
daemon.*	/var/log/daemon.log

Note: 10 Examples for Viewing Huge Log Files in Linux might be helpful when you are manipulating log files.

Templates

Templates are a very important features provided by rsyslog. It allows the user to log the messages in their desirable format. It can also be used to create dynamic file names to log the messages. In case of database logging, the templates are used to convert the message into a proper SQL statement.

A sample template will look like:

$template mytemplate “Text-Before %msg% Text-After\n”

The above template will log the message “This is hello from rsyslog” as:

Text-Before This is hello from rsyslog Text-After

We will see how to use the template for generate the log files dynamically.

Central Logging Server

The above sections should have given an overview about rsyslog and how to configure it. Now we will move on to setup a central logging system.

For our discussion we will have server IP as “192.168.1.1” for the central log server, where all the log messages from client should be forwarded.

Add the following lines to the rsyslog.conf of the central log server servers (In this example, the following line was added on the log server with ip-address 192.168.1.1):

# provides support for local system logging
$ModLoad imuxsock 

# provides kernel logging support (previously done by rklogd)
$ModLoad imklog

# provides UDP syslog reception. For TCP, load imtcp.
$ModLoad imudp

# For TCP, InputServerRun 514
$UDPServerRun 514

# This one is the template to generate the log filename dynamically, depending on the client's IP address.
$template FILENAME,"/var/log/%fromhost-ip%/syslog.log"

# Log all messages to the dynamically formed file. Now each clients log (192.168.1.2, 192.168.1.3,etc...), will be under a separate directory which is formed by the template FILENAME.
*.* ?FILENAME

After adding the above lines to the rsyslog.conf, restart the rsyslog process. Now the rsyslog server will be ready to accept messages.

# service rsyslog restart
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]

Add the following lines to the rsyslog.conf on the individual client machines that should send their log messages to the central server.

$ModLoad imuxsock

$ModLoad imklog

# Provides UDP forwarding. The IP is the server's IP address
*.* @192.168.1.1:514 

# Provides TCP forwarding. But the current server runs on UDP
# *.* @@192.168.1.1:514

Restart the rsyslog process on the clients. Now the rsyslog central server (In this example, 192.168.1.1) will receive all the log messages from the configured clients and each client’s log will be placed under a separate directory.

Tweet

Share

 Comment

If you enjoyed this article, you might also like..