How to Setup Rsyslog Remote Logging on Linux (Central Log Server)

by Lakshmanan Ganapathy on January 25, 2012

Every Linux distribution have some kind of logging mechanism that records all the system activities. A while back we provided a list of 20 log files that are stored under /var/log that you might be helpful during troubleshooting. These logs are very critical for sysadmin for troubleshooting purpose.

The following are the three common methods to log a message:

  1. Logging on the same server: Messages get written into the local hard drive/local database
  2. Logging on a remote server: Many systems forward their logs over the network to a central log server. On the central log server, the messages from various systems are written to the local hard drive/database.
  3. Relay logging: Branch ‘A’ and Branch ‘B’ logs the messages on 2 different servers. These server in-turn logs the message to the ‘Head Office’.


Rsyslog is the default logging program on several Linux distributions including Debian and Red Hat based systems. Apart from implementing the syslog protocol, rsyslog adds additional features such as content-based filtering. This also uses TCP for transporting, and provides lot of configuration options.

This article explains how to implement the method 2 mentioned above. i.e This explains how to setup a central logging server, and send logs from individual servers to the central logging server.

This setup will help you to analyze the log files of all the servers in your infrastructure from a central log server.

Installation

Rsyslog comes as the default logging program in Debian Distribution and Red Hat based systems. If you system doesn’t have rsyslog, install it as shown below depending on your distro.

apt-get install rsyslog rsyslog-doc
(or)
yum install rsyslog rsyslog-doc

Rsyslog configurations are stored in /etc/ryslog.conf file and the files under /etc/rsyslog.d/ directory.

Configuration Structure

Before understanding how to setup the central logging sever, it is good to understand the configuration structure of rsyslog.

Rsyslog configuration files are structed in the following manner

  1. Modules
  2. Configuration Directives
  3. Rule line

Modules

Rsyslog has a modular architecture. It enables functionalities to be added dynamically through these modules. The modules are categorized as:

  • Input Modules – Used to gather messages from various sources
  • Output Modules – Used to write the messages to various places ( file, socket etc.. )
  • Parser Modules – Used to parse the message content

Please note that there are also other categories of modules available. This is to give an overview of what modules can do.

Configuration Directives

All configuration directives must be specified one per line and must start with dollar sign ($). It affects the rules.

Rule line

Every rule line consists of two fields, a ‘selector field’ and an ‘action field’. The selector field is divided into two, ‘facilities & priorities’. Action specifies what action must be taken for the matched rule.

A Sample Configuration

######################
	MODULES
######################

$ModLoad imuxsock
$ModLoad imklog

######################
	Directives
######################
# Set the default permissions for all log files. 

$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755

######################
	RULES
######################
mail.info	/var/log/mail.info
mail.warn	/var/log/mail.warn
mail.err	/var/log/mail.err
daemon.*	/var/log/daemon.log

Note: 10 Examples for Viewing Huge Log Files in Linux might be helpful when you are manipulating log files.

Templates

Templates are a very important features provided by rsyslog. It allows the user to log the messages in their desirable format. It can also be used to create dynamic file names to log the messages. In case of database logging, the templates are used to convert the message into a proper SQL statement.

A sample template will look like:

$template mytemplate “Text-Before %msg% Text-After\n”

The above template will log the message “This is hello from rsyslog” as:

Text-Before This is hello from rsyslog Text-After

We will see how to use the template for generate the log files dynamically.

Central Logging Server

The above sections should have given an overview about rsyslog and how to configure it. Now we will move on to setup a central logging system.

For our discussion we will have server IP as “192.168.1.1” for the central log server, where all the log messages from client should be forwarded.

Add the following lines to the rsyslog.conf of the central log server servers (In this example, the following line was added on the log server with ip-address 192.168.1.1):

# provides support for local system logging
$ModLoad imuxsock 

# provides kernel logging support (previously done by rklogd)
$ModLoad imklog

# provides UDP syslog reception. For TCP, load imtcp.
$ModLoad imudp

# For TCP, InputServerRun 514
$UDPServerRun 514

# This one is the template to generate the log filename dynamically, depending on the client's IP address.
$template FILENAME,"/var/log/%fromhost-ip%/syslog.log"

# Log all messages to the dynamically formed file. Now each clients log (192.168.1.2, 192.168.1.3,etc...), will be under a separate directory which is formed by the template FILENAME.
*.* ?FILENAME

After adding the above lines to the rsyslog.conf, restart the rsyslog process. Now the rsyslog server will be ready to accept messages.

# service rsyslog restart
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]

Add the following lines to the rsyslog.conf on the individual client machines that should send their log messages to the central server.

$ModLoad imuxsock

$ModLoad imklog

# Provides UDP forwarding. The IP is the server's IP address
*.* @192.168.1.1:514 

# Provides TCP forwarding. But the current server runs on UDP
# *.* @@192.168.1.1:514

Restart the rsyslog process on the clients. Now the rsyslog central server (In this example, 192.168.1.1) will receive all the log messages from the configured clients and each client’s log will be placed under a separate directory.


Linux Sysadmin Course Linux provides several powerful administrative tools and utilities which will help you to manage your systems effectively. If you don’t know what these tools are and how to use them, you could be spending lot of time trying to perform even the basic administrative tasks. The focus of this course is to help you understand system administration tools, which will help you to become an effective Linux system administrator.
Get the Linux Sysadmin Course Now!

If you enjoyed this article, you might also like..

  1. 50 Linux Sysadmin Tutorials
  2. 50 Most Frequently Used Linux Commands (With Examples)
  3. Top 25 Best Linux Performance Monitoring and Debugging Tools
  4. Mommy, I found it! – 15 Practical Linux Find Command Examples
  5. Linux 101 Hacks 2nd Edition eBook Linux 101 Hacks Book

Bash 101 Hacks Book Sed and Awk 101 Hacks Book Nagios Core 3 Book Vim 101 Hacks Book

{ 20 comments… read them below or add one }

1 ian January 26, 2012 at 3:01 am

Nice simple tutorial. How about a quick test to confirm the system is working properly? Every project should do this.

2 jef January 26, 2012 at 3:55 am

nice write up. adding the iptables part to open the port would make it more easy for beginners.

3 Lakshmanan January 28, 2012 at 5:26 am

Thanks jef and ion for your comments. I will try to address ur points from my next article

4 roberto February 20, 2012 at 7:15 am

nice tutorial. it’s possible to send the ip address of client via logger command?

5 Lakshmanan Ganapathy February 21, 2012 at 11:30 pm

@roberto,

In logger command you can use the -t option ( used to tag ). So from client you can use logger -t , and I think this should work.

6 Doug April 2, 2012 at 5:15 pm

By chance, do you know how I would configure rsyslog to forward an actual log file?

7 Suresh April 18, 2012 at 2:38 am

Send me Linux Upadates

8 Ashok April 19, 2012 at 8:43 am

Solution to capture commands executed by all the users in Linux is here.

9 Adam Pie April 20, 2012 at 3:06 pm

# For TCP, InputServerRun 514

I think this is incorrect, should be InputTCPServerRun 514.

10 Ian August 1, 2012 at 6:30 pm

Nice succinct article. Helped me out. Thanks!

11 rizwan January 18, 2013 at 9:56 pm

good source to start with .thanks bhai

12 Luis October 18, 2013 at 10:59 am

Hi.
My question: What if the rsyslog server goes down? Does it make the other computers hang?

Thanks

13 Aby December 30, 2013 at 1:03 am

@Luis,

I think the messages will be spooled to disk and it will send once the server comes back , if its specified in configuration file,

from /etc/rsyslog.conf

# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files

14 Luis December 31, 2013 at 9:29 am

Thanks @Aby

Another question? Does the rsyslog mechanism introduce a lot of network traffic??

Thanks

15 fouinix January 8, 2014 at 6:05 am

Little advice :
Before the path of your log file you can add “-” to disable sync to disk. Usefull if you have lot of logs.
See :
http://www.rsyslog.com/doc/rsyslog_conf_actions.html
“You may prefix each entry with the minus “-” sign to omit syncing the file after every logging. Note that you might lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some performance, especially if you run programs that use logging in a very verbose manner.

If your system is connected to a reliable UPS and you receive lots of log data (e.g. firewall logs), it might be a very good idea to turn of syncing by specifying the “-” in front of the file name.”
You can manage the buffer with this directives :
http://www.rsyslog.com/doc/omfile.html
IOBufferSize
FlushInterval
ASyncWriting
FlushOnTXEnd

16 aloe February 23, 2014 at 4:57 am

Nice tutoria, thanksl! Unfortunately it wanted more work for me to make it run.
If it does not work for you. Here is a troubleshooting:
1) in the client try to ping the central server to assure that connection really works
1) in the central server create a folder for client messages under /var/log/
2) in the central server create a file /var/log//rsyslog.log
3) in the central server run chmod and change rights (i.e. 777) for /var/log//rsyslog.log
4) in the central server make sure, your firewall does not block your messages. In Ubuntu there is used “ufw” deamon. For test purpose try to disable it by “sudo service ufw stop”

If problem persists, run “cat var/log/syslog” to see what is wrong in the central server.

17 aloe March 2, 2014 at 3:10 am

to my previous comment no.16, because editor saved id badly. The IP of client is missing there. Correct is:
1) in the central server create a folder for client messages under /var/log/client_ip
2) in the central server create a file /var/log/client_ip/rsyslog.log
3) in the central server run chmod and change rights (i.e. 777) for /var/log/client_ip/rsyslog.log

18 Paul Groen March 26, 2014 at 8:25 am

Works great.
Only my /var/log/messages is also filled with all client logs, now / is almost full.
Now I’ve the same log files on two places.
Is it possible to use /var/log/messages only for the log server it self and that the clients archive their logging via the template location(on another file system /opt/data)?

19 Naidu-A April 2, 2014 at 2:43 am

I want to send only certain logs from /var/log/messages on client server to central rsys log server. So can I define any conditions to send only message like “UserAllowed” from client server to central rsys log server?.
Is there any parameter like “grep” I can define in configuration file?
Thanks!

20 Vishwanath May 1, 2014 at 10:40 pm

Hi Lakshmanan,
Nice article, I am beginner for syslog. We are using log4j for application logs, currently we are storing logs in loc. Can you please let me know how we can point log4j log files to remote host?

Thanks in advance

Leave a Comment

Previous post:

Next post: