Every Linux distribution have some kind of logging mechanism that records all the system activities. A while back we provided a list of 20 log files that are stored under /var/log that you might be helpful during troubleshooting. These logs are very critical for sysadmin for troubleshooting purpose.
The following are the three common methods to log a message:
- Logging on the same server: Messages get written into the local hard drive/local database
- Logging on a remote server: Many systems forward their logs over the network to a central log server. On the central log server, the messages from various systems are written to the local hard drive/database.
- Relay logging: Branch ‘A’ and Branch ‘B’ logs the messages on 2 different servers. These server in-turn logs the message to the ‘Head Office’.
Rsyslog is the default logging program on several Linux distributions including Debian and Red Hat based systems. Apart from implementing the syslog protocol, rsyslog adds additional features such as content-based filtering. This also uses TCP for transporting, and provides lot of configuration options.
This article explains how to implement the method 2 mentioned above. i.e This explains how to setup a central logging server, and send logs from individual servers to the central logging server.
This setup will help you to analyze the log files of all the servers in your infrastructure from a central log server.
Installation
Rsyslog comes as the default logging program in Debian Distribution and Red Hat based systems. If you system doesn’t have rsyslog, install it as shown below depending on your distro.
apt-get install rsyslog rsyslog-doc (or) yum install rsyslog rsyslog-doc
Rsyslog configurations are stored in /etc/ryslog.conf file and the files under /etc/rsyslog.d/ directory.
Configuration Structure
Before understanding how to setup the central logging sever, it is good to understand the configuration structure of rsyslog.
Rsyslog configuration files are structed in the following manner
- Modules
- Configuration Directives
- Rule line
Modules
Rsyslog has a modular architecture. It enables functionalities to be added dynamically through these modules. The modules are categorized as:
- Input Modules – Used to gather messages from various sources
- Output Modules – Used to write the messages to various places ( file, socket etc.. )
- Parser Modules – Used to parse the message content
Please note that there are also other categories of modules available. This is to give an overview of what modules can do.
Configuration Directives
All configuration directives must be specified one per line and must start with dollar sign ($). It affects the rules.
Rule line
Every rule line consists of two fields, a ‘selector field’ and an ‘action field’. The selector field is divided into two, ‘facilities & priorities’. Action specifies what action must be taken for the matched rule.
A Sample Configuration
###################### MODULES ###################### $ModLoad imuxsock $ModLoad imklog ###################### Directives ###################### # Set the default permissions for all log files. $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 ###################### RULES ###################### mail.info /var/log/mail.info mail.warn /var/log/mail.warn mail.err /var/log/mail.err daemon.* /var/log/daemon.log
Note: 10 Examples for Viewing Huge Log Files in Linux might be helpful when you are manipulating log files.
Templates
Templates are a very important features provided by rsyslog. It allows the user to log the messages in their desirable format. It can also be used to create dynamic file names to log the messages. In case of database logging, the templates are used to convert the message into a proper SQL statement.
A sample template will look like:
$template mytemplate “Text-Before %msg% Text-After\n”
The above template will log the message “This is hello from rsyslog” as:
Text-Before This is hello from rsyslog Text-After
We will see how to use the template for generate the log files dynamically.
Central Logging Server
The above sections should have given an overview about rsyslog and how to configure it. Now we will move on to setup a central logging system.
For our discussion we will have server IP as “192.168.1.1” for the central log server, where all the log messages from client should be forwarded.
Add the following lines to the rsyslog.conf of the central log server servers (In this example, the following line was added on the log server with ip-address 192.168.1.1):
# provides support for local system logging $ModLoad imuxsock # provides kernel logging support (previously done by rklogd) $ModLoad imklog # provides UDP syslog reception. For TCP, load imtcp. $ModLoad imudp # For TCP, InputServerRun 514 $UDPServerRun 514 # This one is the template to generate the log filename dynamically, depending on the client's IP address. $template FILENAME,"/var/log/%fromhost-ip%/syslog.log" # Log all messages to the dynamically formed file. Now each clients log (192.168.1.2, 192.168.1.3,etc...), will be under a separate directory which is formed by the template FILENAME. *.* ?FILENAME
After adding the above lines to the rsyslog.conf, restart the rsyslog process. Now the rsyslog server will be ready to accept messages.
# service rsyslog restart Shutting down system logger: [ OK ] Starting system logger: [ OK ]
Add the following lines to the rsyslog.conf on the individual client machines that should send their log messages to the central server.
$ModLoad imuxsock $ModLoad imklog # Provides UDP forwarding. The IP is the server's IP address *.* @192.168.1.1:514 # Provides TCP forwarding. But the current server runs on UDP # *.* @@192.168.1.1:514
Restart the rsyslog process on the clients. Now the rsyslog central server (In this example, 192.168.1.1) will receive all the log messages from the configured clients and each client’s log will be placed under a separate directory.
My name is Ramesh Natarajan. I will be posting instruction guides, how-to, troubleshooting tips and tricks on Linux, database, hardware, security and web. My focus is to write articles that will either teach you or help you resolve a problem. Read more about
Comments on this entry are closed.
Nice simple tutorial. How about a quick test to confirm the system is working properly? Every project should do this.
nice write up. adding the iptables part to open the port would make it more easy for beginners.
Thanks jef and ion for your comments. I will try to address ur points from my next article
nice tutorial. it’s possible to send the ip address of client via logger command?
@roberto,
In logger command you can use the -t option ( used to tag ). So from client you can use logger -t , and I think this should work.
By chance, do you know how I would configure rsyslog to forward an actual log file?
Send me Linux Upadates
Solution to capture commands executed by all the users in Linux is here.
# For TCP, InputServerRun 514
I think this is incorrect, should be InputTCPServerRun 514.
Nice succinct article. Helped me out. Thanks!
good source to start with .thanks bhai
Hi.
My question: What if the rsyslog server goes down? Does it make the other computers hang?
Thanks
@Luis,
I think the messages will be spooled to disk and it will send once the server comes back , if its specified in configuration file,
from /etc/rsyslog.conf
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
Thanks @Aby
Another question? Does the rsyslog mechanism introduce a lot of network traffic??
Thanks
Little advice :
Before the path of your log file you can add “-” to disable sync to disk. Usefull if you have lot of logs.
See :
http://www.rsyslog.com/doc/rsyslog_conf_actions.html
“You may prefix each entry with the minus “-” sign to omit syncing the file after every logging. Note that you might lose information if the system crashes right behind a write attempt. Nevertheless this might give you back some performance, especially if you run programs that use logging in a very verbose manner.
If your system is connected to a reliable UPS and you receive lots of log data (e.g. firewall logs), it might be a very good idea to turn of syncing by specifying the “-” in front of the file name.”
You can manage the buffer with this directives :
http://www.rsyslog.com/doc/omfile.html
IOBufferSize
FlushInterval
ASyncWriting
FlushOnTXEnd
Nice tutoria, thanksl! Unfortunately it wanted more work for me to make it run.
If it does not work for you. Here is a troubleshooting:
1) in the client try to ping the central server to assure that connection really works
1) in the central server create a folder for client messages under /var/log/
2) in the central server create a file /var/log//rsyslog.log
3) in the central server run chmod and change rights (i.e. 777) for /var/log//rsyslog.log
4) in the central server make sure, your firewall does not block your messages. In Ubuntu there is used “ufw” deamon. For test purpose try to disable it by “sudo service ufw stop”
If problem persists, run “cat var/log/syslog” to see what is wrong in the central server.
to my previous comment no.16, because editor saved id badly. The IP of client is missing there. Correct is:
1) in the central server create a folder for client messages under /var/log/client_ip
2) in the central server create a file /var/log/client_ip/rsyslog.log
3) in the central server run chmod and change rights (i.e. 777) for /var/log/client_ip/rsyslog.log
Works great.
Only my /var/log/messages is also filled with all client logs, now / is almost full.
Now I’ve the same log files on two places.
Is it possible to use /var/log/messages only for the log server it self and that the clients archive their logging via the template location(on another file system /opt/data)?
I want to send only certain logs from /var/log/messages on client server to central rsys log server. So can I define any conditions to send only message like “UserAllowed” from client server to central rsys log server?.
Is there any parameter like “grep” I can define in configuration file?
Thanks!
Hi Lakshmanan,
Nice article, I am beginner for syslog. We are using log4j for application logs, currently we are storing logs in loc. Can you please let me know how we can point log4j log files to remote host?
Thanks in advance
hi please let me know that , if you configure the syslog , waht happen to the local log files
Nice intro, thanks for sharing
Hello,
I am trying to change rsyslog server conf file , now i want rsyslog server to reload configuration so that there is no need to restart the rsyslog server
Hello,
I am using above configuration. Logs are directed to /var/log/messages correctly. But when i am using following configuration, the client ip is not detecting. It is creating directory with %fromhost-ip% this name. What to do to create directory with IP address?
Configuration used-
$template FILENAME,”/var/log/%fromhost-ip%/syslog.log”
# Log all messages to the dynamically formed file. Now each clients log (192.168.1.2, 192.168.1.3,etc…), will be under a separate directory which is formed by the template FILENAME.
*.* ?FILENAME
Very concise, pragmatic and helpful, thank you!
Thank you!
Hi,
First of all i want to thank you for this great source.
I noticed this,
# For TCP, InputServerRun 514
$UDPServerRun 514
Is it comment true?