Viewing huge log files for trouble shooting is a mundane routine tasks for sysadmins and programmers.
In this article, let us review how to effectively view and manipulate huge log files using 10 awesome examples.
Example 1: Display specific lines (based on line number) of a file using sed command
View only the specific lines mentioned by line numbers.
Syntax: $ sed -n -e Xp -e Yp FILENAME
- sed : sed command, which will print all the lines by default.
- -n : Suppresses output.
- -e CMD : Command to be executed
- Xp: Print line number X
- Yp: Print line number Y
- FILENAME : name of the file to be processed.
The example mentioned below will print the lines 120, 145, 1050 from the syslog.
$ sed -n -e 120p -e 145p -e 1050p /var/log/syslog
In the following example, you can view the content of var/log/cron from line number 101 to 110.
- M – Starting line number
- N – Ending line number
Syntax: sed -n M,Np FILENAME $ sed -n 101,110p /var/log/cron
Example 2: Display first N lines of a file using head command
This example displays only first 15 lines of /var/log/maillog file. Change 15 to 10 to display the first 10 lines of a log file.
Syntax: head -n N FILENAME $ head -n 15 /var/log/maillog
Example 3: Ignore last N lines of a file using head command
This example shows how to ignore the last N lines, and show only the remaining lines from the top of file.
The following example will display all the lines of the /var/log/secure except the last 250 lines.
Syntax: head -n -N FILENAME $ head -n -250 /var/log/secure
Example 4: Display last N lines of the file using tail command
This example displays only last 50 lines of /var/log/messages file. Change 50 to 100 to display the last 100 lines of the log file.
Syntax: tail -n N FILENAME $ tail -n 50 /var/log/messages
Example 5: Ignore first N-1 lines of the file using tail command
This example shows how to ignore the first N-1 lines and show only the remaining of the lines.
The following example ignores the 1st four lines of the /etc/xinetd.conf, which contains only the comments.
Syntax: tail -n +N FILENAME
$ tail -n +5 /etc/xinetd.conf
defaults
{
instances = 60
log_type = SYSLOG authpriv
log_on_success = HOST PID
log_on_failure = HOST
cps = 25 30
}
includedir /etc/xinetd.d
Example 6: View growing log file in real time using tail command
This is probably one of the most used command by sysadmins.To view a growing log file and see only the newer contents use tail -f as shown below.
The following example shows the content of the /var/log/syslog command in real-time.
Syntax: tail -f FILENAME $ tail -f /var/log/syslog
Example 7: Display specific lines (based on line number) of a file using head and tail command
The example below will display line numbers 101 – 110 of /var/log/anaconda.log file
- M – Starting line number
- N – Ending line number
Syntax: cat file | tail -n +N | head -n (M-N+1) $ cat /var/log/anaconda.log | tail -n +101 | head -n 10
- cat : prints the whole file to the stdout.
- tail -n +101 : ignores lines upto the given line number, and then start printing lines after the given number.
- head -n 10 : prints the first 10 line, that is 101 to 110 and ignores the remaining lines.
Example 8: Display lines matching a pattern, and few lines following the match.
The following example displays the line that matches “Initializing CPU” from the /var/log/dmesg and 5 lines immediately after this match.
# grep "Initializing CPU#1" /var/log/dmesg Initializing CPU#1 [Note: The above shows only the line matching the pattern] # grep -A 5 "Initializing CPU#1" dmesg Initializing CPU#1 Calibrating delay using timer specific routine.. 3989.96 BogoMIPS (lpj=1994982) CPU: After generic identify, caps: bfebfbff 20100000 00000000 00000000 CPU: After vendor identify, caps: bfebfbff 20100000 00000000 00000000 monitor/mwait feature present. CPU: L1 I cache: 32K, L1 D cache: 32K [Note: The above shows the line and 5 lines after the pattern matching]
Refer our earlier article Get a Grip on the Grep! – 15 Practical Grep Command Examples that explains how to use grep command.
As explained in our previous grep command article, the following operations are possible.
- Viewing specific lines identified by patterns, which is grep’s default functionality.
- Viewing only the matched characters.
- Viewing N lines after the match with -A option.
- Viewing N lines before the match with -B option.
- Viewing N lines around the match with -C option.
Example 9: Displaying specific bytes from a file.
The following example explains how to display either the top 40 or the last 30 bytes of a file.
Display first 40 bytes from syslog.
$ head -c40 /var/log/syslog
Display last 30 bytes from syslog.
$ tail -c30 /var/log/syslog
Example 10: Viewing compressed log files
After a specific time all the system log files are rotated, and compressed. You can uncompress it on the fly, and pipe the output to another unix command to view the file as explained below.
Refer to our earlier article The Power of Z Commands – Zcat, Zless, Zgrep, Zdiff Examples
- Display the first N lines of a compressed file.
$ zcat file.gz | head -250
- Display the last N lines of a compressed file.
$ zcat file.gz | tail -250
- Ignoring the last N lines of a compressed file.
$ zcat file.gz | head -n -250
- Ignoring the first N lines of a compressed file.
$ zcat file.gz | tail -n +250
- Viewing the lines matching the pattern
$ zcat file.gz | grep -A2 'error'
- Viewing particular range of lines identified by line number.
$ zcat file.gz | sed -n -e 45p -e 52p
If you need to return, bookmark this page at del.icio.us for handy reference.
My name is Ramesh Natarajan. I will be posting instruction guides, how-to, troubleshooting tips and tricks on Linux, database, hardware, security and web. My focus is to write articles that will either teach you or help you resolve a problem. Read more about
Comments on this entry are closed.
I have used example 8 many times looking for a response to a specific error. I have also used the tail and head commands as well, never really got into sed so much, but that example was a good one. Thanks for posting this information.
FYI: Later versions of less(1) also displays .gz files.
O – and before I forget… You can combine some of the solutions, for example combine #6 and #8: $ tail -f /some/file | grep keyword
This will give you only the lines containing the keyword as it gets logged.
How about
sed -n 300,350p foo.log
Edit: oops — how embarrassing. You covered it and I missed it somehow.
Great post,
very useful article.
The “sed -n 101,110p /var/log/cron” approach is really useful. Although I have to say that I wouldn’t have come up with that command when looking at the sed-manual…. 🙁
Sed:
-n, –quiet, –silent
suppress automatic printing of pattern space
p Print the current pattern space.
Very nice!
@Robert,
Sed can do lot of amazing things. But it is hard to remember those on top of your head. I have close to 100 sed command that solves some common problem in a notebook, which I copy/paste whenever nedded.
Example 1 of this article is in that list. 🙂
@Nico,
Thanks for pointing that out. Most of us forget that grep can be combined with “tail -f”. This indeed can be very handy.
@Nicholas,
Don’t worry. I’ve done similar mistakes several times. So, I can related to it. 🙂
@Ashish, @Koen
I’m very glad that you found this article helpful.
@Ronald,
Yeah. That is the beauty of Unix. We can do amazing things with linux commands that were not even clearly mentioned in the manual. This is also the reason Windows admins don’t like Linux, as they hate reading manuals. Even when they read the manuals, it is not really explained very well.
Nice Article, uses of head, tail, and sed were nicely covered. Thanks for the same.
The sed tip I will use for sure and I also like the tail -f |grep. Thank you Ramesh.
Any chance you will be posting your most useful sed commands from your note book at some time?
Great stuffs. Thanks for sharing.
You showed zcat, but there is also bzcat and lzcat, and probably a few others. so if you have a file that is bzip2’d or lzma’d, then you can use bzcat and lzcat respectively to display them. Also, it can be piped to the less command so it can be scrolled across the terminal.
Is the following any better (i.e. more efficient) than zcat logfile.gz | tail ???
# emulate ztail ???
zless +F logfile.gz
or you can always use
cat logfile | less
which lets you scroll through it
OR
cat logfile | less | grep “PATTERN”
which lets you scroll through matching entries
I search for sshd for my auth.log file and can see all and only the ssh server entries
A nice and handy tutorial. A few comments about the above comment by @logan:
$ cat logfile | less
has the same effect as
$ less logfile
and hence the use of the pipe is redundant (‘$’ stands for the command prompt). Both lets you scroll through the logfile and search for pattern using ‘/’ and ‘?’ (similar to vim).
However, the second
$ cat logfile | less | grep “PATTERN”
does not let you scroll, but the following
$ grep “PATTERN” logfile | less
will — through only the the lines that has the matching pattern “PATTERN”.
Example 7: the letters M,N are reversed. It should be
Syntax: cat file | tail -n +M | head -n (N-M+1)
Hello,
How do I get all the lines between the first and last occurance of a string in a log file.
example: test.log contains following
grape
apple
banana
tomato
potato
apple
papaya
strawberry
want to get all the lines between first and last occurance of apple so the output is following:
apple
banana
tomato
potato
apple
Hi
Really helpful post. But how can we make changes in a large file or we can open a large file in edit mode??
I have a requirement like getting all lines of a log file which were added 5 minutes before ?
@Babu,
You can use regular expression with grep command,
grep “Jan 3 00:5[1-6]” /var/log/messages.
This will print log messages from Hrs 00:51 to 00:56 for January 3rd.
MYFILE=test.log
ARR=(`grep -n apple “$MYFILE”`)
if [ ${#ARR[*]} -lt 2 ];then
echo “Error Need at least 2 ocurrences of ‘apple'”
exit 1
fi
FIRST=`echo ${ARR[0]} | cut -d: -f1`
LAST=`echo ${ARR[-1]} | cut -d: -f1`
echo sed -n ${FIRST},${LAST}p “$MYFILE”
Thank you. This helps
Can you help me to write shell script which will continuously poll the logfile for an error. If any error comes then will email that error.
am looking bash script for grep last 5 minites entry from log file , i have try to many script but ots not working , i think there have some date formate issue is there , can u plz help me .