10 Awesome Examples for Viewing Huge Log Files in Unix

by Ramesh Natarajan on August 12, 2009

View Log Files in LinuxViewing huge log files for trouble shooting is a mundane routine tasks for sysadmins and programmers.

In this article, let us review how to effectively view and manipulate huge log files using 10 awesome examples.

Example 1: Display specific lines (based on line number) of a file using sed command

View only the specific lines mentioned by line numbers.

Syntax: $ sed -n -e Xp -e Yp FILENAME
  • sed : sed command, which will print all the lines by default.
  • -n : Suppresses output.
  • -e CMD : Command to be executed
  • Xp: Print line number X
  • Yp: Print line number Y
  • FILENAME : name of the file to be processed.

The example mentioned below will print the lines 120, 145, 1050 from the syslog.

$ sed -n -e 120p -e 145p -e 1050p /var/log/syslog

In the following example,  you can view the content of var/log/cron from line number 101 to 110.

  • M – Starting line number
  • N – Ending line number
Syntax: sed -n M,Np FILENAME

$ sed -n 101,110p /var/log/cron

Example 2: Display first N lines of a file using head command

This example displays only first 15 lines of /var/log/maillog file. Change 15 to 10 to display the first 10 lines of a log file.

Syntax: head -n N FILENAME

$ head -n 15 /var/log/maillog

Example 3: Ignore last N lines of a file using head command

This example shows how to ignore the last N lines, and show only the remaining lines from the top of file.

The following example will display all the lines of the /var/log/secure except the last 250 lines.

Syntax: head -n -N FILENAME

$ head -n -250 /var/log/secure

Example 4: Display last N lines of the file using tail command

This example displays only last 50 lines of /var/log/messages file. Change 50 to 100 to display the last 100 lines of the log file.

Syntax: tail -n N FILENAME

$ tail -n 50 /var/log/messages

Example 5: Ignore first N-1 lines of the file using tail command

This example shows how to ignore the first N-1 lines and show only the remaining of the lines.

The following example ignores the 1st four lines of the /etc/xinetd.conf, which contains only the comments.

Syntax: tail -n +N FILENAME

$ tail -n +5 /etc/xinetd.conf
defaults
{
        instances               = 60
        log_type                = SYSLOG authpriv
        log_on_success          = HOST PID
        log_on_failure          = HOST
        cps                     = 25 30
}
includedir /etc/xinetd.d

Example 6: View growing log file in real time using tail command

This is probably one of the most used command by sysadmins.To view a growing log file and see only the newer contents use tail -f as shown below.

The following example shows the content of the /var/log/syslog command in real-time.

Syntax: tail -f FILENAME

$ tail -f /var/log/syslog

Example 7: Display specific lines (based on line number) of a file using head and tail command

The example below will display line numbers 101 – 110 of /var/log/anaconda.log file

  • M – Starting line number
  • N – Ending line number
Syntax: cat file | tail -n +N | head -n (M-N+1)

$ cat /var/log/anaconda.log | tail -n +101 | head -n 10
  • cat : prints the whole file to the stdout.
  • tail -n +101 : ignores lines upto the given line number, and then start printing lines after the given number.
  • head -n 10 : prints the first 10 line, that is 101 to 110 and ignores the remaining lines.

Example 8: Display lines matching a pattern, and few lines following the match.

The following example displays the line that matches “Initializing CPU” from the /var/log/dmesg and 5 lines immediately after this match.

# grep "Initializing CPU#1" /var/log/dmesg
Initializing CPU#1
[Note: The above shows only the line matching the pattern]

# grep -A 5 "Initializing CPU#1" dmesg
Initializing CPU#1
Calibrating delay using timer specific routine.. 3989.96 BogoMIPS (lpj=1994982)
CPU: After generic identify, caps: bfebfbff 20100000 00000000 00000000
CPU: After vendor identify, caps:  bfebfbff 20100000 00000000 00000000
monitor/mwait feature present.
CPU: L1 I cache: 32K, L1 D cache: 32K
[Note: The above shows the line and 5 lines after the pattern matching]

Refer our earlier article Get a Grip on the Grep! – 15 Practical Grep Command Examples that explains how to use grep command.

As explained in our previous grep command article, the following operations are possible.

  • Viewing specific lines identified by patterns, which is grep’s default functionality.
  • Viewing only the matched characters.
  • Viewing N lines after the match with -A option.
  • Viewing N lines before the match with -B option.
  • Viewing N lines around the match with -C option.

Example 9: Displaying specific bytes from a file.

The following example explains how to display either the top 40 or the last 30 bytes of a file.

Display first 40 bytes from syslog.

$ head -c40 /var/log/syslog

Display last 30 bytes from syslog.

$ tail -c30 /var/log/syslog

Example 10: Viewing compressed log files

After a specific time all the system log files are rotated, and compressed. You can uncompress it on the fly, and pipe the output to another unix command to view the file as explained below.

Refer to our earlier article The Power of Z Commands – Zcat, Zless, Zgrep, Zdiff Examples

  • Display the first N lines of a compressed file.
    $ zcat file.gz | head -250
  • Display the last N lines of a compressed file.
    $ zcat file.gz | tail -250
  • Ignoring the last N lines of a compressed file.
    $ zcat file.gz | head -n -250
  • Ignoring the first N lines of a compressed file.
    $ zcat file.gz | tail -n +250
  • Viewing the lines matching the pattern
    $ zcat file.gz | grep -A2 'error'
  • Viewing particular range of lines identified by line number.
    $ zcat file.gz | sed -n -e 45p -e 52p

If you need to return, bookmark this page at del.icio.us for handy reference.


Linux Sysadmin Course Linux provides several powerful administrative tools and utilities which will help you to manage your systems effectively. If you don’t know what these tools are and how to use them, you could be spending lot of time trying to perform even the basic administrative tasks. The focus of this course is to help you understand system administration tools, which will help you to become an effective Linux system administrator.
Get the Linux Sysadmin Course Now!

If you enjoyed this article, you might also like..

  1. 50 Linux Sysadmin Tutorials
  2. 50 Most Frequently Used Linux Commands (With Examples)
  3. Top 25 Best Linux Performance Monitoring and Debugging Tools
  4. Mommy, I found it! – 15 Practical Linux Find Command Examples
  5. Linux 101 Hacks 2nd Edition eBook Linux 101 Hacks Book

Bash 101 Hacks Book Sed and Awk 101 Hacks Book Nagios Core 3 Book Vim 101 Hacks Book

{ 21 comments… read them below or add one }

1 Robert Fisher August 12, 2009 at 1:22 am

I have used example 8 many times looking for a response to a specific error. I have also used the tail and head commands as well, never really got into sed so much, but that example was a good one. Thanks for posting this information.

2 Nico August 12, 2009 at 2:23 am

FYI: Later versions of less(1) also displays .gz files.

3 Nico August 12, 2009 at 2:26 am

O – and before I forget… You can combine some of the solutions, for example combine #6 and #8: $ tail -f /some/file | grep keyword

This will give you only the lines containing the keyword as it gets logged.

4 Nicholas Sterling August 12, 2009 at 2:40 am

How about

sed -n 300,350p foo.log

Edit: oops — how embarrassing. You covered it and I missed it somehow.

5 Ashish August 12, 2009 at 5:12 am

Great post,
very useful article.

6 Ronald August 12, 2009 at 8:45 am

The “sed -n 101,110p /var/log/cron” approach is really useful. Although I have to say that I wouldn’t have come up with that command when looking at the sed-manual…. :(

Sed:
-n, –quiet, –silent
suppress automatic printing of pattern space

p Print the current pattern space.

7 Koen De Jaeger August 13, 2009 at 9:27 am

Very nice!

8 Ramesh Natarajan August 14, 2009 at 8:56 am

@Robert,

Sed can do lot of amazing things. But it is hard to remember those on top of your head. I have close to 100 sed command that solves some common problem in a notebook, which I copy/paste whenever nedded.

Example 1 of this article is in that list. :)

@Nico,

Thanks for pointing that out. Most of us forget that grep can be combined with “tail -f”. This indeed can be very handy.

@Nicholas,

Don’t worry. I’ve done similar mistakes several times. So, I can related to it. :-)

@Ashish, @Koen

I’m very glad that you found this article helpful.

@Ronald,

Yeah. That is the beauty of Unix. We can do amazing things with linux commands that were not even clearly mentioned in the manual. This is also the reason Windows admins don’t like Linux, as they hate reading manuals. Even when they read the manuals, it is not really explained very well.

9 ShekarKCB August 24, 2009 at 5:13 am

Nice Article, uses of head, tail, and sed were nicely covered. Thanks for the same.

10 Ted September 13, 2009 at 12:06 am

The sed tip I will use for sure and I also like the tail -f |grep. Thank you Ramesh.

Any chance you will be posting your most useful sed commands from your note book at some time?

11 Bipin May 6, 2010 at 5:49 am

Great stuffs. Thanks for sharing.

12 Brad July 14, 2010 at 8:49 am

You showed zcat, but there is also bzcat and lzcat, and probably a few others. so if you have a file that is bzip2′d or lzma’d, then you can use bzcat and lzcat respectively to display them. Also, it can be piped to the less command so it can be scrolled across the terminal.

13 anton August 6, 2010 at 3:24 am

Is the following any better (i.e. more efficient) than zcat logfile.gz | tail ???

# emulate ztail ???

zless +F logfile.gz

14 logan February 21, 2011 at 12:11 am

or you can always use

cat logfile | less

which lets you scroll through it

OR

cat logfile | less | grep “PATTERN”

which lets you scroll through matching entries
I search for sshd for my auth.log file and can see all and only the ssh server entries

15 ks March 2, 2011 at 5:45 pm

A nice and handy tutorial. A few comments about the above comment by @logan:

$ cat logfile | less
has the same effect as
$ less logfile
and hence the use of the pipe is redundant (‘$’ stands for the command prompt). Both lets you scroll through the logfile and search for pattern using ‘/’ and ‘?’ (similar to vim).

However, the second
$ cat logfile | less | grep “PATTERN”
does not let you scroll, but the following
$ grep “PATTERN” logfile | less
will — through only the the lines that has the matching pattern “PATTERN”.

16 hernan July 2, 2012 at 9:34 am

Example 7: the letters M,N are reversed. It should be

Syntax: cat file | tail -n +M | head -n (N-M+1)

17 shai March 7, 2013 at 4:18 pm

Hello,

How do I get all the lines between the first and last occurance of a string in a log file.

example: test.log contains following
grape
apple
banana
tomato
potato
apple
papaya
strawberry

want to get all the lines between first and last occurance of apple so the output is following:
apple
banana
tomato
potato
apple

18 Ankit September 16, 2013 at 10:31 am

Hi

Really helpful post. But how can we make changes in a large file or we can open a large file in edit mode??

19 Babu November 18, 2013 at 5:40 am

I have a requirement like getting all lines of a log file which were added 5 minutes before ?

20 Aby January 3, 2014 at 2:47 am

@Babu,
You can use regular expression with grep command,
grep “Jan 3 00:5[1-6]” /var/log/messages.
This will print log messages from Hrs 00:51 to 00:56 for January 3rd.

21 Anwsering_Shai November 23, 2014 at 6:05 am

MYFILE=test.log
ARR=(`grep -n apple “$MYFILE”`)
if [ ${#ARR[*]} -lt 2 ];then
echo “Error Need at least 2 ocurrences of ‘apple’”
exit 1
fi
FIRST=`echo ${ARR[0]} | cut -d: -f1`
LAST=`echo ${ARR[-1]} | cut -d: -f1`
echo sed -n ${FIRST},${LAST}p “$MYFILE”

Leave a Comment

Previous post:

Next post: