Unix Sed Tutorial: Multi-Line File Operation with 6 Practical Examples

by Sasikala on November 23, 2009

Linux Sed Examples for Multiple File Line OperationsAs part of our on going UNIX sed tutorial series earlier we covered the printing, deletion, substitution, file write, file manipulation commands etc., with the single line in the pattern space.

In this article let us review how to do the multi-line operation in Sed.

Do you remember the Sed working methodology which we learned in our first sed tutorial ?. In that article we explained that Sed reads line by line, removes any trailing new lines, places a line in a pattern space buffer, process as per the given commands and prints the pattern space.

In case, if you want to delete all the newlines in a file, you cannot use the following method. Because newline is already removed and placed in the pattern space.

$ sed 's/\n//' filename or  $sed 's/\n/ENDOFLINE\n/' filename

For situations like this sed multi-line is appropriate. Sed provides the command “N” for Multi-Line Operations.

N command reads a next line from the input, Append next line to pattern space. Next line is separated from the original pattern space by a newline character.

Let us first create thegeekstuff.txt file that will be used in all the examples mentioned below.

$ cat thegeekstuff.txt
Linux Sysadmin
Databases - Oracle, mySQL etc.
Databases - Oracle, mySQL etc.
Security (Firewall, Network, Online Security etc)


Storage in Linux
Website Design
Website Design
Windows- Sysadmin, reboot etc.
$

Note: There are two consecutive blank lines in the above input. ( 5th and 6th line ).

Sed Example 1. Join Two Consecutive Lines

$ sed -e '{
N
s/\n/ @ /
}' thegeekstuff.txt

Linux Sysadmin @ Databases - Oracle, mySQL etc.
Databases - Oracle, mySQL etc. @ Security (Firewall, Network, Online Security etc)
 @
Storage in Linux @ Website Design
Website Design @ Windows- Sysadmin, reboot etc.

In the above example,

  • The curly braces “{” and “}” used to group the commands. The curly braces and sed commands must be on the seperate lines.
  • Sed reads the first line and place it in the pattern space, N command reads the next line and appends with the pattern space i.e first line seperated by newline. So now pattern space will have firstline\nsecondline.
  • Next substitution of \n to space@space and it prints the pattern space content as its sed default behaviour. So consecutive lines are joined and delimited by ” @ “

Sed Example 2. Number each non-blank line of a file

As mentioned in our previous article, = is a command to get a line number of a file.

$ sed '/./=' thegeekstuff.txt | sed 'N; s/\n/ /'
1 Linux Sysadmin
2 Databases - Oracle, mySQL etc.
3 Databases - Oracle, mySQL etc.
4 Security (Firewall, Network, Online Security etc)

7 Storage in Linux
8 Website Design
9 Website Design
10 Windows- Sysadmin, reboot etc.
  • The first sed command prints the line number and prints the original line in a next line if it is not a blank.( Execute it and see the output of the first sed command ).
  • Next sed command is just appends pair of lines.

Sed Example 3. Delete Two Consecutive Blank Lines from Input

$ sed '/^$/{
N
/^\n$/d
}' thegeekstuff.txt
Linux Sysadmin
Databases - Oracle, mySQL etc.
Databases - Oracle, mySQL etc.
Security (Firewall, Network, Online Security etc)
Storage in Linux
Website Design
Website Design
Windows- Sysadmin, reboot etc.

If the line is blank, read and appends the next line, /^\n$/ represents, two lines are empty,\n is added by N command. Then just delete the pattern space and start the next cycle using command ‘d’.

Sed Example 4. Delete Last 2 Lines of a file

Before viewing this example you must aware of two interesting sed command.

  1. P – which prints the first line of a pattern space. (till first \n).
  2. D – Delete first line from the pattern space. Control then passes to the top of the script.
$ sed 'N;$!P;$!D;$d' thegeekstuff.txt
Linux Sysadmin
Databases - Oracle, mySQL etc.
Databases - Oracle, mySQL etc.
Security (Firewall, Network, Online Security etc)

Storage in Linux
Website Design
  1. Reads the first line and place it in the pattern space.
  2. N command reads the next line and append to pattern space seperated by newline. (Now firstline\nsecond line)
  3. If it not reaches the last line($), print the first line and delete the first line alone from the pattern space. Then cycle starts again.
  4. Like this it joins 2nd\n3rd lines, 3rd\n4th lines and goes on.
  5. Atlast when it has 9th\n10th line in a pattern space, it reaches $ so it just deletes the pattern space. ($!P and $!D wont print and delete if it is $).

Sed Example 5. Print Last 2 Lines of a file

$ sed '$!N;$!D' thegeekstuff.txt
Website Design
Windows- Sysadmin, reboot etc.

The above joins and deletes each line until last two lines are there in the pattern space. When it reaches $, prints the pattern space which will have only last two lines.

Sed Example 6. Delete Duplicate, Consecutive Lines from a file

The below command checks each line joined with the next line, check if both are same then it doesn’t the print pattern space(!P), just delete the first line from the pattern space. So only one line will be remaining in the pattern space.

$ sed '$!N; /^\(.*\)\n\1$/!P; D' thegeekstuff.txt
Linux Sysadmin
Databases - Oracle, mySQL etc.
Security (Firewall, Network, Online Security etc)

Storage in Linux
Website Design
Windows- Sysadmin, reboot etc.

Linux Sysadmin Course Linux provides several powerful administrative tools and utilities which will help you to manage your systems effectively. If you don’t know what these tools are and how to use them, you could be spending lot of time trying to perform even the basic administrative tasks. The focus of this course is to help you understand system administration tools, which will help you to become an effective Linux system administrator.
Get the Linux Sysadmin Course Now!

If you enjoyed this article, you might also like..

  1. 50 Linux Sysadmin Tutorials
  2. 50 Most Frequently Used Linux Commands (With Examples)
  3. Top 25 Best Linux Performance Monitoring and Debugging Tools
  4. Mommy, I found it! – 15 Practical Linux Find Command Examples
  5. Linux 101 Hacks 2nd Edition eBook Linux 101 Hacks Book

Bash 101 Hacks Book Sed and Awk 101 Hacks Book Nagios Core 3 Book Vim 101 Hacks Book

{ 10 comments… read them below or add one }

1 Berry November 25, 2009 at 8:41 am

In Example 2, I think the command provided in the article only works when the number of consecutive blank lines is even. Because in the left part, sed ‘/./=’ only print the line number before the non-blank line, the blank line is not changed. But in the right part, join every two lines (both non-blank and blank) into one line and substitute the \n with space. Then, if the number of consecutive blank lines in the original file is odd, that the last blank line will join the line number of next line.

For example:
aaa
blank
bbb

After the left part:
1
aaa
blank
3
bbb

So, I think the solution can be found in the left part
sed ‘/./=’ thegeekstuff.txt | sed ‘/./{
N
s/\n/ /
}’

2 Sasikala November 25, 2009 at 10:28 pm

Yes Berry, you are absolutely correct.

3 Masood January 25, 2010 at 5:12 pm

In Example 4 ,if total number of lines in sample script are even including blank lines then I think we can also use following sed command
sed ‘N;$d’ thegeekstuff.txt

4 Masood January 25, 2010 at 5:31 pm

In Example 5,the output is only last line and not last two lines.Can you confirm plz?

5 Indraneel Nag September 23, 2010 at 4:45 am

How can you make multiline text file to one single line text file

6 frankhuang November 17, 2010 at 7:58 pm

example 5 works because the last line is empty line, and
sed -n ‘$!h;$H;$x;$p’ thegeekstuff.txt
also works to Print Last 2 Lines of a file

7 Greg December 27, 2011 at 5:45 pm

I got example #6 working in a windows batch file as a multi-line find/replace. Many other sed multi-line examples I found online just didn’t work in all cases. But a modified example #6 like the following works:

sed –text -n -e “s|2011|&|p” -e “$^!N;/Directory.*\n.*2011/P; D” input.txt > output.txt

This sed line, running under the windows ver. of sed, finds either lines with “2011″ or lines with “Directory” only if “2011″ is on the next line, and the output is redirected to a separate file. The –text option helps generate the proper CR/LF for windows. Also, the exclamation char has to be “escaped” in a batch file *IF* enabledelayedexpansion is being used in context.

8 Jeff October 4, 2012 at 1:58 pm

Thanks for sharing.

Alternatively, I solved example 4 this way: sed ‘N; $ d’ thegeekstuff.txt, do you think there is something wrong with it? Thanks.

BTW: similarly, I did example 5 this way: sed -n ‘N; $ p’ thegeekstuff.txt.

9 Tony December 10, 2012 at 11:20 am

Jeff, your solutions didn’t work for me. Both of your commands just printed out the whole text file with nothing deleted.

When I did THIS, it did remove just the last line (not the last 2 lines):
sed ‘$d; N’ thegeekstuff.txt

10 sanjeev January 14, 2014 at 2:27 pm

sed -e :a -e ‘$!N;s/\n=/ /;ta’ -e ‘P;D’
This line it looks if the next line starts with “=” and merges it to the previous line.
: can this command be modified so that it looks if the next line contains “=” and merges it to the previous line.

Leave a Comment

Previous post:

Next post: