≡ Menu

Unix Sed Tutorial: Advanced Sed Substitution Examples

Linux Sed Examples - Advanced Find and Replace OperationThis article is part of the on-going Unix Sed Tips and Tricks series.

In our previous sed articles we learned — sed printing, sed deletion, sed substitute , sed file write, and sed multiple commands.

In this article, let us review some interesting workarounds with the “s” substitute command in sed with several practical examples.

I. Sed Substitution Delimiter

As we discussed in our previous post, we can use the different delimiters such as @ % | ; : in sed substitute command.

Let us first create path.txt file that will be used in all the examples mentioned below.

$ cat path.txt
/usr/kbos/bin:/usr/local/bin:/usr/jbin:/usr/bin:/usr/sas/bin
/usr/local/sbin:/sbin:/bin/:/usr/sbin:/usr/bin:/opt/omni/bin:
/opt/omni/lbin:/opt/omni/sbin:/root/bin

Example 1 – sed @ delimiter: Substitute /opt/omni/lbin to /opt/tools/bin

When you substitute a path name which has ‘/’, you can use @ as a delimiter instead of ‘/’. In the sed example below, in the last line of the input file, /opt/omni/lbin was changed to /opt/tools/bin.

$ sed 's@/opt/omni/lbin@/opt/tools/bin@g' path.txt
/usr/kbos/bin:/usr/local/bin:/usr/jbin/:/usr/bin:/usr/sas/bin
/usr/local/sbin:/sbin:/bin/:/usr/sbin:/usr/bin:/opt/omni/bin:
/opt/tools/bin:/opt/omni/sbin:/root/bin

Example 2 – sed / delimiter: Substitute /opt/omni/lbin to /opt/tools/bin

When you should use ‘/’ in path name related substitution, you have to escape ‘/’ in the substitution data as shown below. In this sed example, the delimiter ‘/’ was escaped in the REGEXP and REPLACEMENT part.

$ sed 's/\/opt\/omni\/lbin/\/opt\/tools\/bin/g' path.txt
/usr/kbos/bin:/usr/local/bin:/usr/jbin/:/usr/bin:/usr/sas/bin
/usr/local/sbin:/sbin:/bin/:/usr/sbin:/usr/bin:/opt/omni/bin:
/opt/tools/bin:/opt/omni/sbin:/root/bin

II. Sed ‘&’ Get Matched String

The precise part of an input line on which the Regular Expression matches is represented by &, which can then be used in the replacement part.

Example 1 – sed & Usage: Substitute /usr/bin/ to /usr/bin/local

$ sed 's@/usr/bin@&/local@g' path.txt
/usr/kbos/bin:/usr/local/bin:/usr/jbin/:/usr/bin/local:/usr/sas/bin
/usr/local/sbin:/sbin:/bin/:/usr/sbin:/usr/bin/local:/opt/omni/bin:
/opt/omni/lbin:/opt/omni/sbin:/root/bin

In the above example ‘&’ in the replacement part will replace with /usr/bin which is matched pattern and add it with /local. So in the output all the occurrance of /usr/bin will be replaced with /usr/bin/local

Example 2 – sed & Usage: Match the whole line

& replaces whatever matches with the given REGEXP.

$ sed 's@^.*$@<<<&>>>@g' path.txt
<<</usr/kbos/bin:/usr/local/bin:/usr/jbin/:/usr/bin:/usr/sas/bin>>>
<<</usr/local/sbin:/sbin:/bin/:/usr/sbin:/usr/bin:/opt/omni/bin:>>>
<<</opt/omni/lbin:/opt/omni/sbin:/root/bin>>>

In the above example regexp has “^.*$” which matches the whole line. Replacement part <<<&>>> writes the whole line with <<< and >>> in the beginning and end of the line respectively.

III. Grouping and Back-references in Sed

Grouping can be used in sed like normal regular expression. A group is opened with “\(” and closed with “\)”.Grouping can be used in combination with back-referencing.

Back-reference is the re-use of a part of a Regular Expression selected by grouping. Back-references in sed can be used in both a Regular Expression and in the replacement part of the substitute command.

Example 1: Get only the first path in each line

$ sed 's/\(\/[^:]*\).*/\1/g' path.txt
/usr/kbos/bin
/usr/local/sbin
/opt/omni/lbin

In the above example, \(\/[^:]*\) matches the path available before first : comes. \1 replaces the first matched group.

Example 2: Multigrouping

In the file path.txt change the order of field in the last line of the file.

$ sed '$s@\([^:]*\):\([^:]*\):\([^:]*\)@\3:\2:\1@g' path.txt
/usr/kbos/bin:/usr/local/bin:/usr/jbin:/usr/bin:/usr/sas/bin
/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/omni/bin:
/root/bin:/opt/omni/sbin:/opt/omni/lbin

In the above command $ specifies substitution to happen only for the last line.Output shows that the order of the path values in the last line has been reversed.

Example 3: Get the list of usernames in /etc/passwd file

This sed example displays only the first field from the /etc/passwd file.

$sed 's/\([^:]*\).*/\1/' /etc/passwd
root
bin
daemon
adm
lp
sync
shutdown

Example 4: Parenthesize first character of each word

This sed example prints the first character of every word in paranthesis.

$ echo "Welcome To The Geek Stuff" | sed 's/\(\b[A-Z]\)/\(\1\)/g'
(W)elcome (T)o (T)he (G)eek (S)tuff

Example 5: Commify the simple number.

Let us create file called numbers which has list of numbers. The below sed command example is used to commify the numbers till thousands.

$ cat  numbers
1234
12121
3434
123

$sed 's/\(^\|[^0-9.]\)\([0-9]\+\)\([0-9]\{3\}\)/\1\2,\3/g' numbers
1,234
12,121
3,434
123

If you enjoyed this article, you might also like..

  1. 50 Linux Sysadmin Tutorials
  2. 50 Most Frequently Used Linux Commands (With Examples)
  3. Top 25 Best Linux Performance Monitoring and Debugging Tools
  4. Mommy, I found it! – 15 Practical Linux Find Command Examples
  5. Linux 101 Hacks 2nd Edition eBook Linux 101 Hacks Book

Bash 101 Hacks Book Sed and Awk 101 Hacks Book Nagios Core 3 Book Vim 101 Hacks Book

{ 35 comments… add one }

  • Berry November 2, 2009, 11:23 pm

    Dear author,
    In the example 4, what is the meaning of \b escape character? and why the REGEXP can finish the task?

  • Sasikala November 3, 2009, 5:31 am

    @Berry,

    \b matches a word boundary.

    If you omit \b from example 4, it just parenthesize all the Uppercase letters.

    $ echo “Welcome To The Geek StuffPost” | sed ‘s/\([A-Z]\)/\(\1\)/g’
    (W)elcome (T)o (T)he (G)eek (S)tuff(P)ost

  • Berry November 7, 2009, 1:38 am

    @Sasikala,

    Oh, I got it. Thank you very much. ^_^

    Following your tips, I found that / can match the right. Thank you again!

  • Berry November 7, 2009, 1:41 am

    @Sasikala,

    Following your tips, I found that / plus “smaller than” can match the left boundary and / plus “larger than” can match the right. Thank you again!

  • Guru Prasad February 27, 2010, 1:37 am

    Hi,
    Regarding the example 4, when i try it in Red Hat linux, it works, but when i try in HP-UX or SunOS it does not. Any reason why so?

  • Guru Prasad February 27, 2010, 1:40 am

    Also, same is the case with example 5 as well…please let me know why

  • Anjum April 9, 2010, 11:17 am

    Example 5 may only work on Linux (I don’t see how) but does not work for other systems. The following snippet does work on Sun OS and am sure will work on most systems out there; including Linux:

    echo ‘1234
    12121
    3434
    123′ | sed ‘s/\([0-9]\{3\}\)$/,\1/g;s/^,//’

    Thanks,
    -Anjum.

  • Anjum April 9, 2010, 4:10 pm

    Example 4 reworked and tested on Solaris 9:

    echo “Welcome To The Geek Stuff” | sed ‘s/\<\([A-Z]\)/\(\1\)/g'

    Thanks,
    -Anjum.

  • Logan August 10, 2010, 4:35 pm

    A complete example of commify (add comma as thousands separator)

    $ cat numbers.txt
    1
    12
    123
    1234
    12345
    123456
    1234567
    12345678
    123456789
    1234567890
    1234567890.1234
    +1234567890.1234
    -1234567890.1234

    With the “-r” option, there is no need to escape the paraenthesis and curly brackets.

    The simple code below doesn’t work for numeric strings larger that 6.

    $ sed -r ‘s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g’ numbers.txt
    1
    12
    123
    1,234
    12,345
    123,456
    1234,567
    12345,678
    123456,789
    1234567,890
    1234567,890.1234
    +1234567,890.1234
    -1234567,890.1234

    This one works. (taken from question 4.14 of http://sed.sourceforge.net/sedfaq.html)
    $ sed -r ‘:a;s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g;ta’ numbers.txt
    1
    12
    123
    1,234
    12,345
    123,456
    1,234,567
    12,345,678
    123,456,789
    1,234,567,890
    1,234,567,890.1234
    +1,234,567,890.1234
    -1,234,567,890.1234

  • Anonymous December 1, 2010, 7:32 am

    Can you please explain this Substitution.
    sed -r ‘s/(^|[^0-9.])([0-9]+)([0-9]{3})/\1\2,\3/g’ numbers.txt

    Thanks,
    Rafi

  • siva November 14, 2011, 12:59 am

    I am not clear about .can you explain this Grouping and Back-references in Sed.

    Thanks in advance.

  • bharathi March 7, 2012, 12:08 am

    HAI
    Can u given one example for replacing current date with * or ** (single digit with * and double digit with **) in calender using sed command.
    thanks

  • Anonymous April 28, 2012, 2:32 am

    can you tell what is this “^” actually does ??

  • Bobby May 9, 2012, 8:02 am

    ‘^’ serves two purposes. When inside of [] it means “not these”, but outside of that it means “the start of input”. So, for example

    echo “why hello” | sed ‘s/hello/goodbye’
    would replace ‘hello’ with ‘goodbye’, but

    echo “why hello” | sed ‘s/^hello/goodbye’
    would not replace the ‘hello’ with ‘goodbye’ because it is not at the beginning of input.

    echo “why hello” | sed ‘s/[^z]ello/goodbye’
    Would replace hello with goodbye (because ‘hello’ does not start with a ‘z’)

  • phani June 18, 2012, 4:25 am

    hi ,
    Using Sed can we append/remove the new content in crontab file without disturbing old entries .

  • Ray July 11, 2012, 7:24 am

    I am trying to parse a header file and the array size and paste later.
    I want to change a line like the following :
    bool SetValues(int array1[30], int array2[30, int size)
    I want it to look like
    SetValues(&array1[0], &array2[0], 30).
    I have got most it, but i cannot figure out how to replace the string :”int size”
    with the string “30”
    I cannot figure out how to hold the pattern I want and use it to substitue later.

  • upalshah August 25, 2012, 5:42 am

    name marks grade
    abc 50 CB
    def 45 CC
    ghhi 55 CA
    jkl 85 A
    mno 75 BA
    pqr 77 BA
    stu 89 A

    my question is..
    add .00 after every mark value in mark column…
    how can I do with sed command??????
    is there any another command???

  • Richard M August 29, 2012, 7:07 am

    $ cat file.txt
    name marks grade
    abc 50 CB
    def 45 CC
    ghhi 55 CA
    jkl 85 A
    mno 75 BA
    pqr 77 BA
    stu 89 A
    $ cat file.txt|awk ‘{print $1, $2″.00″, $3}’
    name marks.00 grade
    abc 50.00 CB
    def 45.00 CC
    ghhi 55.00 CA
    jkl 85.00 A
    mno 75.00 BA
    pqr 77.00 BA
    stu 89.00 A
    $

  • John November 28, 2012, 12:31 pm

    Hi,
    I want to insert null character at particular position in every line of file.
    How can i do that?
    thanks in advance.

  • Anonymous February 4, 2013, 7:21 am

    Hi..
    How to change the second field of the /etc/shadow file to NP without touching other entries in a line

  • nadji February 22, 2013, 1:50 am

    thank you thegeekstuff.com every time you give us interesting informations

  • dude4linux February 23, 2013, 11:32 am

    I use the following sed command to add some additional aliases to .bashrc
    sed -i -e ‘/ls –color=auto/a \
    alias ll=”ls –color=auto -lh” \
    alias la=”ls –color=auto -lAh” \
    alias l=”ls –color=auto -CF”
    ‘ ~/.bashrc
    I would like to use single quotes ‘ in the added lines for cosmetic reasons, but none of the combinations of escape codes will work for me. I’ve tried \’, ”, ”’, and even tried exchanging ‘ and “. Any idea how to include a single quote in the added lines?

  • yusuf April 3, 2013, 1:33 am

    Hi,
    Could u pls explain the Example 5, how it works.

    $sed ‘s/\(^\|[^0-9.]\)\([0-9]\+\)\([0-9]\{3\}\)/\1\2,\3/g’ numbers

  • yusuf April 3, 2013, 11:09 pm

    Boss, pls explain example 5.

  • Sudershan April 15, 2013, 4:49 am

    ANYONE please do let me know , how to add the below information after 10th line in a file in linux using sed command

    ===================================================================
    SSLEngine on
    JkMountCopy On
    SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

    SSLCertificateFile “/opt/test/sys/Apache/conf/ssl.crt/server.crt”
    SSLCertificateKeyFile “/opt/test/sys/Apache/conf/ssl.key/server.key”

    SSLOptions +StdEnvVars

    SSLOptions +StdEnvVars

    CustomLog “/opt/test/sys/Apache/logs/ssl_request.log” “%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \”%r\” %b”

    ===================================================================

  • Logan Palanisamy April 15, 2013, 10:21 am

    Here is you do it.

    1.Save the contents you want to insert after the 10th line into a separate file.
    2. Use read option (“r”) to read the above file into the original file.

    The original file
    $ cat myfile.txt
    this is line 1
    this is line 2
    this is line 3
    this is line 4
    this is line 5
    this is line 6
    this is line 7
    this is line 8
    this is line 9
    this is line 10
    this is line 11
    this is line 12

    Save the contents to be inserted into new_file.txt
    $ cat new_file.txt
    =======================
    new line 1
    new line 2
    new line 3
    new line 4
    new line 5
    =======================

    Read the contents of the new_file into the original file, myfile.txt
    $ sed ’10r new_file.txt’ myfile.txt
    this is line 1
    this is line 2
    this is line 3
    this is line 4
    this is line 5
    this is line 6
    this is line 7
    this is line 8
    this is line 9
    this is line 10
    =======================
    new line 1
    new line 2
    new line 3
    new line 4
    new line 5
    =======================
    this is line 11
    this is line 12

  • Sudershan April 16, 2013, 12:25 am

    Thank you very much Logan Palanisamy ,,,, Thank you very much once again palanisamy…!! it is really working thank you very very much …!!

  • Sudershan April 16, 2013, 12:33 am

    Logan Palanisamy , but in the text i had to pass variables like $HOSTNAME $PWD when i place in the text issue the command sed -i ’10r text’ httpd.conf , it is saving as it is , but it is not replacing the value of $HOSTNAME and $PWD , please help me in this regard logan palanisamy

    # ServerAdmin webmaster@dummy-host.example.com
    # DocumentRoot /www/docs/dummy-host.example.com
    ServerName $HOSTNAME
    # ErrorLog logs/dummy-host.example.com-error_log
    # CustomLog logs/dummy-host.example.com-access_log common
    SSLEngine on
    JkMountCopy On
    SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

  • kushal June 12, 2013, 5:58 am

    would anybody explain the meaning of every field of example 1

  • Eason Yi September 11, 2013, 8:32 am

    Hi author,

    I have one comment regarding example 1 in section III. Grouping and Back-references in Sed).

    You wrote, “\(\/[^:]*\) matches the path available before first : comes.”, which is not correct.

    Explanation:

    Let’s get rid of “\(” and “\)” in the expression, which are used for grouping.

    For the remaining part, “\/[^:]*”, which means, match the strings that starting with “/”, and followed by any strings other than “:”, for zero or more times.

    So, your explanation should be corrected to, “\(\/[^:]*\) matches any path that does not includes a colon.”

    The first ocurrence of the path in the line is represented by “\1″, not “\(\/[^:]*\)”.

  • prasad November 12, 2013, 6:04 am

    please help me
    i have a one text file.
    cat sample.txt
    0
    0
    0
    0
    0
    0
    i want to replace 0 with name using sed command

  • prasad November 12, 2013, 7:43 am

    please help me …for this also

    i have a file with 500 telephone numbers..and i want to replace number with name=data.
    using sed command .

  • J November 22, 2013, 12:19 pm

    @ sasikala
    First, let me say your examples not just on sed are a fantastic starting point for those who are passionate to learn

    My question is example 5

    ———————————————————————
    $sed ‘s/\(^\|[^0-9.]\)\([0-9]\+\)\([0-9]\{3\}\)/\1\2,\3/g’ numbers
    ———————————————————————

    can you tell me what does the dot “.” after 9 in the pattern \(^\|[^0-9.]\) do?
    The pattern should match all chars which are not digits anchored at the beginnng and defines the first group as that.

    But it works even without the dot after 9 . Can you explain what does the dot after 9 do?

  • Bhaswar October 13, 2014, 4:34 am

    What is the output if we do this-

    echo “something” | sed -e ‘s/searched_string_name//’

  • Sergey November 19, 2014, 5:54 am

    Very nice way to avoid nice \/\/\/\/\/\/\/\/\/ fence-like constructions: use colon as separator
    sed ‘s:old:new:g’
    This way
    sed ‘s/\/opt\/omni\/lbin/\/opt\/tools\/bin/g’ path.txt
    Turns into
    sed ‘s:/opt/omni/lbin:/opt/tools/bin:g’ path.txt
    which is so much easier to read!

Leave a Comment