≡ Menu

8 Powerful Awk Built-in Variables – FS, OFS, RS, ORS, NR, NF, FILENAME, FNR

Linux Awk Tutorial - Awk Built-in Variables ExamplesThis article is part of the on-going Awk Tutorial Examples series. Awk has several powerful built-in variables. There are two types of built-in variables in Awk.

  1. Variable which defines values which can be changed such as field separator and record separator.
  2. Variable which can be used for processing and reports such as Number of records, number of fields.

1. Awk FS Example: Input field separator variable.

Awk reads and parses each line from input based on whitespace character by default and set the variables $1,$2 and etc. Awk FS variable is used to set the field separator for each record. Awk FS can be set to any single character or regular expression. You can use input field separator using one of the following two options:

  1. Using -F command line option.
  2. Awk FS can be set like normal variable.
Syntax:

$ awk -F 'FS' 'commands' inputfilename

(or)

$ awk 'BEGIN{FS="FS";}'
  • Awk FS is any single character or regular expression which you want to use as a input field separator.
  • Awk FS can be changed any number of times, it retains its values until it is explicitly changed. If you want to change the field separator, its better to change before you read the line. So that change affects the line what you read.

Here is an awk FS example to read the /etc/passwd file which has “:” as field delimiter.

$ cat etc_passwd.awk
BEGIN{
FS=":";
print "Name\tUserID\tGroupID\tHomeDirectory";
}
{
	print $1"\t"$3"\t"$4"\t"$6;
}
END {
	print NR,"Records Processed";
}
$awk -f etc_passwd.awk /etc/passwd
Name    UserID  GroupID        HomeDirectory
gnats	41	41	/var/lib/gnats
libuuid	100	101	/var/lib/libuuid
syslog	101	102	/home/syslog
hplip	103	7	/var/run/hplip
avahi	105	111	/var/run/avahi-daemon
saned	110	116	/home/saned
pulse	111	117	/var/run/pulse
gdm	112	119	/var/lib/gdm
8 Records Processed

2. Awk OFS Example: Output Field Separator Variable

Awk OFS is an output equivalent of awk FS variable. By default awk OFS is a single space character. Following is an awk OFS example.

$ awk -F':' '{print $3,$4;}' /etc/passwd
41 41
100 101
101 102
103 7
105 111
110 116
111 117
112 119

Concatenator in the print statement “,” concatenates two parameters with a space which is the value of awk OFS by default. So, Awk OFS value will be inserted between fields in the output as shown below.

$ awk -F':' 'BEGIN{OFS="=";} {print $3,$4;}' /etc/passwd
41=41
100=101
101=102
103=7
105=111
110=116
111=117
112=119

3. Awk RS Example: Record Separator variable

Awk RS defines a line. Awk reads line by line by default.

Let us take students marks are stored in a file, each records are separated by double new line, and each fields are separated by a new line character.

$cat student.txt
Jones
2143
78
84
77

Gondrol
2321
56
58
45

RinRao
2122
38
37
65

Edwin
2537
78
67
45

Dayan
2415
30
47
20

Now the below Awk script prints the Student name and Rollno from the above input file.

$cat student.awk
BEGIN {
	RS="\n\n";
	FS="\n";

}
{
	print $1,$2;
}

$ awk -f student.awk  student.txt
Jones 2143
Gondrol 2321
RinRao 2122
Edwin 2537
Dayan 2415

In the script student.awk, it reads each student detail as a single record,because awk RS has been assigned to double new line character and each line in a record is a field, since FS is newline character.

4. Awk ORS Example: Output Record Separator Variable

Awk ORS is an Output equivalent of RS. Each record in the output will be printed with this delimiter. Following is an awk ORS example:

$  awk 'BEGIN{ORS="=";} {print;}' student-marks
Jones 2143 78 84 77=Gondrol 2321 56 58 45=RinRao 2122 38 37 65=Edwin 2537 78 67 45=Dayan 2415 30 47 20=

In the above script,each records in the file student-marks file is delimited by the character “=”.

5. Awk NR Example: Number of Records Variable

Awk NR gives you the total number of records being processed or line number. In the following awk NR example, NR variable has line number, in the END section awk NR tells you the total number of records in a file.

$ awk '{print "Processing Record - ",NR;}END {print NR, "Students Records are processed";}' student-marks
Processing Record -  1
Processing Record -  2
Processing Record -  3
Processing Record -  4
Processing Record -  5
5 Students Records are processed

6. Awk NF Example: Number of Fields in a record

Awk NF gives you the total number of fields in a record. Awk NF will be very useful for validating whether all the fields are exist in a record.

Let us take in the student-marks file, Test3 score is missing for to students as shown below.

$cat student-marks
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122 38 37
Edwin 2537 78 67 45
Dayan 2415 30 47

The following Awk script, prints Record(line) number, and number of fields in that record. So It will be very simple to find out that Test3 score is missing.

$ awk '{print NR,"->",NF}' student-marks
1 -> 5
2 -> 5
3 -> 4
4 -> 5
5 -> 4

7. Awk FILENAME Example: Name of the current input file

FILENAME variable gives the name of the file being read. Awk can accept number of input files to process.

$ awk '{print FILENAME}' student-marks
student-marks
student-marks
student-marks
student-marks
student-marks

In the above example, it prints the FILENAME i.e student-marks for each record of the input file.

8. Awk FNR Example: Number of Records relative to the current input file

When awk reads from the multiple input file, awk NR variable will give the total number of records relative to all the input file. Awk FNR will give you number of records for each input file.

$ awk '{print FILENAME, FNR;}' student-marks bookdetails
student-marks 1
student-marks 2
student-marks 3
student-marks 4
student-marks 5
bookdetails 1
bookdetails 2
bookdetails 3
bookdetails 4
bookdetails 5

In the above example, instead of awk FNR, if you use awk NR, for the file bookdetails the you will get from 6 to 10 for each record.

Recommended Reading

Sed and Awk 101 Hacks, by Ramesh Natarajan. I spend several hours a day on UNIX / Linux environment dealing with text files (data, config, and log files). I use Sed and Awk for all my my text manipulation work. Based on my Sed and Awk experience, I’ve written Sed and Awk 101 Hacks eBook that contains 101 practical examples on various advanced features of Sed and Awk that will enhance your UNIX / Linux life. Even if you’ve been using Sed and Awk for several years and have not read this book, please do yourself a favor and read this book. You’ll be amazed with the capabilities of Sed and Awk utilities.

Add your comment

If you enjoyed this article, you might also like..

  1. 50 Linux Sysadmin Tutorials
  2. 50 Most Frequently Used Linux Commands (With Examples)
  3. Top 25 Best Linux Performance Monitoring and Debugging Tools
  4. Mommy, I found it! – 15 Practical Linux Find Command Examples
  5. Linux 101 Hacks 2nd Edition eBook Linux 101 Hacks Book

Bash 101 Hacks Book Sed and Awk 101 Hacks Book Nagios Core 3 Book Vim 101 Hacks Book

Comments on this entry are closed.

  • Harsh January 29, 2010, 12:23 pm

    Thanks for posting good article.

  • Guru Prasad March 10, 2010, 10:36 am

    Hi Ramesh
    I tried the example 3..but it gives the file as is in the output..not as expected..is there anything else i need to do….

  • frankhuang November 18, 2010, 8:59 pm

    to Guru
    because there is an error in example 3
    line
    FS”\n”;
    should be
    FS=”\n”;

  • arthur December 13, 2010, 9:07 am

    example 3 is still not producing the desired the output.The output produced is the same as the input

  • Ramesh Natarajan January 14, 2011, 10:56 pm

    @Frank Huang,

    Thanks for pointing it out. I’ve corrected it. Just FYI. For me both FS”\n” and FS=”\n” worked. I’m using GNU awk.

    $ awk --version | head -1
    GNU Awk 3.1.7
    

    @Guru, @Arthur,

    Please try again with FS=”\n” and see if it works again. If not, can you let me know what version of Awk you are using?

  • Anonymous April 7, 2011, 4:56 am

    The Best description about Built-in Variables of awk.

  • SAID AL_SARIRI April 11, 2011, 4:23 am

    In breif this useful web for anyone went to study, so i am intresting in its subjects , specially lunix

  • Colin June 24, 2011, 3:12 am

    I have the following which works fine on AIX but does not on Linux using Gawk

    echo “a:b:c” | awk ‘{FS=”:”; print $2}’

    On AIX it produces
    b
    as expected

    On Linux it produces a blank line
    ???

    I have a 100+ line awk script which makes a lot of use of FS so I need to be able to change it from : to a space to a % within the script rather than simply use -F

  • fhefh November 16, 2011, 9:17 pm

    you blog is good
    thanks

  • Hemanth November 22, 2011, 8:08 am

    Good aricle
    Helped me a lot 🙂

  • Manu December 17, 2011, 11:46 pm

    Ramesh,
    Still ex:3 is not giving expected result(shown in your example).The op is each filed is still in a new line with a space after end of field (before \n on each line).
    My env is AIX 6.1 (not sure of awk version) awk –version|head -1 did not give any result.I have even tried with nawk but op did not change.

  • Jim April 5, 2012, 8:37 am

    Can you please help me solve this problem. These statements run fine on their own but I’d like to combine them into one. I can’t seem to get it to work.

    # First statement
    egrep -w ‘Deny TCP|Deny UDP’ $FW_LOG | awk ‘{print $1 ” ” $2 ” ” $3}’ >> $OUTFILE

    #Output
    TCP 109.75.171.98 in
    TCP 210.128.108.48 in

    ===================

    # Second statement
    echo 109.75.171.98 | geo
    echo 210.128.108.48 | geo

    #Output
    Japan
    United Kingdom

    What I’d like to do is combine the 2 commands above so the output looks like this –

    TCP 109.75.171.98 in Japan
    TCP 210.128.108.48 in United Kingdom

    this is what I have so far –

    egrep -w ‘Deny TCP’ IPs.txt | awk ‘{cmd=”geo “$2;cmd | getline rslt;close(cmd);print $2” “rslt}’ >> $OUTPUT

    also, the way geo script works is that is accepts input – for example
    echo 111.222.333.444 | geo

    doesn’t seem to work like geo 111.222.333.444 ..so maybe why it doesn’t work.

  • Anonymous April 9, 2012, 4:30 pm

    Jim,

    the “awk” command has built-in pattern matching similar to “grep”, so you should be able to do what you want with this command line: (JETS)

    awk ‘/Deny (UD|TC)P/{print $1,$2,$3}’ $FW_LOG >> $OUTFILE

  • Ider April 18, 2012, 10:52 pm

    In sample 3, the RS=”\n\n” does not work on my Mac terminal. Seems awk only retrieve the first character as Record Separator.

    However, if I set RS=”” (empty string). It works.
    I test with other cases, seems RS=”” would make awk to treat multiple newline mark as separator, kind like “\n{2,}”. Though RS does not support regex.

    Maybe it is because the version of awk?

  • vineesh April 27, 2012, 3:15 am

    Hello Guys,

    can you please let me know meaning of awk(‘/Started in/{print $NF} ) server.log

    log entry is like Started in 09m:03sec:20ms

    I want to know here what is meaning o

    Thanks,
    Vineesh

  • muthu July 27, 2012, 12:07 pm

    @Vineesh
    I believe you would have found your answer by now 😛 anways..
    it just searches for the string “Started in” and when matches is found prints the last column value.

    i love this website 🙂

  • sankar August 23, 2012, 12:33 am

    colin,

    this is syntex for linux

    echo “a:b:c” | awk -F”:” ‘{ print $2}’

    or

    echo “a:b:c” | awk ‘BEGIN {FS=”:”};{ print $2}’

  • Kaushik November 6, 2012, 1:23 am

    Hi,

    What does this mean?

    gsub(/[/^/~]/,” “,instring);

  • Anonymous January 17, 2013, 5:53 am

    Hi,

    I am new to unix and can someone please explain me what the below command does ?
    day=`cal $month $year | awk ‘NF != 0{ last = $0 }; END{ print last }’ | awk ‘{ print $NF }’`

  • shweta February 10, 2013, 11:50 am

    Hi,

    In example 3, why we r using FS=”\n” without using this ans is still same…….
    i think RS=”\n\n” is sufficient….., pls clear my confusion ……..

    thanx in advance 🙂

  • alan February 21, 2013, 9:17 pm

    Jim,
    an old but interesting question. You didn’t provide input but it looks like $1 would be Deny so I started at $2, and I don’t have geo but you may have to put the full path if it’s another script.

    awk ‘/Deny (UD|TC)P/{printf “%s %s %s “,$2,$3,$4;cmd=”echo ” $3 “|./geo”;cmd|getline $result;print $result}’ $FW_LOG >> $OUTFILE

  • Siva April 9, 2013, 12:56 am

    @Shweta: While going through the above docu…I also came with the same doubt…
    NOTE: “Awk RS defines a line,Awk reads line by line by default”…

    I share my views with the below examples,

    =====
    Eg., 3.1

    cat student1.txt | awk ‘BEGIN{RS=”\n\n”;FS=”\n”}{print $1,$2;}END{print “Done”;}’
    Jones 2143
    Gondrol 2321
    RinRao 2122
    Edwin 2537
    Dayan 2415
    Done

    Eg., 3.2

    cat student1.txt | awk ‘BEGIN{RS=”\n\n”}{print $1,$2;}END{print “Done”;}’
    Jones 2143
    Gondrol 2321
    RinRao 2122
    Edwin 2537
    Dayan 2415
    Done

    Eg., 3.3

    cat student1.txt | awk ‘BEGIN{RS=”\n\n”;FS=”\n\n”}{print $1,$2;}END{print “Done”;}’
    Jones
    2143
    78
    84
    77
    Gondrol
    2321
    56
    58
    45
    RinRao
    2122
    38
    37
    65
    Edwin
    2537
    78
    67
    45
    Dayan
    2415
    30
    47
    20
    Done
    ======
    I also tried with FS=”\n\n\n\n”, n number of times using \n inside FS, but got the same output as Eg., 3.3 only.

    Records are always seperated by “\n” new line characters,obviously every line in a file is a record, including blank lines.This is default.

    When FS is a single character, then the newline character always serves as a field separator, in addition to whatever value FS may have. Leading and trailing newlines in a file are ignored.

    Hence according to above example 3.1,3.2,3.3, when we use FS=”\n” or not, we are getting the desired output( as expected).

    Cheers,

  • Gourav May 4, 2013, 4:44 pm

    Hi ,
    Its regarding example 3 .As we have mentioned FS=”\n” and RS=”\n\n”.the newline character of RS is also considered as FS character and is treated as $1,$2
    i.e
    a
    b

    c
    d
    here there are two \n after and b so these are also counted as fields by FS.So finally $1 =a,$2=b,$3 and $4 are null,$5=c and $6=d.

    I think it clarifies why we are not getting desired output.

  • Roy July 16, 2013, 6:39 am

    Hi,
    I want to separate the fields in a file using ‘~’
    I am using FS=”‘~'”, but its not working..

  • Blake October 17, 2013, 8:56 pm

    This is actually very useful!!
    Many thanks!!

  • esrreddy October 25, 2013, 1:00 am

    Good pages to know about AWK

  • shiva November 11, 2013, 5:20 am

    Hi All,
    I have two files A and B files
    >cat A
    empid empname deptid
    1 bond 10
    2 james 11
    > cat B
    deptid deptname
    10 IT
    11 HR

    From the above two files, i want the 3rd file C which should contain the output like below :-
    >cat C
    empid empname deptname
    1 bond IT
    2 james HR

    Please help me how to get the bove output in unix shell scripting

  • Joseph February 2, 2014, 5:32 pm

    I have 2 files of the same data, 2nd file is more recent with some values changed.
    File one contains
    City,174,533,1,11,0,1,0,48,m,,1200174568,1200176241
    City,627,34,1,ronaldo,0,1777,0,0,m,primus,1200119084,1200120025

    File two contains
    City,14,153,1,11,0,1,0,485,m,,1200174568,1200176241
    City,627,34,1,ronaldo,0,1777,0,0,m,primus,1200119084,1200120025
    City,748,332,1,janana,0,1811,0,14,m,,1200169533,1200171129

    The files are csv, the fields that are not updated are $0, $9, $11, $12, the other can change, what I want to do is compare the 2 files, search file 2 for $11, and then look in file 1 to see if it is there and then if any of the data in $1, $2, $3, $4, $5, $6, $7, $8, or $10 have changed then output that to another file. Also if file 2 has a record that does not show up in file 1 then print that to the output file also.

    So the output would look like this
    City,14,153,1,11,0,1,0,485,m,,1200174568,1200176241
    City,748,332,1,janana,0,1811,0,14,m,,1200169533,1200171129

    seeing how in line 1, both $1, $2, and $8 changed
    line 2 is a new entry not in file 1

    I hope this makes sense, and thanks in advance for any help someone can shed on this for me, its driving me bonkers.

  • Karn Gusain June 16, 2014, 10:23 am

    this is very nice post. could you please explain the below awk code, i am never been a programmer .

    [root@spacelab1 ~]# fuser -cu /apps01
    /apps01: 859ce(oracle) 882ce(oracle) 1156ce(oracle) 2847ce(oracle) 2882ce(oracle) 3034ce(oracle) 3290ce(oracle) 3560ce(oracle) 6104ce(oracle)
    7663ce(oracle) 8260ce(oracle) 8312ce(oracle) 8547ce(oracle) 8549ce(oracle) 8551ce(oracle) 8555ce(oracle) 8557ce(oracle) 8559ce(oracle) 8561ce(oracle) 8563ce

    [root@spacelab1 ~]# fuser -cu /apps01 2>/dev/null | awk ‘{ for (i=1; i<=NF; i++) print $i }' | more
    859
    882
    1156
    3034
    3290
    3560
    6104

    Thanks Much!

    -Karn

  • Sams George June 17, 2014, 2:27 am

    Dear All, sample 3, the RS=”\n\n” does not work on my AIX box.

    Regards,
    Sams

  • aditya manoj July 18, 2014, 2:40 am

    Dear Shiva,
    This is the solution for your query. This is may not be the apt solution. But i put my efforts to get it. This is working fine.
    grep -i “bond” a | awk ‘BEGIN {print “empid”,”empname”}{print $1,”\t”,$2}’ >> aa
    cat aa
    empid empname
    1 bond

    grep -i “it” b | awk ‘BEGIN {print “DEPTNAME”}{print $2}’ >>bb
    cat bb
    DEPTNAME
    IT

    awk ‘NR==FNR{_[NR]=$0;next}{print $1,$2,_[FNR]}’ bb aa
    empid empname DEPTNAME
    1 bond IT

  • aditya manoj July 18, 2014, 2:42 am

    Thank you shiva for posting such a query. This query is similar to the pl/sql query.

  • Prabhu August 7, 2014, 12:37 am

    how to delete a particular line from file using AWK….

    For ex: My input is given below ( I want to delete a particular line) in my case line no:3 will be delete
    Sample Input:

    “Unix is multitasking and multiuser system
    Unix is open source
    Unix contain lot of commands
    Unix is my favourite subject”

    so I want to get below output

    “Unix is multitasking and multiuser system
    Unix is open source
    Unix is my favourite subject”

  • Tan Tin Wee January 10, 2015, 1:02 pm

    An awk one liner can do the job for shiva.

    awk ‘BEGIN {while (getline < "b") {arr[$1]=$2}} { print $1 ,$2, arr[$3] }' a
    empid empname deptname
    1 bond IT
    2 james HR
    3 john RnD

    First read file b into an awk associative array arr [$1]=$2 in the BEGIN block. This becomes your look-up table for the main program which prints from file a, $1 $2 and arr [$3] instead of just $3.
    awk can open another file b and read each line using getline and populate the array arr before reading from file a. You can add close ("b") in the BEGIN block for good programming practice.

  • Midul February 16, 2016, 1:03 am

    Hi,
    I have a CDR file with fields separated by ‘|’.Each line is a single cdr. some cdr has has 54 fields and some have 41 fields.
    i want to get only the cdrs with 41 fields.
    I can count the lines with number of filed value with below command but can not find the lines themselves. Can you please help
    zcat push.cdr.20160216.105825.gz|awk -F ‘|’ ‘{print NF}’|sort -r |uniq -c

  • Tan Tin Wee February 17, 2016, 1:09 am

    zcat x.gz | awk -F’|’ ‘NF==41′
    -F’|’ will reset the field separator FS variable in to fields delimited by the pipe.
    Then, use NF==41 as the awk pattern/condition to print on by default.
    To print specific fields you can use
    zcat x.gz | awk -F’|’ ‘NF==41 { print $1 $2 }’ # where $1 and $2 are the fields you want printed.
    Awk will read the unzipped file from the output of the zcat command and split the lines one by one with the pipe as delimiter and assign the number of fields to NF, automatically. Whenever the condition NF==41 is met, it will print the output line by default. If you specify the print command, with specific field variables, in this case the first and second variable $1 and $2 respectively, columns one and two will be printed.

  • Gopal February 23, 2016, 4:10 am

    Example 3 is working for me as below given…

    # cat student.awk
    BEGIN {
    RS = “”;
    FS = “\n”;

    }
    {
    print $1,$2;
    }

    #awk -f student.awk student.txt
    Jones 2143
    Gondrol 2321
    RinRao 2122
    Edwin 2537
    Dayan 2415

    here

  • Priyanka Singh March 31, 2017, 4:51 am

    Hands down the best article on awk variables I’ve come across…Keep up the good work !!!