8 Powerful Awk Built-in Variables – FS, OFS, RS, ORS, NR, NF, FILENAME, FNR

by Sasikala on January 27, 2010

Linux Awk Tutorial - Awk Built-in Variables ExamplesThis article is part of the on-going Awk Tutorial Examples series. Awk has several powerful built-in variables. There are two types of built-in variables in Awk.

  1. Variable which defines values which can be changed such as field separator and record separator.
  2. Variable which can be used for processing and reports such as Number of records, number of fields.

1. Awk FS Example: Input field separator variable.

Awk reads and parses each line from input based on whitespace character by default and set the variables $1,$2 and etc. Awk FS variable is used to set the field separator for each record. Awk FS can be set to any single character or regular expression. You can use input field separator using one of the following two options:

  1. Using -F command line option.
  2. Awk FS can be set like normal variable.
Syntax:

$ awk -F 'FS' 'commands' inputfilename

(or)

$ awk 'BEGIN{FS="FS";}'
  • Awk FS is any single character or regular expression which you want to use as a input field separator.
  • Awk FS can be changed any number of times, it retains its values until it is explicitly changed. If you want to change the field separator, its better to change before you read the line. So that change affects the line what you read.

Here is an awk FS example to read the /etc/passwd file which has “:” as field delimiter.

$ cat etc_passwd.awk
BEGIN{
FS=":";
print "Name\tUserID\tGroupID\tHomeDirectory";
}
{
	print $1"\t"$3"\t"$4"\t"$6;
}
END {
	print NR,"Records Processed";
}
$awk -f etc_passwd.awk /etc/passwd
Name    UserID  GroupID        HomeDirectory
gnats	41	41	/var/lib/gnats
libuuid	100	101	/var/lib/libuuid
syslog	101	102	/home/syslog
hplip	103	7	/var/run/hplip
avahi	105	111	/var/run/avahi-daemon
saned	110	116	/home/saned
pulse	111	117	/var/run/pulse
gdm	112	119	/var/lib/gdm
8 Records Processed

2. Awk OFS Example: Output Field Separator Variable

Awk OFS is an output equivalent of awk FS variable. By default awk OFS is a single space character. Following is an awk OFS example.

$ awk -F':' '{print $3,$4;}' /etc/passwd
41 41
100 101
101 102
103 7
105 111
110 116
111 117
112 119

Concatenator in the print statement “,” concatenates two parameters with a space which is the value of awk OFS by default. So, Awk OFS value will be inserted between fields in the output as shown below.

$ awk -F':' 'BEGIN{OFS="=";} {print $3,$4;}' /etc/passwd
41=41
100=101
101=102
103=7
105=111
110=116
111=117
112=119

3. Awk RS Example: Record Separator variable

Awk RS defines a line. Awk reads line by line by default.

Let us take students marks are stored in a file, each records are separated by double new line, and each fields are separated by a new line character.

$cat student.txt
Jones
2143
78
84
77

Gondrol
2321
56
58
45

RinRao
2122
38
37
65

Edwin
2537
78
67
45

Dayan
2415
30
47
20

Now the below Awk script prints the Student name and Rollno from the above input file.

$cat student.awk
BEGIN {
	RS="\n\n";
	FS="\n";

}
{
	print $1,$2;
}

$ awk -f student.awk  student.txt
Jones 2143
Gondrol 2321
RinRao 2122
Edwin 2537
Dayan 2415

In the script student.awk, it reads each student detail as a single record,because awk RS has been assigned to double new line character and each line in a record is a field, since FS is newline character.

4. Awk ORS Example: Output Record Separator Variable

Awk ORS is an Output equivalent of RS. Each record in the output will be printed with this delimiter. Following is an awk ORS example:

$  awk 'BEGIN{ORS="=";} {print;}' student-marks
Jones 2143 78 84 77=Gondrol 2321 56 58 45=RinRao 2122 38 37 65=Edwin 2537 78 67 45=Dayan 2415 30 47 20=

In the above script,each records in the file student-marks file is delimited by the character “=”.

5. Awk NR Example: Number of Records Variable

Awk NR gives you the total number of records being processed or line number. In the following awk NR example, NR variable has line number, in the END section awk NR tells you the total number of records in a file.

$ awk '{print "Processing Record - ",NR;}END {print NR, "Students Records are processed";}' student-marks
Processing Record -  1
Processing Record -  2
Processing Record -  3
Processing Record -  4
Processing Record -  5
5 Students Records are processed

6. Awk NF Example: Number of Fields in a record

Awk NF gives you the total number of fields in a record. Awk NF will be very useful for validating whether all the fields are exist in a record.

Let us take in the student-marks file, Test3 score is missing for to students as shown below.

$cat student-marks
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122 38 37
Edwin 2537 78 67 45
Dayan 2415 30 47

The following Awk script, prints Record(line) number, and number of fields in that record. So It will be very simple to find out that Test3 score is missing.

$ awk '{print NR,"->",NF}' student-marks
1 -> 5
2 -> 5
3 -> 4
4 -> 5
5 -> 4

7. Awk FILENAME Example: Name of the current input file

FILENAME variable gives the name of the file being read. Awk can accept number of input files to process.

$ awk '{print FILENAME}' student-marks
student-marks
student-marks
student-marks
student-marks
student-marks

In the above example, it prints the FILENAME i.e student-marks for each record of the input file.

8. Awk FNR Example: Number of Records relative to the current input file

When awk reads from the multiple input file, awk NR variable will give the total number of records relative to all the input file. Awk FNR will give you number of records for each input file.

$ awk '{print FILENAME, FNR;}' student-marks bookdetails
student-marks 1
student-marks 2
student-marks 3
student-marks 4
student-marks 5
bookdetails 1
bookdetails 2
bookdetails 3
bookdetails 4
bookdetails 5

In the above example, instead of awk FNR, if you use awk NR, for the file bookdetails the you will get from 6 to 10 for each record.

Recommended Reading

Sed and Awk 101 Hacks, by Ramesh Natarajan. I spend several hours a day on UNIX / Linux environment dealing with text files (data, config, and log files). I use Sed and Awk for all my my text manipulation work. Based on my Sed and Awk experience, I’ve written Sed and Awk 101 Hacks eBook that contains 101 practical examples on various advanced features of Sed and Awk that will enhance your UNIX / Linux life. Even if you’ve been using Sed and Awk for several years and have not read this book, please do yourself a favor and read this book. You’ll be amazed with the capabilities of Sed and Awk utilities.


Linux Sysadmin Course Linux provides several powerful administrative tools and utilities which will help you to manage your systems effectively. If you don’t know what these tools are and how to use them, you could be spending lot of time trying to perform even the basic administrative tasks. The focus of this course is to help you understand system administration tools, which will help you to become an effective Linux system administrator.
Get the Linux Sysadmin Course Now!

If you enjoyed this article, you might also like..

  1. 50 Linux Sysadmin Tutorials
  2. 50 Most Frequently Used Linux Commands (With Examples)
  3. Top 25 Best Linux Performance Monitoring and Debugging Tools
  4. Mommy, I found it! – 15 Practical Linux Find Command Examples
  5. Linux 101 Hacks 2nd Edition eBook Linux 101 Hacks Book

Bash 101 Hacks Book Sed and Awk 101 Hacks Book Nagios Core 3 Book Vim 101 Hacks Book

{ 33 comments… read them below or add one }

1 Harsh January 29, 2010 at 12:23 pm

Thanks for posting good article.

2 Guru Prasad March 10, 2010 at 10:36 am

Hi Ramesh
I tried the example 3..but it gives the file as is in the output..not as expected..is there anything else i need to do….

3 frankhuang November 18, 2010 at 8:59 pm

to Guru
because there is an error in example 3
line
FS”\n”;
should be
FS=”\n”;

4 arthur December 13, 2010 at 9:07 am

example 3 is still not producing the desired the output.The output produced is the same as the input

5 Ramesh Natarajan January 14, 2011 at 10:56 pm

@Frank Huang,

Thanks for pointing it out. I’ve corrected it. Just FYI. For me both FS”\n” and FS=”\n” worked. I’m using GNU awk.

$ awk --version | head -1
GNU Awk 3.1.7

@Guru, @Arthur,

Please try again with FS=”\n” and see if it works again. If not, can you let me know what version of Awk you are using?

6 Anonymous April 7, 2011 at 4:56 am

The Best description about Built-in Variables of awk.

7 SAID AL_SARIRI April 11, 2011 at 4:23 am

In breif this useful web for anyone went to study, so i am intresting in its subjects , specially lunix

8 Colin June 24, 2011 at 3:12 am

I have the following which works fine on AIX but does not on Linux using Gawk

echo “a:b:c” | awk ‘{FS=”:”; print $2}’

On AIX it produces
b
as expected

On Linux it produces a blank line
???

I have a 100+ line awk script which makes a lot of use of FS so I need to be able to change it from : to a space to a % within the script rather than simply use -F

9 fhefh November 16, 2011 at 9:17 pm

you blog is good
thanks

10 Hemanth November 22, 2011 at 8:08 am

Good aricle
Helped me a lot :)

11 Manu December 17, 2011 at 11:46 pm

Ramesh,
Still ex:3 is not giving expected result(shown in your example).The op is each filed is still in a new line with a space after end of field (before \n on each line).
My env is AIX 6.1 (not sure of awk version) awk –version|head -1 did not give any result.I have even tried with nawk but op did not change.

12 Jim April 5, 2012 at 8:37 am

Can you please help me solve this problem. These statements run fine on their own but I’d like to combine them into one. I can’t seem to get it to work.

# First statement
egrep -w ‘Deny TCP|Deny UDP’ $FW_LOG | awk ‘{print $1 ” ” $2 ” ” $3}’ >> $OUTFILE

#Output
TCP 109.75.171.98 in
TCP 210.128.108.48 in

===================

# Second statement
echo 109.75.171.98 | geo
echo 210.128.108.48 | geo

#Output
Japan
United Kingdom

What I’d like to do is combine the 2 commands above so the output looks like this –

TCP 109.75.171.98 in Japan
TCP 210.128.108.48 in United Kingdom

this is what I have so far –

egrep -w ‘Deny TCP’ IPs.txt | awk ‘{cmd=”geo “$2;cmd | getline rslt;close(cmd);print $2″ “rslt}’ >> $OUTPUT

also, the way geo script works is that is accepts input – for example
echo 111.222.333.444 | geo

doesn’t seem to work like geo 111.222.333.444 ..so maybe why it doesn’t work.

13 Anonymous April 9, 2012 at 4:30 pm

Jim,

the “awk” command has built-in pattern matching similar to “grep”, so you should be able to do what you want with this command line: (JETS)

awk ‘/Deny (UD|TC)P/{print $1,$2,$3}’ $FW_LOG >> $OUTFILE

14 Ider April 18, 2012 at 10:52 pm

In sample 3, the RS=”\n\n” does not work on my Mac terminal. Seems awk only retrieve the first character as Record Separator.

However, if I set RS=”" (empty string). It works.
I test with other cases, seems RS=”" would make awk to treat multiple newline mark as separator, kind like “\n{2,}”. Though RS does not support regex.

Maybe it is because the version of awk?

15 vineesh April 27, 2012 at 3:15 am

Hello Guys,

can you please let me know meaning of awk(‘/Started in/{print $NF} ) server.log

log entry is like Started in 09m:03sec:20ms

I want to know here what is meaning o

Thanks,
Vineesh

16 muthu July 27, 2012 at 12:07 pm

@Vineesh
I believe you would have found your answer by now :P anways..
it just searches for the string “Started in” and when matches is found prints the last column value.

i love this website :)

17 sankar August 23, 2012 at 12:33 am

colin,

this is syntex for linux

echo “a:b:c” | awk -F”:” ‘{ print $2}’

or

echo “a:b:c” | awk ‘BEGIN {FS=”:”};{ print $2}’

18 Kaushik November 6, 2012 at 1:23 am

Hi,

What does this mean?

gsub(/[/^/~]/,” “,instring);

19 Anonymous January 17, 2013 at 5:53 am

Hi,

I am new to unix and can someone please explain me what the below command does ?
day=`cal $month $year | awk ‘NF != 0{ last = $0 }; END{ print last }’ | awk ‘{ print $NF }’`

20 shweta February 10, 2013 at 11:50 am

Hi,

In example 3, why we r using FS=”\n” without using this ans is still same…….
i think RS=”\n\n” is sufficient….., pls clear my confusion ……..

thanx in advance :)

21 alan February 21, 2013 at 9:17 pm

Jim,
an old but interesting question. You didn’t provide input but it looks like $1 would be Deny so I started at $2, and I don’t have geo but you may have to put the full path if it’s another script.

awk ‘/Deny (UD|TC)P/{printf “%s %s %s “,$2,$3,$4;cmd=”echo ” $3 “|./geo”;cmd|getline $result;print $result}’ $FW_LOG >> $OUTFILE

22 Siva April 9, 2013 at 12:56 am

@Shweta: While going through the above docu…I also came with the same doubt…
NOTE: “Awk RS defines a line,Awk reads line by line by default”…

I share my views with the below examples,

=====
Eg., 3.1

cat student1.txt | awk ‘BEGIN{RS=”\n\n”;FS=”\n”}{print $1,$2;}END{print “Done”;}’
Jones 2143
Gondrol 2321
RinRao 2122
Edwin 2537
Dayan 2415
Done

Eg., 3.2

cat student1.txt | awk ‘BEGIN{RS=”\n\n”}{print $1,$2;}END{print “Done”;}’
Jones 2143
Gondrol 2321
RinRao 2122
Edwin 2537
Dayan 2415
Done

Eg., 3.3

cat student1.txt | awk ‘BEGIN{RS=”\n\n”;FS=”\n\n”}{print $1,$2;}END{print “Done”;}’
Jones
2143
78
84
77
Gondrol
2321
56
58
45
RinRao
2122
38
37
65
Edwin
2537
78
67
45
Dayan
2415
30
47
20
Done
======
I also tried with FS=”\n\n\n\n”, n number of times using \n inside FS, but got the same output as Eg., 3.3 only.

Records are always seperated by “\n” new line characters,obviously every line in a file is a record, including blank lines.This is default.

When FS is a single character, then the newline character always serves as a field separator, in addition to whatever value FS may have. Leading and trailing newlines in a file are ignored.

Hence according to above example 3.1,3.2,3.3, when we use FS=”\n” or not, we are getting the desired output( as expected).

Cheers,

23 Gourav May 4, 2013 at 4:44 pm

Hi ,
Its regarding example 3 .As we have mentioned FS=”\n” and RS=”\n\n”.the newline character of RS is also considered as FS character and is treated as $1,$2
i.e
a
b

c
d
here there are two \n after and b so these are also counted as fields by FS.So finally $1 =a,$2=b,$3 and $4 are null,$5=c and $6=d.

I think it clarifies why we are not getting desired output.

24 Roy July 16, 2013 at 6:39 am

Hi,
I want to separate the fields in a file using ‘~’
I am using FS=”‘~’”, but its not working..

25 Blake October 17, 2013 at 8:56 pm

This is actually very useful!!
Many thanks!!

26 esrreddy October 25, 2013 at 1:00 am

Good pages to know about AWK

27 shiva November 11, 2013 at 5:20 am

Hi All,
I have two files A and B files
>cat A
empid empname deptid
1 bond 10
2 james 11
> cat B
deptid deptname
10 IT
11 HR

From the above two files, i want the 3rd file C which should contain the output like below :-
>cat C
empid empname deptname
1 bond IT
2 james HR

Please help me how to get the bove output in unix shell scripting

28 Joseph February 2, 2014 at 5:32 pm

I have 2 files of the same data, 2nd file is more recent with some values changed.
File one contains
City,174,533,1,11,0,1,0,48,m,,1200174568,1200176241
City,627,34,1,ronaldo,0,1777,0,0,m,primus,1200119084,1200120025

File two contains
City,14,153,1,11,0,1,0,485,m,,1200174568,1200176241
City,627,34,1,ronaldo,0,1777,0,0,m,primus,1200119084,1200120025
City,748,332,1,janana,0,1811,0,14,m,,1200169533,1200171129

The files are csv, the fields that are not updated are $0, $9, $11, $12, the other can change, what I want to do is compare the 2 files, search file 2 for $11, and then look in file 1 to see if it is there and then if any of the data in $1, $2, $3, $4, $5, $6, $7, $8, or $10 have changed then output that to another file. Also if file 2 has a record that does not show up in file 1 then print that to the output file also.

So the output would look like this
City,14,153,1,11,0,1,0,485,m,,1200174568,1200176241
City,748,332,1,janana,0,1811,0,14,m,,1200169533,1200171129

seeing how in line 1, both $1, $2, and $8 changed
line 2 is a new entry not in file 1

I hope this makes sense, and thanks in advance for any help someone can shed on this for me, its driving me bonkers.

29 Karn Gusain June 16, 2014 at 10:23 am

this is very nice post. could you please explain the below awk code, i am never been a programmer .

[root@spacelab1 ~]# fuser -cu /apps01
/apps01: 859ce(oracle) 882ce(oracle) 1156ce(oracle) 2847ce(oracle) 2882ce(oracle) 3034ce(oracle) 3290ce(oracle) 3560ce(oracle) 6104ce(oracle)
7663ce(oracle) 8260ce(oracle) 8312ce(oracle) 8547ce(oracle) 8549ce(oracle) 8551ce(oracle) 8555ce(oracle) 8557ce(oracle) 8559ce(oracle) 8561ce(oracle) 8563ce

[root@spacelab1 ~]# fuser -cu /apps01 2>/dev/null | awk ‘{ for (i=1; i<=NF; i++) print $i }' | more
859
882
1156
3034
3290
3560
6104

Thanks Much!

-Karn

30 Sams George June 17, 2014 at 2:27 am

Dear All, sample 3, the RS=”\n\n” does not work on my AIX box.

Regards,
Sams

31 aditya manoj July 18, 2014 at 2:40 am

Dear Shiva,
This is the solution for your query. This is may not be the apt solution. But i put my efforts to get it. This is working fine.
grep -i “bond” a | awk ‘BEGIN {print “empid”,”empname”}{print $1,”\t”,$2}’ >> aa
cat aa
empid empname
1 bond

grep -i “it” b | awk ‘BEGIN {print “DEPTNAME”}{print $2}’ >>bb
cat bb
DEPTNAME
IT

awk ‘NR==FNR{_[NR]=$0;next}{print $1,$2,_[FNR]}’ bb aa
empid empname DEPTNAME
1 bond IT

32 aditya manoj July 18, 2014 at 2:42 am

Thank you shiva for posting such a query. This query is similar to the pl/sql query.

33 Prabhu August 7, 2014 at 12:37 am

how to delete a particular line from file using AWK….

For ex: My input is given below ( I want to delete a particular line) in my case line no:3 will be delete
Sample Input:

“Unix is multitasking and multiuser system
Unix is open source
Unix contain lot of commands
Unix is my favourite subject”

so I want to get below output

“Unix is multitasking and multiuser system
Unix is open source
Unix is my favourite subject”

Leave a Comment

Previous post:

Next post: