Awk Tutorial: Understand Awk Variables with 3 Practical Examples

by Sasikala on January 20, 2010

Linux Awk Tutorial - User-defined and Built-in Awk VariablesThis article is part of the on-going Awk Tutorial and Examples series. Like any other programming languages, Awk also has user defined variables and built-in variables.

In this article let us review how to define and use awk variables.

  • Awk variables should begin with the letter, followed by it can consist of alpha numeric characters or underscore.
  • Keywords cannot be used as a awk variable
  • Awk does not support variable declaration like other programming languages
  • Its always better to initialize awk variables in BEGIN section, which will be executed only once in the beginning.
  • There are no datatypes in Awk. Whether a awk variable is to be treated as a number or as a string depends on the context it is used in.

Now let us review few simple examples to learn how to use user-defined awk variables.

Awk Example 1: Billing for Books

In this example, the input file bookdetails.txt contains records with fields — item number, Book name, Quantity and Rate per book.

$ cat bookdetails.txt
1 Linux-programming 2 450
2 Advanced-Linux 3 300
3 Computer-Networks 4 400
4 OOAD&UML 3 450
5 Java2 5 200

Now the following Awk script, reads and processes the above bookdetails.txt file, and generates report that displays — rate of each book sold, and total amount for all the books sold.

So far we have seen Awk reads the commands from the command line, but Awk can also read the commands from the file using -f option.

Syntax:

$ awk -f script-filename inputfilename

Now our Awk script for billing calculation for books is given below.

$ cat book-calculation.awk
BEGIN {
	total=0;
}
{
	itemno=$1;
	book=$2;
	bookamount=$3*$4;
	total=total+bookamount;
	print itemno," ", book,"\t","$"bookamount;
}
END {
	print "Total Amount = $"total;
}

In the above script,

  • Awk BEGIN section initializes the variable total. itemno, total, book, bookamount are userdefined awk variables.
  • In the Awk Action section, Quantity*bookprice will be stored in a variable called bookamount. Each bookamount will be added with the total.
  • Finally in the Awk END section, total variable will have total amount.

Now execute the book-calculation.awk script to generate the report that displays each book rate and total amount as shown below.

$ awk -f book-calculation.awk bookdetails.txt
1   Linux-programming 	 $900
2   Advanced-Linux 	 $900
3   Computer-Networks 	 $1600
4   OOAD&UML 	 $1350
5   Java2 	 $1000
Total Amount = $5750

Awk Example 2. Student Mark Calculation

In this example, create an input file “student-marks.txt” with the following content — Student name, Roll Number, Test1 score, Test2 score and Test3 score.

$ cat student-marks.txt
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122 38 37 65
Edwin 2537 78 67 45
Dayan 2415 30 47 20

Now the following Awk script will calculate and generate the report to show the Average marks of each student, average of Test1, Test2 and Test3 scores.

$cat student.awk

BEGIN {
	test1=0;
	test2=0;
	test3=0;
	print "Name\tRollNo\t Average Score";

}
{
	total=$3+$4+$5;
	test1=test1+$3;
	test2=test2+$4;
	test3=test3+$5;
	print $1"\t"$2"\t",total/3;
}
END{
	print "Average of Test1="test1/NR;
	print "Average of Test2="test2/NR;
	print "Average of Test3="test3/NR;

}

In the above Awk script,

  • In the Awk BEGIN section all the awk variables are initialized to zero. test1, test2, test3 and total are user-defined awk variables.
  • In the Awk ACTION section, $3, $4, $5 are Test1, Test2 and Test3 scores respectively. total variable is the addition of 3 test scores for each student. The awk variable test1, test2 and test3 has the total scores of each corresponding test.
  • So in the Awk END section, dividing each test total by total number of records (i.e student) will give you the average score.  NR is an Awk built-in variable which gives total number of records in input.

Awk Example 3. HTML Report for Student Details

In the above two example, we have seen awk variable which has numbers as its values. This example shows awk script to generate the html report for the students name and their roll number.

$ cat string.awk
BEGIN{
title="AWK";
print "<html>\n<title>"title"</title><body bgcolor=\"#ffffff\">\n<table border=1><th  colspan=2 align=centre>Student Details</th>";

}
{
name=$1;
rollno=$2;
print "<tr><td>"name"</td><td>"rollno"</td></tr>";
   
}
END {
    print "</table></body>\n</html>";
}

Use the same student-marks.txt input file that we created in the above example.

$ awk -f string.awk student-marks.txt
<html>
<title>AWK</title><body bgcolor="#ffffff">
<table border=1><th  colspan=2 align=centre>Student Details</th>
<tr><td>Jones</td><td>2143</td></tr>
<tr><td>Gondrol</td><td>2321</td></tr>
<tr><td>RinRao</td><td>2122</td></tr>
<tr><td>Edwin</td><td>2537</td></tr>
<tr><td>Dayan</td><td>2415</td></tr>
</table></body>
</html>

We can store the above output, which gives the following html table. In the above script, variable called name and rollno are string variable, because it is used in string context.

Student Details
Jones 2143
Gondrol 2321
RinRao 2122
Edwin 2537
Dayan 2415

Recommended Reading

Sed and Awk 101 Hacks, by Ramesh Natarajan. I spend several hours a day on UNIX / Linux environment dealing with text files (data, config, and log files). I use Sed and Awk for all my my text manipulation work. Based on my Sed and Awk experience, I’ve written Sed and Awk 101 Hacks eBook that contains 101 practical examples on various advanced features of Sed and Awk that will enhance your UNIX / Linux life. Even if you’ve been using Sed and Awk for several years and have not read this book, please do yourself a favor and read this book. You’ll be amazed with the capabilities of Sed and Awk utilities.


Linux Sysadmin Course Linux provides several powerful administrative tools and utilities which will help you to manage your systems effectively. If you don’t know what these tools are and how to use them, you could be spending lot of time trying to perform even the basic administrative tasks. The focus of this course is to help you understand system administration tools, which will help you to become an effective Linux system administrator.
Get the Linux Sysadmin Course Now!

If you enjoyed this article, you might also like..

  1. 50 Linux Sysadmin Tutorials
  2. 50 Most Frequently Used Linux Commands (With Examples)
  3. Top 25 Best Linux Performance Monitoring and Debugging Tools
  4. Mommy, I found it! – 15 Practical Linux Find Command Examples
  5. Linux 101 Hacks 2nd Edition eBook Linux 101 Hacks Book

Bash 101 Hacks Book Sed and Awk 101 Hacks Book Nagios Core 3 Book Vim 101 Hacks Book

{ 23 comments… read them below or add one }

1 Rishabh January 20, 2010 at 4:31 am

hi,
Can the AWK command work on data that is comma separated or for that matter any other special character(say pipe)?

2 Chris F.A. Johnson January 20, 2010 at 1:29 pm

Anything can be used as the separator by assigning that value to the FS variable, either in the BEGIN block or on the command line:

awk -F\| ‘….’ ## assigment on the command line

awk ‘BEGIN {FS = “|”} …’ ## assigment in the BEGINblock

3 Rishabh January 20, 2010 at 10:01 pm

Thanks Chris.

4 S.RAGHU January 21, 2010 at 11:37 pm

You can even work with data files with no delimiters by using substring function.
Eg.
name=substring($0,1,18);
id=substring($0,19,5);
,etc.
where $0 refers to the whole line.

5 S.RAGHU January 21, 2010 at 11:38 pm

$0 refers to the whole line of input data.

6 Chris F.A. Johnson January 22, 2010 at 2:15 am

There is no substring function; the function is substr().

7 S.RAGHU January 22, 2010 at 3:15 am

Thanks Chris. Sorry for the slip up.

8 boo January 25, 2010 at 2:17 pm

awwww(k), people should really be using perl(*)

(*) Or other appropriate, but modern, interpreter. python, whatever pleases you.

9 how to sort a line using awk October 6, 2010 at 7:15 am

I want to sort and remove a huge data file with respect to delimiter.

10 Chris F.A. Johnson October 7, 2010 at 6:23 am

boo.
“If you have an awk script that you don’t want anyone to be able to understand, just rewrite it in perl”
— Ed Morton in comp.unix shell

11 Chris F.A. Johnson October 7, 2010 at 6:25 am

You don’t use awk to sort a file; use the corect tool, sort.

12 Prasanna N August 15, 2012 at 11:58 pm

Thanks a ton as usual! Learning a lot of cool stuff from the geek stuff :)

13 Ankit March 24, 2013 at 11:36 am

How to analyse data using multiple files in awk

14 Chris F.A. Johnson March 25, 2013 at 6:29 pm

Ankit, what data do you want to analyse and what information do you want from it?

15 Ankit March 27, 2013 at 2:50 am

No i’ve not got any specific project. I just want to try that out and hence asked for the same.

16 Anonymous April 1, 2013 at 11:52 am

what does awk -v do?

17 Chris F.A. Johnson April 1, 2013 at 3:29 pm

man awk:

-v var=val
Assign the value val to the variable var, before execution of
the program begins. Such variable values are available to the
BEGIN block of an AWK program.

18 daixtr August 6, 2013 at 10:45 am

i disagree with the commenter who said that one should right away use perl/python to solve their problems. I love perl/python but that commenter obviously has little or limited idea of what happens over the command-line which sysadmins interact with every day every minute across AIX, Linux, Solaris, Freebsd and etc. DO NOT right away solve and decide to use perl/python. Understand first what is available with awk/sed/m4/ex/ed then scale up to python/perl when needed..

19 Deiveegaraja Andaver September 30, 2013 at 12:19 am

Useful for me to know what is AWK..

20 Amit October 20, 2013 at 1:41 pm

Hi,

I have a question related to awk-sorting of data files that are located in different folders.

I have got multiple data files (>500 in number) all with the same name ‘file.dat’ but located under different folders (e.g. ~/amit/folder1/file.dat, ~/amit/folder2/file.dat, etc.). Each of these files are essentially one-columned and have the same number of entries (that is, rows). What I want to do is to collect all these files serially and create a new file ‘fileout.dat’ which has file.dat from folder 1 followed by file.dat from folder 2, etc. Had this been a small finite number of files, I could have simply used ‘cat ~/amit/folder/file.dat ~/amit/folder2/file.dat > fileout.dat’ but this is out of the question with more than 500 files to deal with.

Any ideas on now to get this done?

Cheers,
Amit

21 Anonymous November 26, 2013 at 1:17 am

Thanks a lot.. Very useful article

22 ajay January 8, 2014 at 12:21 am

Use
cat ~/amit/folder*/file.dat > fileout.dat

23 Aashish September 19, 2014 at 1:02 am

Use
find / -type f -name file.dat | xargs cat > fileout.dat
If you want to restrict your search use -mindepth -maxdepth option with a find command

Leave a Comment

Previous post:

Next post: