Chapter 5: AWK

AWK
• The awk command was introduced in1977.
• It is named after itsauthor Aho, Weinberger & Kernighan.
• awk utility is powerful data manipulation/scripting
programming language.
• awk combines features of several filters.
• awk appears asgawk(GNU awk) in Linux.
• awk operates at field level. It can easily access, transform
andformat individual fields in a record.
• It also accepts regular expressions for pattern matching.

• The general syntax is

awk options ‘address{action} ‘ files
• Here, the address and action constitute an awk program. The
action is enclosed within the curly braces. Theseprograms are
mostly one line long, but they can also include several lines.

Filters
• A group of UNIX commands accepts some data as input
performs some manipulation on it and produces some
output. These groups of commands are called filters.
• The filters use standard input and standard output and can
be used with redirection and pipelines.
•eg. head ,tail, cut, paste, sort etc.
•grep, egrep
•eg. grep sales emp.lst
•egrep ‘(sen|das)gupta’ emp.lst

Simple awk filtering

• An awk command specifies an address and an action.
• To select directors from emp.lst use -
$ awk ‘/directors/{print}’ emp.lst # lists directors from
emp.lst
• The address section (/directors/) selects lines that are processed
inaction section ({print}).
• If the address is missing, the action applies to all lines of the file.
Ifthe action is missing, the entire line will be printed.
• Either the address or the action is optional, but both must be
enclosed within a pair of single (not double) quotes.
• The print statement, when used without any field specifiers,
prints the entire line. Printing is the default action of awk.

• $ awk ’/director/’ emp.lst

• $ awk ’/director/{print }’emp.lst
• $ awk ’/director/{print$0}’emp.lst
• All above commands are equivalent

• For pattern matching, awk uses regular expressions in sed style.

• $ awk -F “|” ’/sa[kx]s*ena/’ emp.lst

• It also allows multiple patterns.

The regular expressions used by awk belong to the egrep variety which
means that you can also use multiple patterns by delimiting each
pattern with a |
Demo file- stud

Rollno Name PCM USA IOM

101 Ameya 18 18 16

135 Bharat 16 16 15

145 Chaitanya 15 16 14

123 Deepak 16 16 16

125 Hari 17 16 15

156 Ganesha 17 17 16

159 Ratan 17 17 17

SPLITTING A LINE INTO FIELDS

• awk uses the special "variable" $0 to indicate the entire line.
• It also Iidentifies by $1, $2, $3,……..
• These parameters also have a special meaning to the shell.
• Single-quoting an awk program protects them from
interpretationby the shell.
• Unlike other UNIX filters which operate on fields, awk also uses
acontiguous sequence of spaces and tabs as a single delimiter.
• If the sample database uses the |, then -F option must be used
to specify it in programs.
• You can use awk to print the name, marks in all or some subjects :
• $ awk ‘ { print $1,$2 }’ stud

101 Ameya
135 Bharat
145 Chaitanya
123 Deepak
125 Hari
156 Ganesha
159 Ratan
Note: a, (comma) is used to delimit the field specifications. This
ensures that each field is separated from the other by a space. If
youdon't put the comma, the fields will be glued together.
• User can use awk with a line address to select lines.
• Use the built-in variable NR to specify the line numbers:
• $ awk 'NR == 3, NR == 5 { print $1,$2 }’ stud
145 Chaitanya
123 Deepak
125 Hari

• The statement NR == 3 is really a condition that is being

tested,rather than an assignment. NR is just one of those
built-in variables used in awk programs, and == is one of the
many operators employed in comparison tests.

awk ‘ { print $1,$2, $3+$4+$5 }’ stud

101 Ameya 52
135 Bharat 47
145 Chaitanya 45
123 Deepak 48
125 Hari 48
156 Ganesha 50
159 Ratan 51

$ awk ‘ { print $1,$2, $3,$4,$5,$3+$4+$5 }’ stud101

• Ameya 18 18 16 52
135 Bharat 16 16 15 47
145 Chaitanya 15 16 14 45
123 Deepak 16 16 16 48
125 Hari 17 16 15 48
156 Ganesha 17 17 16 50
159 Ratan 17 17 17 51
 printf: FORMATTING OUTPUT
• The previous output is unformatted, but with the C-like printf
statement, we can use awk as a stream formatter.
• awk accepts most of the formats used by the C printf function.
• The %s format will be used for string data and %d for numeric.

• We can produce a list of all the students -

• $ awk ‘ { printf "%5d %-15s %7d %7d %7d %7d\n",
$1,$2,$3,$4,$5, $3+$4+$5 }' stud

101 Ameya 18 18 16 52
135 Bharat 16 16 15 47
145 Chaitanya 15 16 14 45
123 Deepak 16 16 16 48
125 Hari 17 16 14 48
156 Ganesha 17 17 16 50
159 Ratan 17 17 17 51

To produce a list of all the agarwals:

 Like in sed, an awk command is considered complete only when the quote
is closed.
 The name and designation have been printed in spaces 20 and 12
characters wide, respectively; the – symbol left - justifies the output.
 The record number is three characters wide, .right-justified.
 printf requires an \n to print a newline after each line.
 Variables

 awk has certain built-in variables, like NR and $0, representing the line
number and the entire line respectively, it also permits the user to use
variables of her choice.

 A user-defined variable used by awk has two special features:

 No type declarations are needed.
 By default, variables are initialized to zero or a null string. awk has a
mechanism of identifying the type of variable used from its context

 You can now use these properties to print a serial number, using the
variable kount, and apply it to select those directors drawing a salary
exceeding 6700:

 The initial value of kount was 0 (by default).

 That's why the first r cord is correctly assigned the number 1.

 awk also accepts the C-style incrementing forms

Note: No type declarations or initial values are required for user-defined

variables used in an awk program. awk identifies their type and initializes them
to zero or null strings.
Built in variables
 awk has several built in variables.
 The values are assigned automatically.
 The user can reassign some of them.

• FS-Field Separator
For identification of fields, awk uses a contiguous string of spaces
as the default field delimiter.
If the database uses | as a field separator then it must occur in
BEGIN section so that the body of program knows its value before it
starts processing. (Blank or tab is default)
BEGIN { FS=“|” }
This is equivalent to –F option.
Note: awk is the only filter that uses white space as the default
delimiter instead of a single space.

• OFS - Output Field Separator

When print statement is used with comma separated
arguments,value of each argument is separated by a space in
output. (Blank is default) .
This is awk's default output field separator, and can be
reassigned using the variable OFS in the BEGIN section.
This can be assigned with new value as-
BEGIN {OFS=“~”}
When you reassign this variable with a ~ (tilde), awk will use this
character for delimiting the print arguments.
 • ORS -
Output record separator string (Default is new line)

• NF - Number of fields in a line

This variable can be used to check the number of fields
incurrent line.

• NR-
Cumulative Number of lines read

• FILENAME-
This stores the name of the current file being processed.
awk can also handle multiple filenames in the command line. By
default awk doesn't print the filename, but it can be instructed to do
so:

• ARGC-
Number of arguments in command line

• ARGV-
List of arguments

• RS –
Input record separator character (Default is new line)

• OFMT-
Output format of number
User Defined variables in awk

A user-defined variable used by awk has two special features:

No type declarations are needed.
By default, variables are initialized to zero or a null string. awk has a
mechanism of identifying the type of variable used from its context

You can also define your own variable in awk program, as follows:
$ cat > maths1
{
no1 = $1
no2 = $2
ans = $1 + $2
print no1 " + " no2 " = " ans
}

Run the program as follows

$ awk -f maths1
2 7
2+7=9

In the above program, no1, no2, ans all are user defined variables. Value of
first and second field are assigned to no1, no2 variable respectively and the
addition to ans variable.
The-f Option: Storing awk programs in a File-

 Large awk program should be stored in files.

 The extension should be preferably . awk for easier identificaton.
 Let’s store the previous program in the file empawk. awk.

 no quotes are used to enclose the awk program this time.

 awk with the -f filename option can also be used to obtain the same
output.

Caution: If you use awk with the -f option, don't use quotes to enclose the
program that is stored in the file. \
 Arithmetic in awk
• You can easily, do the arithmetic with awk as follows

• $ cat > maths

{
print $1 " + " $2 " = " $1+$2
print $1 " - " $2 " = " $1 - $2
print $1 " / " $2 " = " $1 / $2
print $1 " x " $2 " = " $1 * $2
print $1 " mod " $2 " = " $1 % $2
}

• Run the awk program as follows:

• $ awk -f maths
20 7
20 + 7 = 27
20 - 7 = 13
20 / 7 = 2.8572
20 x 7 = 140
20 mod 7 = 6
(Press CTRL + D to terminate)

Number Processing

 awk can perform computations on numbers using the arithmetic

operators +,-,*,/ and % (modulus).
 It overcomes one of the limitations of the shell – the inability to handle
the decimal numbers.

 In any environment maintaining data for the payroll application, it's

common practice to store only the basic pay in the file.
 This is acceptable because normally the other elements of pay can be
derived from this basic pay.
 For the purpose of using awk, the sixth field will now signify the basic
pay, rather than the salary.
 Assume that the total salary consists of basic pay, dearness allowance
(@40 %of basic pay) and house rent (@15 %of basic pay).
 Awks is used to print a rudimentary pay-slip for the directors:

LOGICAL AND RELATIONAL OPERATORS:

• awk supports various logical and relational operators.

• How do you print the three fields for the directors and the chairman?

Since the, designation field is $3, you have to use it in the address:

 Awk also uses the || and && logical operators.

• e.g

.
 $ awk '$1 == 125 || $1 == 135{
> printf "%-15s %5d %5d %5d\n", $2,$3,$4,$5 }'
stud Hari 17 16 15

• This command looks for two numbers only in the first field
($1). The second string is searched only if ( || ) the first
search fails.

• For negating the above condition, use the != and && operators as:

$1 != 125 || $1 != 135

The address here translates into this: “Select those records where
the third field doesn't (!=) completely match the string "director" and
(&&) also doesn't (!=) completely match the string "chairman".

Note:-
that the match is made for the entire field, rather than a-string
embedded in the field space. We couldn't verify that because the file
empn.1st has all trailing spaces trimmed. But will a similar awk
program work on the file emp.1st which has trailing spaces attached
to the strings?

No, it won't; the third field contains trailing white space, so a perfect
match wasn't found.

Field matching is better done with regular expressions

 Using Relational Operators
• awk can also handle numbers, both integer and floating type,
andall the relational tests can be handled by awk.
• Using relational operators, the list of students scoring greater
than 50 marks can be printed-

$ awk '$3+$4+$5 >= 50 {

printf "%-20s %d\n", $2, $3+$4+$5 }' stud

101 Ameya 52
156 Ganesha 50
159 Ratan 51

Relational Operators and Regular Expression Matching Operators in

awk

Operator Significance
< Less than
<= Less than or equal to
== Equal to
!= Not equal to
>= Greater than or equal to
> Greater than
~ Matches a regular expression
!~ Does not match a regular expression

~ and !~: The Regular Expression Operators

 HOW does one match regular expressions?

 Previously we had used awk with regular expressions in this manner:
 This matches the pattern "saxena" and "saksena" anywhere in the line
and not in a specific field.
 For matching anywhere in a field, awk offers the ~ and !~ operators to
match and negate a match., respectively.
 With these operators matching becomes more specific, as seen in
following examples:

 Remember that the operators ~ and !~ work only with field specifiers ($1,
$2 etc.).
 The delimiting of patterns with the | is an egrep feature which awk also
uses.

 In fact, awk uses all the regular expression characters used by egrep,
but doesn't accept the BRE and TRE used by grep and sed.
 Tip: To match a string embedded in a field, you must use ~ instead of ==.
Similarly, to negate a match, use !~ instead of !=.
File ‘store’
• Create the following file

Sr.No Product Qty Unit Price

1 Pen 100 40.00
2 Rubber 60 5.00
3 Pencil 60 3.50
4 GeoBox 20 145.00
5 ColorBox 30 70.00
6 NoteBook 200 32.00
7 Steppler 30 42.00

> cat > bill1

{
total =
$3*$4
srno =
$1 item
= $2
grandtotal = grandtotal +
totalprint srno item “ Rs.
“ total
}

• $ awk –f bill1 stores

pen Rs. 4000
pencil Rs. 210
rubber Rs. 300
geobox Rs. 2900
colbox Rs. 2100
notebook Rs. 7400
steppler Rs. 1260
The BEGIN and END sections
• In awk program, the action part is applied to the addressed
lines.If, it is required to print something before the
processing of first line, eg heading, the BEGIN section is
useful. Similarly the END section is useful in printing some
totals after the processing is over.

• The BEGIN and END sections are optional.

• The general form is-

BEGIN {action} # Both require braces

END {action}

• cat > bill2

BEGIN{
print ” ”
print ” Inventory Cost”
print ” ”}
{
total =
$3*$4
srno =
$1 item
= $2
grandtotal = grandtotal + total
print srno item “ Rs. “ total }

END {
print ” ”
print “ Total Rs. “ grandtotal
print ”===================================“ }
 • $ awk –f bill2 stores

Inventory Cost

1 pen Rs. 4000

2 pencil Rs. 210
3 rubber Rs. 300
4 geobox Rs. 2900
5 colbox Rs. 2100
6 notebook Rs. 7400
7 steppler Rs. 1260

Grand Total Rs.= 18170

================================

 $ cat > bill3

BEGIN {
print " "
print " Inventory Cost including Tax"
print " "}
{
total =
$3*$4
srno =
$1 item
= $2
grtotal += total
print srno " "item " ""Rs. " total
}
END {
print " "
print " Grand Total Rs.= " grtotal
print " "
print " Grand Total + VAT Rs.= " grtotal + 0.05*grtotal
print “========================================"
}
• $ awk -f bill3 stores

Inventory Cost including Tax

1 pen Rs. 4000

2 pencil Rs. 210
3 rubber Rs. 300
4 geobox Rs. 2900
5 colbox Rs. 2100
6 notebook Rs. 7400
7 steppler Rs. 1260

Grand Total Rs.= 18170

Grand Total + VAT Rs.= 19078.5

==============================

AWK - Arrays

AWK handles one dimensional array.

The index for an array can be virtually anything; it can even be a string.
No array declaration are necessary; an array is considered to be
declared the moment it is used, and is automatically initialized to zero,
unless initialized explicitly.
Syntax: array_name[index] = value
Where array_name is the name of array, index is the array index,
and value is any value assigning to the element of the array.
There are no type declarations, no initialization and no statement
terminators.
Creating Array
To gain more insight on array, let us create and access the elements of an
array.Example:

$ awk 'BEGIN {
fruits["mango"] = "yellow";
fruits["orange"] = "orange"
print fruits["orange"] "\n" fruits["mango"]
}'

On executing this code, you get the following result −

orange
yellow
In the above example, we declare the array as fruits whose index is
fruit name and the value is the color of the fruit. To access array
elements, we use array_name[index] format.

Deleting Array Elements

For insertion, we used assignment operator. Similarly, we can

use delete statement to remove an element from the array. The syntax of
delete statement is as follows −
Syntax
Delete array_name[index]
The following example deletes the element orange. Hence the
command does not show any output.
$ awk 'BEGIN {
fruits["mango"] = "yellow";
fruits["orange"] = "orange";
delete fruits["orange"];
print fruits["orange"]
}'
 FUNCTIONS
• awk has several built-in functions, performing both arithmetic
andstring operations.

• The parameters can be passed to a function in C-style.

• When a function is used without any parameter, the symbols

( )need not be used.

• awk even enables you to define your own functions.

Built-In functions in AWK

• Function Description

int(x) Returns integer value of x

sqrt(x) Returns square root of x
length Returns length of complete record
length(x) Returns length of x

substr(s1, s2,s3) Returns portion of string of length s3,

startingfrom position s2 in string s1
index(s1, s2) Returns position of string s2 in string s1
split(s, a) Splits string s into array a; optionally
returnsnumber of fields
system("cmd") Runs UNIX command cmd, and returns its
exit status

emp.lst file
2365 | barun sengupta | director | personne1 | 11/05/47 |
7800 3564 | sudhir Agarwal | executive | personnel |
06/07/47 |7500 4290|jayant Choudhury | executive |
production | 07/09/50 | 60009876 | jai Sharma | director |
production | 2/03/50 | 7000
 The length and length( ) Functions
• length() determines the length of its argument, and if no
argument is present, then it assumes the entire line as its
argument. You canuse the-following statement to locate
records whose length exceeds 57 characters:

awk –F “|" 'length> 57' emp.lst

• Observe that we used length without any argument. But
weprovide argument
• The following program selects those people who have
shortnames:
awk -F"|" ‘ length($2) < 11' emp.lst # $2 is the second
field

Example

$ awk 'BEGIN {
str = "Hello, World !!!"
print "Length = ", length(str)
}'
On executing this code, you get the following result −

Length = 16

The index() and substr() Functions

• The index() function determines the position of a string within
a larger string. This function is especially useful in validating
single character fields. If you have a field which can take the
values a, b,c, d or e, you can use this function to find whether
this single character field can be located within the string
abcde:

x = index ("abcde", “b”)

This returns the value 2.

• The substr() function takes three arguments.
• The first arguments1 represents the string to be used for
extraction, s2 represents the starting point of extraction, and
s3 indicates the number ofcharacters to be extracted.
 • Because string values can also be used for computation, the
returned string from this function can be used to select those
bornbetween 1946 and 1951.

$ awk –F ”|” 'substr($5,7,2) > 45 && substr($5,7,2) < 52' emp.lst

2365 | barun sengupta | director | personne1 | 11/05/47 | 7800 |
2365 3564 | sudhir Agarwal | executive | personnel | 06/07/47
|7500 | 23659876 | jai Sharma | director | production | 2/03/50 |
7000 | 9876

$ awk 'BEGIN {
str = "Hello, World !!!"
subs = substr(str, 1, 5)

print "Substring = " subs

}'
On executing this code, you get the following result −

Output
Substring = Hello

The split() Function :

• Split (), which also takes three arguments, breaks up a string
(firstargument) into fields, and stores them individually in an
array (second argument). The delimiter is used as the third
argument. The conversion of the date format to yyyymmdd is
in a simpler way:

$ awk –F “|” '{split($5,ar, "/") ; print

“19"ar[3]ar[2]ar[1]}`
em
p.lst19521212
19501203
19431904
• This method is superior because it explicitly picks up the fifth
field,whereas sed just transformed the only date field that it
could match.
Example

$ awk 'BEGIN {
str = "One,Two,Three,Four"
split(str, arr, ",")
print "Array contains following values"

for (i in arr) {
print arr[i]
}
}'
On executing this code, you get the following result −

Output
Array contains following values
One
Two
Three
Four

Write output of
split(“Mumbai-Pune-Kolhapur”, a, “-“)

BEGIN {
str = "Mumbai-Pune-Kolhapur"
split(str, arr, "-")
print "Array contains following values"

for (i in arr) {
print arr[i]
}
}

Array contains following values

Mumbai
Pune
Kolhapur
 getline function
• getline function is used to read input from keyboard and
thenassign the data to variable.

• Syntax:
getline variable-name < "-"
| | |
1 2 3

1 --> getline is function name

2 --> variable-name is used to assign the value read from
input
3 --> Means read from stdin (keyboard)

BEGIN {
printf "Your name please:"
getline na < "-"
printf “ your age please:"
getline age < "-"
print "Hello " na, ", your age is “, age
}

Begin {
printf " Enter name :"
getline na < "-"
printf " Enter marks :"
getline marks < "-"

printf " Your name and marks: %-10s %d \n", na, marks

if (marks >=66)
printf " You got Distinction! Congratulations!!!\n"
else if (marks >=60)
printf " You got First Class! Congratulations!!!\n"
else if (marks >=45)
printf " You got Second Class! Congratulations!"
}
CONTROL FLOW

Awk has practically all the features of a modern programming language.

It has conditional structures (the if statement) and loops (while and for).
They all execute a body of statements depending on the success or failure
of
the control command
(i.e., the condition specified in the first line of the construct).

if condition in awk:
The if statement can be used when the && and || are found to be
inadequate for certain tasks.
if condition in awk:
•General syntax of if condition is as follows:
Syntax:
if ( condition )
{
Statement 1
Statement 2
Statement N
}
else{
Statement 1
Statement 2
Statement N
}

 Like in C, none of the control flow constructs require to use

terminators if there's only one statement to be executed.
 But, when there are multiple actions implied, the statements must
 be enclosed within a pair of curly braces.
 Moreover, the control command must be enclosed in parenthesis
Example:
$ awk 'BEGIN {
num = 11;
if (num % 2 == 0)
printf "%d is even number.\n", num;
else
printf "%d is odd number.\n", num
}'
Output:
11 is odd number.

If-Else-If Ladder
We can easily create an if-else-if ladder by using multiple if-else
statements. The following example demonstrates this −
Example:

$ awk 'BEGIN {
a = 30;
if (a==10)
print "a = 10";
else if (a == 20)
print "a = 20";
else if (a == 30)
print "a = 30";
}'

On executing this code, you get the following result -

Output:
a = 30
 Loops in awk
• for loop and while loop are used for looping purpose in awk.
• Awk supports two loops for and while. They both execute as long
as control command returns a true value.

• Syntax of for loop-

for (initialization; condition; increment/decrement){
Statement 1
Statement 2
Statement N
}

• Example

BEGIN {
sum = 0
i=1
for (i=1; i<=10; i++)
{
sum = sum + i
}
printf "Sum for 1 to 10 numbers = %d \n", sum
}

This form also consists of three components; the first component initializes
the value of variable, the second checks the condition with every iteration,
while the third sets the increment used for every iteration.

If the condition is true, it executes actions, thereafter it performs increment

or decrement operation.

Example

$ awk 'BEGIN { for (i = 1; i <= 5; ++i) print i }‘

On executing this code, you get the following result −

Output
1
2
3
4
5

while loop
• The while loop has a similar role to play; it repeatedly iterates the loop till
the control command succeeds.

• Syntax of while loop is as follows:

while (condition)
{
statement1
statement2
statementN
}

• While loop will continue as long as given condition is TRUE.

Reversing the number

BEGIN{
no = $1
remn =
0
while ( no > 1 )
{
remn =
no % 10
no /= 10
printf "%d" ,remn }
printf "\nNext number please (CTRL+D to stop):";}
 BEGIN {

no = 12345
remn = 0
while ( no > 1 )
{
remn = no % 10
no /= 10
printf "%d" ,remn
}
printf "\nNext number please (CTRL+D to stop):";
}

Output:

54321
Next number please (CTRL+D to stop):

Do-While Loop
The do-while loop is similar to the while loop, except that the
test condition is evaluated at the end of the loop.
Syntax

do
action
while (condition)

In a do-while loop, the action statement gets executed at least once

even when the condition statement evaluates to false.
For instance, the following example prints 1 to 5 numbers using
do-while loop

Example:
$ awk 'BEGIN {i = 1; do { print i; ++i } while (i < 6) }‘

Output
1
2
3
4
5
Break Continue and Exit in AWK
1. Break
 It is used to end the loop execution.
 Here is an example which ends the loop when the sum
becomes greater than 50.

2. Continue
 The continue statement is used inside a loop to skip to the next
iteration of the loop.
 It is useful when you wish to skip the processing of some data
inside the loop.
 The following example uses continue statement to print the even
numbers between 1 to 20.

Example
$ awk 'BEGIN {
for (i = 1; i <= 20; ++i) {
if (i % 2 == 0) print i ; else continue
}
}'
 Output
2
4
6
8
10
12
14
16
18
20

3. Exit
It is used to stop the execution of the script. It accepts an
integer as an argument which is the exit status code for AWK
process. If no argument is supplied, exit returns status zero.
Let us check the return status of the script.

Example
$ echo $?

On executing this code, you get the following result −

Output
10
Here is an example that stops the execution when the sum
becomes greater than 50.