0% found this document useful (0 votes)

14 views22 pages

Unix and AWK Guide POLISHED FINAL FIXED

The document is a comprehensive guide on Unix and AWK programming, detailing various Unix commands with examples and their applications. It includes practical examples of using AWK for data manipulation, such as extracting fields, performing calculations, and filtering data. Additionally, it covers sorting, searching, and editing text files using commands like sort, grep, and sed.

Uploaded by

ragerspirit037

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

14 views22 pages

Unix and AWK Guide POLISHED FINAL FIXED

Uploaded by

ragerspirit037

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 22

Unix & AWK Programming Guide (Final Polished Version)

1. Unix Basics with Examples (From PPT)

Note: These commands can be tested on a common dataset like 'fruits.txt' or any structured text file.

LS
Lists files and directories.

Example: ls -l

=> Shows file permissions, ownership, size, and modification time.

CD
Changes the current working directory.

Example: cd /home/user

=> Moves into the /home/user directory.

MKDIR
Creates a new directory.

Example: mkdir myfolder

=> Creates a folder named 'myfolder'.

RM
Removes files or directories.

Example: rm file.txt

=> Deletes 'file.txt'.

TOUCH
Creates empty files.

Example: touch demo.txt

=> Creates a blank file 'demo.txt'.

CAT
Displays file contents.

Example: cat file.txt

=> Outputs all content of 'file.txt'.

NANO/VI
Edits files in terminal.

Example: nano file.txt

=> Opens 'file.txt' in nano editor.

CHMOD
Changes file permissions.

Example: chmod 755 script.sh

=> Full permission for user, read+execute for others.

CHOWN
Changes file ownership.

Example: chown user:group file.txt

=> Changes owner of 'file.txt'.

PS
Shows running processes.

Example: ps aux

=> Detailed list of active processes.

TOP
Displays real-time process list.

Example: top

=> Live CPU and memory usage.

KILL
Terminates a process.

Example: kill 1234

=> Ends process with PID 1234.

GREP
Searches for patterns.

Example: grep 'main' script.sh

=> Finds 'main' in script.sh.

FIND
Searches for files.

Example: find . -name '*.txt'

=> Lists all .txt files.

SORT
Sorts lines in a file.

Example: sort data.txt

=> Sorts lines alphabetically.

UNIQ
Filters repeated lines.
Example: sort data.txt | uniq

=> Removes duplicates.

CUT
Extracts specific fields.

Example: cut -d',' -f2 data.csv

=> Gets 2nd column.

HEAD
Shows beginning lines.

Example: head -n 3 file.txt

=> First 3 lines.

TAIL
Shows ending lines.

Example: tail -n 3 file.txt

=> Last 3 lines.

ECHO
Displays a string.

Example: echo Hello

=> Outputs 'Hello'.

REDIRECTION
Redirects output.

Example: echo Hello > file.txt

=> Writes 'Hello' to file.

PIPE
Connects commands.

Example: ls | grep txt

=> Lists only .txt files.

2. Detailed Programs and Explanations from Unix 1 (1).txt
Example 1
Example 1: To Print the 2nd, 3rd and 5th Columns only

fruit_id,fruit_name,fruit_qty,unit_price,total_price

1,Mango,2,10,20

2,Apple,6,15,90

3,Banana,4,8,int

4,Watermelon,7,9,63

Solution:

awk 'BEGIN{

FS=",";

OFS="|";

print $2,$3,$5;

}' fruits.txt

--------------------------------------------------------------------------------------

This block uses AWK for tasks such as pattern matching, field extraction, looping, and arithmetic calculations.

Example 2
Example 2: To Find the total price of an apple (PIPELINING grep & awk)

fruit_id,fruit_name,fruit_qty,unit_price,total_price

1,Mango,2,10,20

2,Apple,6,15,90

3,Banana,4,8,int

4,Watermelon,7,9,63

5,apple,3,15,45
Solution:

grep -i "[Aa]pple" fruits.txt | awk 'BEGIN{FS=",";s=0;}

s = s+$5;

END{

print "The total price spent on apple fruit is " s;

--------------------------------------------------------------------------------------

This block uses AWK for tasks such as pattern matching, field extraction, looping, and arithmetic calculations.

Example 3
Example 3: Printing the Multiplication of Number 5

Solution:

awk 'BEGIN{

i=1;

while(i<=10)

print "5 * "i" = "i*5;

i++;

--------------------------------------------------------------------------------------

This block uses AWK for tasks such as pattern matching, field extraction, looping, and arithmetic calculations.

Example 4
Example 4: Find the number of fields in a record and total no.of records present in the text file
mango

watermelon

pine apple

custard apple

banana

Solution:

awk 'BEGIN{FS=" ";}

print "The Number of fields present in record " NR " is " NF".";

END{

print NR" records are present.";

}' fruits2.txt

---------------------------------------------------------------------------------------------

Input Data:

29,Arun

26,Karthik

28,Kiran

52,Raju

78,Rachel

This block uses AWK for tasks such as pattern matching, field extraction, looping, and arithmetic calculations.

Example 5
Example 5:

awk 'BEGIN{FS=",";}

if($1>50)
{

print "Value is greater than 50.";

else

print "Value is less than 50";

i=1;

while(i<=1)

print "Row " NR ", Column 2 (loop "i "): " $2;

i++;

END{

print match("End of Code", /of/);

print RSTART, RLENGTH;

}' example.txt

This block uses AWK for tasks such as pattern matching, field extraction, looping, and arithmetic calculations.

Example 6
Example 6:

grep ',A' input3.txt | awk 'BEGIN{FS=",";total_age=0;}

total_age += $1;

count++;

END{

if(count>0)

avg_age = total_age/count;
printf "Average Age for Group A:%.2f\n", avg_age;

else

print "No Data Found";

--------------------------------------------------------------------------------------

ROYAL MAIL HOTEL (CASE STUDY)

employeeDetails.txt (DATA) :

Name,Age,Place,Experience,Salary

Anish,26,Chennai,2,10000

Jai,24,Chennai,2,10000

Kumar,29,Hyderabad,5,32000

John,32,Mumbai,2,11000

Neethu,21,Nagpur,3,13000

Satish,22,Ahmedabad,2,10000

Situation: To Print the complete data

awk 'BEGIN{FS=",";}

print;

}' employeeDetails.txt

Situation: Manager wish to display the employee name and salary working in royal mail hotel

awk 'BEGIN{FS=",";}

print $1,$5;
}' employeeDetails.txt

Situation: Manager wishes to print details of Kumar and Satish

awk '/Kumar|Satish/' employeeDetails.txt

Situation: Manager wishes to find the total expenses of hotel per month in the form of salary

awk -F"," 'BEGIN{

s=0;

s=s+$5;

END{

print "Total Exprenses per month in form of salary is " s;

}' employeeDetails.txt

Situation: Manager wishes to find the total no.of employees earning 10000 per month

awk '/10000/{

++count;

END{

print "No.of employees earning 10000 per month is " count;

}' employeeDetails.txt

Situation: Manager wishes to find the employees as best performers who completed 2 years of exprerience

and earning more than 10000

awk 'BEGIN{

FS=",";

print "Performance Report";

}

if(NR!=1)

if($4>=2 && $5>10000)

print $1" is a good performer";

else if($4==2 && $5<=10000)

print $1" Needs to improve";

END{

print "Performance Report";

}' employeeDetails.txt

-----------------------------------------------------------------------------------------

SORT COMMAND

input data file: list

1,John Cena,Title 758,Price $7.30

2,Randy Orton,Title 739,Price $6.20

4,Triple H,Title 893,Price $6.42

5,GoldBerg,Title 392,Price $1.98

input data file: list1

7,Shawn Michales,Title 620,Price $1.64

8,Roman Reigns,Title 920,Price $1.03

3,Brock Lesnar,Title 201,Price $6.71

6,Rey mysterio,Title 109,Price $12.4

Situation: Sort on the 2nd field of file named "list". File list is comma seperated value

Solution: sort -k 2 list;

Situation: Sort the input data file

Solution: sort list;

Situation: Sort can be applied on multiple files as well

Solution: sort -n list list1;

Situation: Sort in reverse order of first numeric column from multiple files

Solution: sort -nr list list1;

Situation: Sort the above input file by removing the duplicate records/lines.

Solution: sort -u list;

Situation: Sort two input files and merge it.

Solution: sort -m list list1;

---------------------------------------------------------------------------------------------------------
UNIQ COMMAND

input data file: list.txt

unix operating system

unix dedicated server

linux dedicated server

Situation: surpass the duplicate records

Solution: uniq list.txt

Situation: Pipelining command using (Sort and Uniq)

Solution: sort list.txt | uniq

Situation: Count the repeated lines

Solution: uniq -c list.txt

Situation: Display only the duplicate lines

Solution: uniq -d list.txt

Situation: Display all duplicate lines

Solution: uniq -D list.txt

Situation: Print only unique lines

Solution: uniq -u list.txt

------------------------------------------------------------------------------------------

GREP COMMAND

demo_file

THIS LINE IS THE 1ST UPPER CASE IN THIS FILE.

this line is the 1st lower case line in the file.

This Line Has All Its First Characters as Upper Case.

Two lines above this line is empty.

And this is the last line.

Situation: Search for a given string and also you can check a string in multiple files

Solution: grep "this" demo_file.txt

Situation: Search for a given string with caseinsensitive approach

Solution: grep -i "this" demo_file.txt

Situation: Search for a line starting with 'Two' and ending with 'empty'

Solution: grep "Two.*empty" example.txt

Situation: To Search for a word and to avoid it to match the substrings -w option is used

Solution: grep -w "is" example.txt

Situation: If you want to display the lines which does not matches the given string/pattern then
Solution: grep -v "Two" example.txt

-----------------------------------------------------------------------------------------------------------

SED COMMAND

file_name

1.Linux - System Admin, Scripting etc.

2.Database - Oracle, mySQL etc.

3.Hardware

4.Security (Firewall, Network etc.)

5.Storage

Operation in Sed

-p = Prints specific number of lines based on line number or pattern

-n = suppress automatic printing of patternspace. It will not print any thing until explicit request to print is

found

-d = delete the line when pattren matches

-i = modifies the text in the input file

SYNTAX:

sed -n '/pattern/'p file_name

Replaces the occurrences of 'Security' with 'security' from file_name.txt

sed 's/Security/security/g' file_name.txt

Prints lines containing 'Security' by duplicating from file_name.txt

sed '/Security/p' file_name.txt

Prints the lines containing 'Security' until explicit request to print is found
sed -n '/Security/p' file_name.txt

Prints lines 2 to 4 from file_name.txt

sed -n '2,4p' file_name.txt

Modifies the text 'Hardware' to 'hardwares' from file_name.txt

sed -i 's/Hardware/hardwares/g' file_name.txt

Prints every 2nd line starts from line 3.

sed -n '3~2p' file_name.txt

------------------------------------------------------------------------------AWK Command-------------------------------------

Regular Expressions

DOT

echo -e "cat\ncut\nfun\nfin\nfan" | awk '/c.t/'

START OF THE LINE

echo -e "This\nThat\nThere\nTheir\nthese" | awk '/^The/'

END OF THE LINE

echo -e "knife\nknow\nfun\nfin\nfan\nnine" | awk '/n$/'

MATCH CHARACTER SET

echo -e "Call\nTall\nBall" | awk '/[CT]all/'

EXCLUSIVE SET
echo -e "Call\nTall\nBall" | awk '/[^CT]all/'

ALTERATION

echo -e "Call\nTBall\nSmall\nShall" | awk '/Call|Ball/'

ZERO OR ONE OCCURRENCE

echo -e "Colour\nColor" | awk '/Colou?r/'

ZERO OR MORE OCCURRENCE

echo -e "ca\ncat\ncatt" | awk '/cat*/'

ONE OR MORE OCCURRENCE

echo -e "111\n22\n123\n234\n456\n222" | awk '/2+/'

GROUPING

echo -e "Apple Juice\nApple Pie\nApple Tart\nApple Cake" | awk '/Apple (Juice|Cake)/'

----------------------------------------------------------------------------------

USER DEFINED FUNCTIONS

# Returns minimum number

function find_min(num1, num2){

if (num1 < num2)

return num1

return num2

}
# Returns maximum number

function find_max(num1, num2){

if (num1 > num2)

return num1

return num2

# Main function

function main(num1, num2){

# Find minimum number

result = find_min(10, 20)

print "Minimum =", result

# Find maximum number

result = find_max(10, 20)

print "Maximum =", result

# Script execution starts here

BEGIN {

main(10, 20)

--------------------------------------------------------------------------------

CONTROL FLOW STATEMENTS

IF STATEMENT

awk 'BEGIN {num = 10; if (num % 2 == 0) printf "%d is even number.\n", num }'

IF-ELSE STATEMENT
awk 'BEGIN {

num = 11;

if (num % 2 == 0)

printf "%d is even number.\n", num;

else