Agenda

 Introduction
• UNIX/LINUX and Shell
• UNIX Commands and Utilities
• Basic Shell Scripting Structure
 Shell Programming
• Variable
• Operators
• Logic Structures
 Examples of Application in Research Computing
 Hands-on Exercises

The PPT/WORD format of this presentation is available here:

http://its2.unc.edu/divisions/rc/training/scientific/
its.unc.edu /afs/isis/depts/its/public_html/divisions/rc/training/scientific/short_courses/ 1
 Why Shell Scripting ?

 Shell scripts can be used to prepare

input files, job monitoring, and output
processing.
 Useful to create own commands.
 Save lots of time on file processing.
 To automate some task of day to day life.
 System Administration part can be also
automated.

its.unc.edu 2
 Objectives & Prerequisites
 After this workshop, you should be:
• Familiar with UNIX/LINUX, Borne Shell, shell
variables/operators
• Able to write simple shell scripts to illustrate
programming logic
• Able to write scripts for research computing purposes
 We assume that you have/know
• An account on the Emerald cluster
• Basic knowledge of UNIX/LINUX and commands
• UNIX editor e.g. vi or emacs

its.unc.edu 3
 History of
UNIX/Linux
 Unix is a command line operating system developed
around 1969 in the Bell Labs
 Originally written using C
 Unix is designed so that users can extend the
functionality
• To build new tools easily and efficiently
• To customize the shell and user interface.
• To string together a series of Unix commands to
create new functionality.
• To create custom commands that do exactly what
we want.
 Around 1990 Linus Torvalds of Helsinki University
started off a freely available academic version of Unix
 Linux is the Antidote to a Microsoft dominated future
its.unc.edu 4
 What is
UNIX/Linux ?
Simply put
 Multi-Tasking O/S
 Multi-User O/S
 Available on a range of Computers
 SunOS Sun Microsystems
 IRIX Silicon Graphics
 HP-UX Hewlett Packard
 AIX IBM
 Linux ….
its.unc.edu 5
 UNIX/LINUX
Architecture

its.unc.edu 6
 What is a “Shell”?

 The “Shell” is simply another program on top of the

kernel which provides a basic human-OS interface.
• It is a command interpreter
 Built on top of the kernel
 Enables users to run services provided by the UNIX OS
• In its simplest form, a series of commands in a file is a
shell program that saves having to retype commands to
perform common tasks.

use
 How to know what shell you use Shel
l
r

echo $SHELL
use OS
r

use
r
its.unc.edu 7
 UNIX Shells

 sh Bourne Shell (Original Shell) (Steven Bourne of

AT&T)
 bash Bourne Again Shell (GNU Improved Bourne
Shell)
 csh C-Shell (C-like Syntax)(Bill Joy of Univ. of
California)
 ksh Korn-Shell (Bourne+some C-shell)(David Korn
of AT&T)
 tcsh Turbo C-Shell (More User Friendly C-Shell).
 To check shell:
• $ echo $SHELL (shell is a pre-defined variable)
 To switch shell:
its.unc.edu
• $ exec shellname (e.g., $ exec bash or simply type $ 8
 Which Shell to Use?

 sh ( Bourne shell) was considered better for programming

 csh (C-Shell ) was considered better for interactive work.
 tcsh and korn were improvements on c-shell and bourne shell
respectively.
 bash is largely compatible with sh and also has many of the
nice features of the other shells
 On many systems such as our LINUX clusters sh is symbolically
linked to bash, /bin/sh -> /bin/bash
 We recommend that you use sh/bash for writing new shell
scripts but learn csh/tcsh to understand existing scripts.
 Many, if not all, scientific applications require csh/tcsh
environment (GUI, Graphics Utility Interface)
 All Linux versions use the Bash shell (Bourne Again Shell) as the
default shell
• Bash/Bourn/ksh/sh prompt: $
• All UNIX system include C shell and its predecessor Bourne shell.
• Csh/tcsh prompt: %

its.unc.edu 9
 What is Shell Script?

 A shell script is a script written for the shell

 Two key ingredients

• UNIX/LINUX commands
• Shell programming syntax

its.unc.edu 10
 A Shell Script Example
#!/bin/sh

`ls -l *.log| awk '{print $8}' |sed 's/.log//g' > file_list`

cat file_list|while read each_file

do
babel -ig03 $each_file".log" -oxyz $each_file".xyz“

echo '# nosymmetry integral=Grid=UltraFine scf=tight rhf/6-311++g**

pop=(nbo,chelpg)'>head
echo ' ' >>head
echo ''$each_file' opt pop nbo chelp aim charges ' >> head
echo ' ' >>head
echo '0 1 ' >>head

`sed '1,2d' $each_file.xyz >junk`

input=./$each_file".com"
cat head > $input
cat junk >> $input
echo ' ' >> $input

done
/bin/rm ./junk ./head ./file_list

its.unc.edu 11
 UNIX/LINUX Commands

File Management and Viewing To understand the working of the

command and possible options
Filesystem Mangement use (man command)
Help,Job/Process
Management Using the GNU Info System
(info, info command)
Network Management
Listing a Description of a Program
System Management (whatis command)
User Management
Many tools have a long−style
Printing and Programming option, `−−help', that outputs
Document Preparation usage information about the tool,
including the options and
Miscellaneous arguments the tool takes. Ex:
whoami --help

its.unc.edu 12
 File and Directory Management

cd Change the current directory. With no arguments "cd" changes to the users home
directory. (cd <directory path>)
chmod Change the file permissions.
Ex: chmod 751 myfile : change the file permissions to rwx for owner, rx for group
and x for others (x=1,r=4,w=2)
Ex: chmod go=+r myfile : Add read permission for the group and others (character
meanings u-user, g-group, o-other, + add permission,-remove,r-read,w-write,x-exe)
Ex: chmod +s myfile - Setuid bit on the file which allows the program to run with
user or group privileges of the file.
chown Change owner.
Ex: chown <owner1> <filename> : Change ownership of a file to owner1.
chgrp Change group.
Ex: chgrp <group1> <filename> : Change group of a file to group1.
cp Copy a file from one location to another.
Ex: cp file1 file2 : Copy file1 to file2; Ex: cp –R dir1 dir2 : Copy dir1 to dir2

its.unc.edu 13
 File and Directory Management

ls List contents of a directory.

Ex: ls, ls –l , ls –al, ls –ld, ls –R
mkdir Make a directory.
Ex: mkdir <directory name> : Makes a directory
Ex mkdir –p /www/chache/var/log will create all the directories starting from www.
mv Move or rename a file or directory.
Ex: mv <source> <destination>
find Find files (find <start directory> -name <file name> -print)
Ex: find /home –name readme -print
Search for readme starting at home and output full path, “/home" = Search starting at the
home directory and proceed through all its subdirectories; "-name readme" = Search
for a file named readme "-print" = Output the full path to that file
locate File locating program that uses the slocate database.
Ex: locate –u to create the database,
locate <file/directory> to find file/directory

its.unc.edu 14
 File and Directory Management
pwd Print or list the present working directory with full path.
rm Delete files (Remove files). (rm –rf <directory/file>)
rmdir Remove a directory. The directory must be empty. (rmdir <directory>)
touch Change file timestamps to the current time. Make the file if it doesn't exist. (touch
<filename>)
whereis Locate the binary and man page files for a command. (whereis
<program/command>)
which Show full path of commands where given commands reside. (which <command>)

File viewing and editing

emacs Full screen editor.
pico Simple text editor.
vi Editor with a command mode and text mode. Starts in command mode.
gedit GUI Text Editor
tail Look at the last 10 lines of a file.
Ex: tail –f <filename> ; Ex: tail -100 <filename>
head Look at the first 10 lines of a file. (head <filename>)

its.unc.edu 15
 File and Directory Management

File compression, backing up and restoring

compress Compress data.
uncompress Expand data.
cpio Can store files on tapes. to/from archives.
gzip - zip a file to a gz file.
gunzip - unzip a gz file.
tar Archives files and directories. Can store files and directories on tapes.
Ex: tar -zcvf <destination> <files/directories> - Archive copy groups of files. tar –zxvf
<compressed file> to uncompress
zip – Compresses a file to a .zip file.
unzip – Uncompresses a file with .zip extension.
cat View a file
Ex: cat filename
cmp Compare two files.
cut Remove sections from each line of files.

its.unc.edu 16
 File and Directory Management

diff Show the differences between files.

Ex: diff file1 file2 : Find differences between file1 & file2.
echo Display a line of text.
grep List all files with the specified expression.
(grep pattern <filename/directorypath>)
Ex: ls –l |grep sidbi : List all lines with a sidbi in them.
Ex: grep " R " : Search for R with a space on each side
sleep Delay for a specified amount of time.
sort Sort a file alphabetically.
uniq Remove duplicate lines from a sorted file.
wc Count lines, words, characters in a file. (wc –c/w/l <filename>).
sed stream editor, extremely powerful!
awk an extremely versatile programming language for working on files

its.unc.edu 17
 Useful Commands in
Scripting
 grep
• Pattern searching
• Example: grep ‘boo’ filename
 sed
• Text editing
• Example: sed 's/XYZ/xyz/g' filename
 awk
• Pattern scanning and processing
• Example: awk ‘{print $4, $7}’ filename

its.unc.edu 18
 Shell Scripting

 Start vi scriptfilename.sh with the line

#!/bin/sh
 All other lines starting with # are
comments.
• make code readable by including comments
 Tell Unix that the script file is executable
$ chmod u+x scriptfilename.sh
$ chmod +x scriptfilename.sh
 Execute the shell-script
$ ./scriptfilename.sh
its.unc.edu 19
 My First Shell Script
$ vi myfirstscript.sh
#! /bin/sh

# The first example of a shell script

directory=`pwd`
echo Hello World!
echo The date today is `date`
echo The current directory is $directory
$ chmod +x myfirstscript.sh
$ ./myfirstscript.sh
Hello World!
The date today is Mon Mar 8 15:20:09 EST 2010
The current directory is /netscr/shubin/test

its.unc.edu 20
 Shell Scripts
 Text files that contain sequences of UNIX
commands , created by a text editor
 No compiler required to run a shell script, because
the UNIX shell acts as an interpreter when reading
script files
 After you create a shell script, you simply tell the OS
that the file is a program that can be executed, by
using the chmod command to change the files’ mode
to be executable
 Shell programs run less quickly than compiled
programs, because the shell must interpret each
UNIX command inside the executable script file
before it is executed

its.unc.edu 21
 Commenting
 Lines starting with # are comments except the
very first line where #! indicates the location of
the shell that will be run to execute the script.
 On any line characters following an unquoted #
are considered to be comments and ignored.
 Comments are used to;
• Identify who wrote it and when
• Identify input variables
• Make code easy to read
• Explain complex code sections
• Version control tracking
• Record modifications

its.unc.edu 22
 Quote Characters

There are three different quote characters with

different behaviour. These are:
“ : double quote, weak quote. If a string is enclosed
in “ ” the references to variables (i.e $variable )
are replaced by their values. Also back-quote and
escape \ characters are treated specially.
‘ : single quote, strong quote. Everything inside
single quotes are taken literally, nothing is treated
as special.
` : back quote. A string enclosed as such is treated as
a command and the shell attempts to execute it. If
the execution is successful the primary output from
the command replaces the string.
Example: echo “Today is:” `date`
its.unc.edu 23
 Echo

Echo command is well appreciated when trying to

debug scripts.
Syntax : echo {options} string
Options: -e : expand \ (back-slash ) special
characters
-n : do not output a new-line at the end.
String can be a “weakly quoted” or a ‘strongly
quoted’ string. In the weakly quoted strings the
references to variables are replaced by the value
of those variables before the output.
As well as the variables some special
backslash_escaped symbols are expanded during
the output. If such expansions are required the –e
its.unc.edu option must be used. 24
 User Input During Shell Script
Execution
 As shown on the hello script input from the
standard input location is done via the read
command.
 Example
echo "Please enter three filenames:”
read filea fileb filec
echo “These files are used:$filea $fileb $filec”
 Each read statement reads an entire line. In the
above example if there are less than 3 items in the
response the trailing variables will be set to blank ‘
‘.
 Three items are separated by one space.

its.unc.edu 25
 Hello script exercise continued…

 The following script asks the user to enter

his name and displays a personalised
hello.
#!/bin/sh
echo “Who am I talking to?”
read user_name
echo “Hello $user_name”
 Try replacing “ with ‘ in the last line to see
what happens.

its.unc.edu 26
 Debugging your shell scripts

 Generous use of the echo command will help.

 Run script with the –x parameter.

E.g. sh –x ./myscript
or set –o xtrace before running the
script.

 These options can be added to the first line of

the script where the shell is defined.
e.g. #!/bin/sh -xv

its.unc.edu 27
 Shell Programming

 Programming features of the UNIX/LINUX shell:

 Shell variables: Your scripts often need to keep values in memory for later use.
Shell variables are symbolic names that can access values stored in memory
 Operators: Shell scripts support many operators, including those for performing
mathematical operations
 Logic structures: Shell scripts support sequential logic (for performing a series of
commands), decision logic (for branching from one point in a script to another),
looping logic (for repeating a command several times), and case logic (for
choosing an action from several possible alternatives)

its.unc.edu 28
 Variables

 Variables are symbolic names that represent values stored in memory

 Three different types of variables
• Global Variables: Environment and configuration variables, capitalized, such as
HOME, PATH, SHELL, USERNAME, and PWD.
When you login, there will be a large number of global System variables
that are already defined. These can be freely referenced and used in your
shell scripts.
• Local Variables
Within a shell script, you can create as many new variables as needed. Any
variable created in this manner remains in existence only within that shell.
• Special Variables
Reversed for OS, shell programming, etc. such as positional parameters $0,
$1 …

its.unc.edu 29
 A few global (environment)
variables
SHELL Current shell
DISPLAY Used by X-Windows system to identify the
display
HOME Fully qualified name of your login directory

PATH Search path for commands

MANPATH Search path for <man> pages

PS1 & PS2 Primary and Secondary prompt strings

USER Your login name

TERM terminal type
PWD Current working directory
its.unc.edu 30
 Referencing Variables

Variable contents are accessed using ‘$’:

e.g. $ echo $HOME

$ echo $SHELL
To see a list of your environment variables:

$ printenv

or:

$ printenv | more

its.unc.edu 31