Introduction to Bourne Shell Programming
==============================================================================
Overview: Shell programming is very similar in concept in many operating
systems. If you know how to write "batch" files in MS-DOS, then
you know the basic ideas behind shell programming in UNIX.
However, the syntax is altogether different. This tutorial focuses
solely on the Bourne shell, not the Cshell.
==============================================================================
Section Topic
------- -----
The need for shell programming
How to create simple scripts
How to make a file executable and put it in your path
Parameters
The test command
Variables
Use of variables in the shell
Arithmetic variables
Expressions and true and false
Input and Output
Built-in variables
Case statements
Here document
Executing commands
Recursion
Debugging
Performance considerations
Learning more about shell programming
-------------------------------------------------------------------------------
The need for shell programming
-------------------------------------------------------------------------------
Do you ever find that you often do a number of UNIX commands together and
you would like to "bundle" them up into one name? You can do this, in effect,
creating a brand new command. Other operating systems permit this convenience,
most notably MS-DOS, which calls such files "BAT" or batch files. In UNIX,
such a file is called a shell script.
First, make sure you know about the various UNIX shells (Bourne and C-shell).
There is information in the glossary menu.
Both the Bourne shell and the C shell permit you to create and use shell
scripts, but because the syntax of the commands that these two shells use is
slightly different, your shell script must match the shell that is interpret-
ing it, or you will get errors.
A shell script is just a readable file that you create and into which you put
shell commands. The first line determines which shell will interpret or
execute this shell script.
* If the first line begins with a C-shell comment (starting with # in
position 1) then this will be interpreted by the C-shell and must use
C-shell syntax.
* Otherwise, the file will be considered a Bourne shell script.
You can have comments in either type of shell script, although the syntax
differs. Bourne shell comments begin with a colon or a pound sign (#), whereas
C-shell comments commence with the pound sign (#).
For the rest of this tutorial, we will concentrate on the Bourne shell.
-------------------------------------------------------------------------------
How to create simple scripts
-------------------------------------------------------------------------------
Most shell scripts that you write will be very simple. They will consist of
a number of UNIX commands that you would have typed at the prompt, possibly
substituting a different file name. These substitutions are called positional
parameters.
To create a shell script that has no parameters and does the same thing every
time it is called, just put the commands in a file. Change the permissions on
the file so that it is executable and then use it. The name of the file should
be something that you can easily remember and which makes sense given the
operation that you are performing.
Let's make one that clears the screen, prints out the date, time, hostname,
current working directory, current username, and how many people are logged on.
The name of the script will be "status". So edit a file called "status" and
put the following lines into it: (Don't type the "frame" of dashes and vertical
bars -- these are meant to show you what the file looks like.)
+----------------------------------------------------------
| clear
| echo -n "It is currently: ";date
| echo -n "I am logged on as ";who am i
| echo -n "This computer is called ";hostname
| echo -n "I am currently in the directory ";pwd
| echo -n "The number of people currently logged on is:"
| who | wc -l
+----------------------------------------------------------
-------------------------------------------------------------------------------
How to make a file executable and put it in your path
-------------------------------------------------------------------------------
Make sure that you put the # in line 1. Now set the permissions:
% chmod 0700 status
This makes it executable and readable, both of which are necessary.
To use, just type
% status
If you see a cryptic command saying "command not found", it is probably
because your path does not include the current directory. To remedy this,
put the dot and a slash in front of the name:
% ./status
or you can modify your path:
% set path=($path .)
Note the space in front of the period.
Let's explain just a few things in the shell script above. Note that
echo -n is used a lot. The echo command just prints lines to the screen,
and normally it puts a newline after the thing it prints. -n inhibits
this, so that the output looks better.
You can string together more than one command on a line by using a semicolon.
Thus, clear;date;whoami;pwd could be put all on one line and all four of
the commands would be executed, one after the other. This is similar to the
vertical bar (the pipe), although it is simpler.
-------------------------------------------------------------------------------
Parameters
-------------------------------------------------------------------------------
Now let's get more complicated by adding positional parameters. Parameters
are given after the name of the file when you start the shell script. Each
parameter has a position, first, second, etc. When the shell interpreter
reads and executes each line of the shell script file, it looks for symbols
like $1, $2, etc and it substitutes for these symbols the positional parameters.
Let's do a very simple example. Our shell script will attempt to find the
word "unix" (irrespective of case) in a file that we give as a positional
parameter:
+----------------------------------------------------------
| grep -i unix $1
+----------------------------------------------------------
The -i option says ignore case. Since we are always looking for the word
unix (or UNIX, or Unix, etc.), all we need to vary is the file name. Suppose
that we called this file "funix" for "find unix", and we made it executable
using chmod. Now to use it on a file, we would type
% funix myjunk
and it would search file "myjunk" for the word unix (or Unix, or UNIX, etc.),
printing out each line that it found.
You can have any number of parameters. The second is $2, the third is $3,
etc.
Another common variation is to refer to all the parameters at once by using
$*. Our little shell script only looks at one file at a time. If we typed
funix myjunk yourjunk theirjunk ourjunk
it would only search the first file "myjunk". To make it search all, we
could do
+----------------------------------------------------------
| for i in $*
| do grep -i unix $i
| done
+----------------------------------------------------------
"for" is one of the many control structures of C-shell scripts. It takes
a list of items $*, which are all the parameters you gave to this script, and
assigns each one to the shell variable i in turn. Then this shell variable is
referenced (i.e., used) in the grep command by saying $i. All shell variables
must have a $ in front when they are used. The done keyword says that this is
the end of the for construct, not the end of the shell script.
In many situations, UNIX commands themselves are set up to accept multiple
filenames, and grep is one of these. So you could have done
+----------------------------------------------------------
| grep -i unix $*
+----------------------------------------------------------
instead. But not all cases work this easily. You just have to know your
UNIX commands.
Let us review the syntax of parameters. Each parameter is identified by $1,
$2, $3 and so on. The name of the command is $0. A short hand for all the
parameters is $*. To find out how many parameters there are, $# is used.
Here's an example of the beginning of a shell script which checks to see if
the user entered enough parameters, because some scripts require a certain
number. For example, grep needs at least one parameter, which is the string
to search for.
+----------------------------------------------------------
| if test $# -lt 2
| then echo "Not enough parameters"
| echo "usage: slop file searchstring"
| exit
| else echo "OK"
| fi
+----------------------------------------------------------
It literally says "Is the number of parameters ($#) less than 2?" If so, then
echo back a bunch of messages and exit the whole shell script. This example
gives you a flavor of the syntax of the if statement, use of the echo command
to act as output from a shell script, and the exit command which terminates
the shell script immediately.
The general syntax of if and if-then-else is:
if test-command if test-command
then statements then statements
fi else statements
fi
Some people do not like the dense spacing that the Bourne shell imposes. But
you cannot have a then line or an else line without a command following it.
So a way to fix this is to merely continue the line by using the UNIX escape
character, the backslash:
if test-command
then \
statements
else \
statements
fi
The backslash says not to interpret the newline as anything special. It is just
whitespace, just like tabs or blanks.
-------------------------------------------------------------------------------
The test command
-------------------------------------------------------------------------------
At the heart of the if statement is the test command, which is a UNIX command.
Actually, the Bourne shell accepts the return status code of any UNIX command
as a possible boolean value. But most of the time, the shell programmer wants
to compare values in variables, and the test command is the way this is done.
However, always remember that you can use "raw" UNIX commands. For instance,
to search for a string in a file and see if you found it or not, you could use
grep directly in an if statement. For example:
+-----------------------------------
| if grep -s $1 $2
| then echo "Found it"
| else echo "Did not find it"
| fi
+-----------------------------------
The -s option is "silent", which causes grep not to produce any output, but
merely to return its status code (0 or 1).
File queries are used a lot in if statements using the test command. These are
expressions that are used to determine characteristics of files so that
appropriate action may be taken. For example, suppose that you want to see if
a certain file exists and is readable (by you):
if test -r somefile
then grep $1 somefile
else echo "ERROR! Database file does not exist or is not readable".
fi
The name of the file does not have to be "hard coded" into the if statement, but
may be a parameter:
if test -f $2
Here is a full list of the Bourne shell file queries used in a test command:
-r file file exists and is readable by user
-w file file is writable by user
-x file file is executable by user
-o file file is owned by user
-s file file is longer than zero bytes
-f file file is an ordinary file
-d file file is a directory
There are other queries that are used with strings:
-l string returns the length of the string
-n s1 True if the length of the string s1 is non-zero
-z s1 True if the length of string s1 is zero
s1 = s2 True if the strings s1 and s2 are equal
s1 != s2 True if the strings s1 and s2 are not equal
s1 True if s1 is not the null string
Parameters count as strings, as do any values for variables that are not
numeric. Actually, a string may consist of only digits, so it may treated as
a string by one command and as an integer by another.
Here are the parameters to the test command that deal with numeric values:
n1 -eq n2 True if the integers n1 and n2 are numerically equal
n1 -ne n2 True if the integer n1 is not numerically equal to the
integer n2
n1 -gt n2 True if the integer n1 is numerically greater than the
integer n2
n1 -ge n2 True if the integer n1 is numerically greater than or
equal to the integer n2
n1 -lt n2 True if the integer n1 is numerically less than the
integer n2
n1 -le n2 True if the integer n1 is numerically less than or equal
to the integer n2
Notice that -eq is used instead of = or ==, and lt is "less than". You cannot
use < or >. Perhaps this is because the early FORTRAN language used .EQ., .NE.,
.LT., .LE., and so forth.
In addition to these primitive tests, you can combine them with Boolean oper-
ators to make more complicated tests:
! Unary negation operator
-a Binary and operator
-o Binary or operator
(expression) Parentheses for grouping
-a has higher precedence than -o. Unfortunately, the parentheses are meaning-
ful to the shell (they cause the enclosed commands to be executed in a new
shell) so you must escape them by using backslashes.
For example the way to test to see if a file does not exist would be:
if test ! -f somefile -a ! -d somefile ! -p somefile
then echo does not exist
Make sure to put spaces before and after all the elements of the test command,
or you will confuse the C shell. Here's another example, showing how to tell
if "somefile" is a regular file and is writable:
if test -f somefile -a -w somefile
then echo the file exists, is not a directory and I can write it
Here's an example using parentheses:
if test \( -f somefile -o -d somefile \) -a ! -w somefile
This would be true if somefile were either a regular file or a directory and
it is not writable.
-------------------------------------------------------------------------------
Variables
-------------------------------------------------------------------------------
The Bourne shell scripting language permits variables. The variables can have
long, mnemonic names and can be either character or numeric, although floating
point variables are not allowed. You cannot have arrays as you can in the C
shell, however.
When you refer to a variable's value in a command, you must prefix the variable
name with a dollar sign. The only time you don't use a dollar sign is in the
assignment statement which assigns a value to a variable, or changes the value
of an existing variable. The format of the assignment is
name=expression
Beware! Do not put any blanks on either side of the equals sign! The blanks
are the only way that Bourne shell knows if = means "assign" or "compare".
Here are a few examples:
dirname=$1
x=5
y=$x
name=Mark
location="Buffalo, NY"
Shell variables are dynamic. They are not declared and come into existence
when they are first set. Consequently, you delete them in a shell by using
"unset".
unset name
There is a special value, called the NULL value, and it is assigned to a
variable by doing
name=
with no expression. This also undefines the variable, i.e. the variable in
some sense no longer exists. If you use the variable's name in a test state-
ment you can tell if it is set or not, i.e. if it has a non-NULL value:
x=5
x=
if test $x
then echo x is set
else echo x is not set
fi
In the above, x was set to 5, then unset. The if statement will print out
that x is not set.
To give a character value to a variable, you can use double quotes or you
can forego them. If the character string contains special characters, such as
a blank, then you must use double quotes. Here are some examples:
name=Mark
echo $name
if test $name = Mark
then ...
name="Mark Meyer"
echo $name
dirname=/usr/local/doc/HELP
ls $dirname
To change a variable's value, just use set again, but do not use $.
dirname=/mnt1/dept/glorp
To add on to an existing character string variable, you can do something like
the following:
sentence="Hi"
sentence="$sentence there everybody"
Now $sentence, if echoed, would have "Hi there everybody" in it. The following
also works:
sentence=Hi
sentence="$sentence there everybody"
There is a special variable called $$ which has the process id number of the
process that is running this shell script. Many programmers use this to create
unique file names, often the in the /tmp directory. Here's an example of copy-
ing the first parameter (which is obviously a filename) into a temp file whose
name uses the pid number:
cp $1 /tmp/tempfile.$$
This will create a file whose name is something like /tmp/tempfile.14506, if
the pid number is 14506.
Actually, the computer cycles through the pid numbers eventually, but usually
the same pid does not occur for several days, so there is seldom any need to
worry.
-------------------------------------------------------------------------------
Use of variables in the shell
-------------------------------------------------------------------------------
One of the nice features about shell programming is that there is no clear
line between what you can do in a shell script and what you can type in from
the prompt. Thus, you can assign values to variables, use for loops and do all
sorts of things at the command prompt. Some things will not work, like using
the parameters $1, $2, etc because there are none. But other features can be
used, like setting variables is quite handy, especially when you want to use
a long, complex pathname repeatedly:
% X=/usr/local/doc/HELP
% ls $X
% ls $X/TUTORIALS
You can even embed the shell variables inside other strings, as shown above in
$X/TUTORIALS. Obviously, you cannot follow the shell variable with a string
that begins with an alphabetic or numeric character because the shell will
not know which variable you are talking about, such as $XHITHERE.
-------------------------------------------------------------------------------
Arithmetic variables
-------------------------------------------------------------------------------
The Bourne shell does not have arithmetic variables, but there is another UNIX
command that can be used, called "expr" for arithmetic expression. Here's how
it might be used to increment a variable:
x=47
x=`expr $x + 1`
Notice the backquotes that surround the expr command, which cause UNIX to
actually execute the command and then replace it with the standard output that
it produces.
But the use of special characters in expr, like * and &, causes trouble, so
they must be escaped with a backslash. For example, the following script
prints out the powers of 2:
n=0
n2=1
while :
do
echo $n $n2
n=`expr $n + 1`
n2=`expr $n2 \* 2`
done
Here are the operators you can use without the escape character:
+, -, /, % Usual meanings (% is modulus)
The following operators need the escape:
*, |, &, (, )
The parentheses used for grouping to override precedence. The vertical bar is
used as a sort of OR, and the & symbol is an AND. These are actually short
circuit operators, which cause the least amount of work for the computer.
For example, in
expr1 | expr2
If expr1 is neither null (the empty string) nor 0, then expr1 is the returned
value, and expr2 is not evaluated. On the other hand, if expr1 is null or 0,
then expr2 is evaluated and its result returned. A similar method applies
to &:
expr1 & expr2
If expr1 is null or 0, then expr2 is not performed. If expr1 is not null, then
expr2 is performed and its value returned.
The expr command can also do extensive pattern matching, substring extraction
and other string functions. See the man page for these.
-------------------------------------------------------------------------------
Expressions and true and false
-------------------------------------------------------------------------------
The Bourne shell works slightly different from C or the Cshell. It does not
think of the value 0 as false. Instead, it uses NULL. Thus, the following
script fragment produces an infinite loop because it will not steop when n
becomes 0:
n=5
while test $n
do echo $n
n=`expr $n - 1`
done
Instead, you would have to change the test statement to
while test $n -ge 0
The two special UNIX commands "true" and "false" are used to provide logical
constants for the benefit of shell programs. Neither command "does" anything
except return the proper status code.
To get out of a loop, use break or exit. Of course, exit also causes the
entire shell script to end! In the following while statement, the user is
asked to type in something. If 0 is entered, then the while loop ends.
while true
do echo -n "Gimme something: "
read x
if test $x
break
fi
done
There is also a continue statement that can be used in while loops to bail out
of the current iteration of the loop and start the next one immediately.
-------------------------------------------------------------------------------
Input and output
-------------------------------------------------------------------------------
Output is fairly simple. You can use echo to show literals and variable values.
If you do not want to cause a newline to be printed, use -n. This is especial-
ly valuable in prompts, as in the while loop in the last section.
echo "Hi there world"
echo -n "Please type in your name: "
echo "The current directory is " $cwd
$cwd is the current working directory, and is a built-in variable (discussed
next).
To get something from the user, use the read command. This causes the shell to
pause until the user types a carriage return. What the user typed before the
RETURN is the value that read returns.
read x
You can also read more than one variable at a time:
read x y z
Of course, if you expect to get something intelligent from the user, make sure
to prompt her for what type of information you are requesting!
-------------------------------------------------------------------------------
Built-in variables
-------------------------------------------------------------------------------
There are many built-in variables, like $USER and $HOME. To see what they are
do
set
There are a few built-in variables, like $cwd and $HOME. $Cwd is the current
working directory, what you see when you use "pwd". $HOME is the home
directory. Here are others:
$user -- who am i?
$hostess -- name of the computer I am logged on to
$path -- my execution path (list of directories to be searched
for executables)
$term -- what kind of terminal I am using
$status -- a numeric variable, usually used to retun error codes
$prompt -- what I am currently using for a prompt
$shell -- which shell am I using (usu. either /bin/csh or /bin/sh)
-------------------------------------------------------------------------------
Case statements
-------------------------------------------------------------------------------
The case statement provides a multi-way branch, much as in C. Here's the
general format:
case expression in
val1) commands ;;
val2) commands ;;
...
esac
Notice that TWO semicolons are used instead of break, unlike C. Humorously,
the Bourne shell uses an old method of choosing ending keywords. Thus we see
"esac" as "case" backwards, and "fi" as "if" backwards. Thank heavens we do
not end our while statements with "elihw"! This is actually an atavism from
the old Algol days.
Quite often case statements are used to process flags.
for i in $*
do case $i in
-t) echo "do something for the -t option" ;;
-r) echo "do something for the -r option, etc" ;;
-*) echo "ERROR, unknown parameter!"
esac
done
-------------------------------------------------------------------------------
Here document
-------------------------------------------------------------------------------
We know how > and < work in I/O redirection. There is a use for >>, to append
data to the end of an existing file. What about < tempdata <
-------------------------------------------------------------------------------
Executing commands
-------------------------------------------------------------------------------
Occasionally we need to execute a command inside another command in order to get
its output or its return code. To get the output, use the backquotes. For
example, the following could be put inside a shell script:
echo "Hello there `whoami`. How are you today?"
echo "You are currently using `hostname` and the time is `date`"
echo "Your directory is `pwd`"
Of course all of these commands have equivalents in shell variables except the
date command. Following is a better example:
echo "There are `wc -l $1` lines in file $1"
Another use of commands is to use their return codes inside conditional expres-
sions. For example, the -s option of the grep command stands for silent mode.
It causes grep to do its job without producing any output, but the return code
is then used. You cannot see the return code, but you can use it if you sur-
round the command in curly braces:
if grep -s junk $1
then echo "We found junk in file $1"
fi
The return code of a shell script is set by the exit statement, which can take
an integer argument:
exit -1
exit 0
exit 12
Good script programmers follow the convention that 0 means "all ok" while a
non-zero value indicates some error code. If you use "exit" with no argument,
0 is assumed.
The return code of a C program is set by the exit() system call, which also
takes an integer argument. The same convention is followed that 0 is "all ok".
-------------------------------------------------------------------------------
Recursion
-------------------------------------------------------------------------------
Shell scripts can be recursive. The reason why this works is that each UNIX
command is started in its own shell, with its own process and its own process
id. This is also the reason why shell scripts run so much more slowly, because
starting processes is slow. So if the same shell script filename appears
inside itself, UNIX just blindly starts up another process and runs the shell
in it, interpreting the commands in the file.
Recursive shell scripts are very common when the script is naturally recursive
with regard to the tree structure. Many in-built UNIX commands allow the -R
option to specify that the command is done recursively to all components of
the directory:
% ls -RC /
As an example of a recursive shellscript, here's one that prints the head of
each file in the current directory and in every subdirectory. Let us suppose
that this shellscript is in a file called "headers":
#
cd $1
for i in *
do if test -f $i
then echo "============= $i ==================="
head $i
fi
if test -d $i
then (cd $i; headers)
fi
done
Note the use of parentheses in (cd $i;headers). The parentheses here mean to
do the commands in a new shell, for the cd command normally changes the current
directory, which would be disastrous for later functioning of the script, which
would have no way to return to the previous directory when it finished. But
isolating the commands in its own shell makes this secure and modular.
To run headers, just do
% cd whatever dir you want...
% headers
-------------------------------------------------------------------------------
Debugging
-------------------------------------------------------------------------------
There is not much support for debugging in shell scripts. You can always rely
on the good old standard way of debugging: peppering your code with output
statements, echo in this case, to see what is going on. To deactivate some of
them without getting rid of them, comment them out by putting a pound sign in
column 1.
About the only other support for debugging is using some options to the shell.
-v is verbose and -x echoes the commands after ALL substitutions are made. In
order to use these, however, you cannot run your shellscript by just typing in
the name followed by arguments. Rather, you must give the name of the file as
an argument itself to the csh command, followed by the arguments to your own
script:
% sh -vx somescript args
Both options are needed because the shell does its variable substitutions after
it reads each line. -vx causes both the original line from the script file to
be printed, as well as the revised form after the substitutions are made.
Another handy option is -n, which parses the script commands without execution
in order to check for shell syntax errors.
% sh -n somescript
-------------------------------------------------------------------------------
Performance considerations
-------------------------------------------------------------------------------
Shell scripts are interpreted in UNIX. That is, there does not exist a
compiler to translate the code into machine language, such as the C or Ada
compiler does. As you might have heard, interpreted languages tend to be
very slow in execution speed, so do not write a numerical analysis program in
the Bourne shell script language! Most shell scripts run very slowly.
The interpreter of shell scripts is the /bin/sh program itself. The cshell
"knows" when it is executing commands from a file as opposed to reading them
from the user sitting at a terminal, but it is still the same interpreter
program.
Whenever you run a command in a shell, a new copy of the shell is started up
(or "forked off", to use proper UNIX lingo). The new copy is actually another
process that is also running the sh interpreter. Unlike the C shell, the
Bourne shell does not read any initialization file when it starts a script,
although it does read .profile upon logging in. Thus, the Bourne shell tends
to be faster when running script files.
Sometimes you will see a pound sign on the first line, followed by the name of
the program which is supposed to interpret this file. Thus, you might see
#/bin/sh
The comment symbols (#) is actually a special UNIX symbol that means "the name
of the program to interpret this file follows me". Thus, /bin/sh appears be-
cause it is the interpreter for this file.
You can even put options to the interpreter on this line. For instance if
you wanted the shell script lines echoed for purposes of debugging, you could
use instead:
#/bin/sh -vx
Generally shell scripts are either short or are used because it is too clumsy
to write a C program to make all the file decisions that need to be made.
Shell programming is a convenience, and it has a clearly defined niche, but
that niche is not general purpose problem solving such as you might use C or
Ada for.
-------------------------------------------------------------------------------
Learning more about shell programming
-------------------------------------------------------------------------------
We have only just skimmed the surface! Shell programming is about as deep and
as complex as any other kind of programming. Indeed you could write all sorts
of programs in Bourne shell script language, but they would be terribly slow
compared to their C or Pascal counterparts.
Many important topics have been omitted from this tutorial, such as the role
of environment variables and how variables' values are either inherited or
lost. But with this tutorial almost 920 lines long, some cuts had to be made!
Whole books have been written about shell programming, although most of them
focus on the Bourne shell, which is still widely used. One of the best books
on Bourne shell programming is
"UNIX shell programming" by Stephen G. Kochan and Patrick H. Wood,
Hayden Book Co., 1985.
Bouwhuis CALL NUMBER: QA76.76.063 K64 1985
Another classic
"The UNIX System" by S. R. Bourne, Addison-Wesley, 1983.
One of the best ways to learn about any programming language is just to read
other people's programs and try to discover how the elements of the language
are being used. If you stumble across an unfamiliar item, look it up in a
reference book or the man page. (Most of the sh man page is devoted to the
minutiae of shell programming.) You can look at some of the scripts in
the public directory /usr/local/bin, for starters.