Advanced Bash-Scripting HOWTO
A guide to shell scripting, using Bash
Mendel Cooper
[email protected]
v0.2, 30 October 2000
This is a major update on version 0.1. -- a couple of bugs fixed, plus
much additional material and more example scripts added.
This document is both a tutorial and a reference on shell scripting
with Bash. It assumes no previous knowledge of scripting or
programming, but progresses rapidly toward an intermediate/advanced
level of instruction. The exercises and heavily-commented examples
invite active reader participation. Still, it is a work in progress.
The intention is to add much supplementary material in future updates
to this HOWTO, so that it will gradually evolve into an LDP "guide",
i.e., a complete book.
_________________________________________________________________
Table of Contents
1. [1]Why Shell Programming?
2. [2]Starting Off With a Sha-Bang
2.1. [3]Invoking the script
2.2. [4]Shell wrapper, self-executing script
3. [5]Tutorial / Reference
3.1. [6]exit and exit status
3.2. [7]Special characters used in shell scripts
3.3. [8]Introduction to Variables and Parameters
3.4. [9]Quoting
3.5. [10]Tests
3.6. [11]Operations and Related Topics
3.7. [12]Variables Revisited
3.8. [13]Loops
3.9. [14]Internal Commands and Builtins
3.10. [15]External Filters, Programs and Commands
3.11. [16]System and Administrative Commands
3.12. [17]Backticks (`...`)
3.13. [18]I/O Redirection
3.14. [19]Recess Time
3.15. [20]Regular Expressions
3.16. [21]Subshells
3.17. [22]Process Substitution
3.18. [23]Functions
3.19. [24]List Constructs
3.20. [25]Arrays
3.21. [26]Files
3.22. [27]Here Documents
3.23. [28]Of Zeros and Nulls
3.24. [29]Debugging
3.25. [30]Options
3.26. [31]Gotchas
3.27. [32]Miscellany
3.28. [33]Bash, version 2
4. [34]Credits
[35]Bibliography
A. [36]Contributed Scripts
B. [37]Copyright
List of Tables
3-1. [38]bash options
List of Examples
2-1. [39]cleanup: A script to clean up the log files in /var/log
2-2. [40]cleanup: An enhanced and generalized version of above script.
2-3. [41]shell wrapper
2-4. [42]A slightly more complex shell wrapper
3-1. [43]exit / exit status
3-2. [44]Code blocks and I/O redirection
3-3. [45]Saving the results of a code block to a file
3-4. [46]Backup of all files changed in last day
3-5. [47]Variable assignment and substitution
3-6. [48]Using param substitution and :
3-7. [49]Renaming file extensions:
3-8. [50]Using pattern matching to parse arbitrary strings
3-9. [51]What is truth?
3-10. [52]Equivalence of [ ] and test
3-11. [53]Tests, command chaining, redirection
3-12. [54]arithmetic and string comparisons
3-13. [55]zmost
3-14. [56]Compound Condition Tests Using && and ||
3-15. [57]Representation of numerical constants:
3-16. [58]Variable Assignment
3-17. [59]Variable Assignment, plain and fancy
3-18. [60]Positional Parameters
3-19. [61]wh, whois domain name lookup
3-20. [62]Using shift
3-21. [63]Using declare to type variables
3-22. [64]Indirect References
3-23. [65]Generating random numbers
3-24. [66]Simple for loops
3-25. [67]Missing in [list] in a for loop
3-26. [68]Using efax in batch mode
3-27. [69]Simple while loop
3-28. [70]Another while loop
3-29. [71]until loop
3-30. [72]Effects of break and continue in a loop
3-31. [73]Using case
3-32. [74]Creating menus using case
3-33. [75]Creating menus using select
3-34. [76]Creating menus using select in a function
3-35. [77]Using getopts to read the flags/options passed to a script
3-36. [78]Using set with positional parameters
3-37. [79]basename and dirname
3-38. [80]Variable assignment, using read
3-39. [81]Changing the current working directory
3-40. [82]"Including" a data file
3-41. [83]Waiting for a process to finish before proceeding
3-42. [84]Using ls to create a table of contents for burning a CDR
disk
3-43. [85]Badname, eliminate file names in current directory
containing bad characters and white space.
3-44. [86]Log file using xargs to monitor system log
3-45. [87]copydir, copying files in current directory to another,
using xargs
3-46. [88]Showing the effect of eval
3-47. [89]Forcing a log-off
3-48. [90]Using expr
3-49. [91]Letting let do some arithmetic.
3-50. [92]printf in action
3-51. [93]Using cpio to move a directory tree
3-52. [94]toupper: Transforms a file to all uppercase.
3-53. [95]lowercase: Changes all filenames in working directory to
lowercase.
3-54. [96]nl: A self-numbering script.
3-55. [97]Formatted file listing.
3-56. [98]Using date
3-57. [99]uuencoding encoded files
3-58. [100]Using seq to generate loop arguments
3-59. [101]Effects of exec
3-60. [102]killall, from /etc/rc.d/init.d
3-61. [103]Perl embedded in a bash script
3-62. [104]Variable scope in a subshell
3-63. [105]Running parallel processes in subshells
3-64. [106]Simple function
3-65. [107]Function Taking Parameters
3-66. [108]Converting numbers to Roman numerals
3-67. [109]Local variable visibility
3-68. [110]Recursion, using a local variable
3-69. [111]Using an "and list" to test for command-line arguments
3-70. [112]Using "or lists" in combination with an "and list"
3-71. [113]Simple array usage
3-72. [114]Some special properties of arrays
3-73. [115]An old friend: The Bubble Sort
3-74. [116]Complex array application: Sieve of Erastosthenes
3-75. [117]dummyfile: Creates a 2-line dummy file
3-76. [118]broadcast: Sends message to everyone logged in
3-77. [119]Multi-line message using cat
3-78. [120]upload: Uploads a file pair to "Sunsite" incoming directory
3-79. [121]Setting up a swapfile using /dev/zero
3-80. [122]test23, a buggy script
3-81. [123]test24, another buggy script
3-82. [124]Trapping at exit
3-83. [125]Cleaning up after Control-C
3-84. [126]String expansion
3-85. [127]Indirect variable references - the new way
3-86. [128]Using arrays and other miscellaneous trickery to deal four
random hands from a deck of cards
A-1. [129]manview: A script for viewing formatted man pages
A-2. [130]manview: A script for uploading to an ftp site, using a
locally encrypted password
A-3. [131]behead: A script for removing mail and news message headers
A-4. [132]ftpget: A script for downloading files via ftp
_________________________________________________________________
Chapter 1. Why Shell Programming?
The shell is a command interpreter. It is the insulating layer between
the operating system kernel and the user. Yet, it is also a fairly
powerful programming language. A shell program, called a script , is
an easy-to-use tool for building applications by "gluing" together
system calls, tools, utilities, and compiled binaries. Virtually the
entire repertoire of UNIX commands, utilities, and tools is available
for invocation by a shell script. If that were not enough, internal
shell commands, such as testing and loop constructs, give additional
power and flexibility to scripts. Shell scripts lend themselves
exceptionally well to to administrative system tasks and other routine
repetitive jobs not requiring the bells and whistles of a full-blown
tightly structured programming language.
A working knowledge of shell scripting is essential to everyone
wishing to become reasonably adept at system administration, even if
they do not anticipate ever having to actually write a script.
Consider that as a Linux machine boots up, it executes the shell
scripts in /etc/rc.d to restore the system configuration and set up
services. A detailed understanding of these scripts is important for
analyzing the behavior of a system, and possibly modifying it.
Writing shell scripts is not hard to learn, since the scripts can be
built in bite-sized sections and there is only a fairly small set of
shell-specific operators and options to learn. The syntax is simple
and straightforward, similar to that of invoking and chaining together
utilities at the command line, and there are only a few "rules" to
learn. Most short scripts work right the first time, and debugging
even the longer ones is straightforward.
A shell script is a "quick and dirty" method of prototyping a complex
application. Getting even a limited subset of the functionality to
work in a shell script, even if slowly, is often a useful first stage
in project development. This way, the structure of the application can
be tested and played with, and the major pitfalls found before
proceeding to the final coding in C, C++, Java, or Perl.
Shell scripting hearkens back to the classical UNIX philosophy of
breaking complex projects into simpler subtasks, of chaining together
components and utilities. Many consider this a better, or at least
more esthetically pleasing approach to problem solving than using one
of the new generation of high powered all-in-one languages, such as
Perl, which attempt to be all things to all people, but at the cost of
forcing you to alter your thinking processes to fit the tool.
When not to use shell scripts
* resource-intensive tasks, especially where speed is a factor
* complex applications, where structured programming is a necessity
* file handling (Bash is limited to serial file access, and that
only in a particularly clumsy and inefficient line-by-line
fashion)
* need to generate or manipulate graphics or GUIs
* need direct access to system hardware
* need port or socket I/O
* need to use libraries or interface with legacy code
If any of the above applies, consider a more powerful scripting
language, perhaps Perl, Tcl, Python, or even a high-level compiled
language such as C, C++, or Java. Even then, prototyping the
application as a shell script might still be a useful development
step.
We will be using Bash, an acronym for "Born-Again Shell" and a pun on
Stephen Bourne's now classic Bourne Shell. Bash has become the de
facto standard for shell scripting on all flavors of UNIX. Most of the
principles dealt with in this document apply equally well to scripting
with other shells, such as the Korn Shell, from which Bash derives
some of its features, and the C Shell and its variants. (Note that C
Shell programming is not recommended due to certain inherent problems,
as pointed out in a [133]news group posting by Tom Christiansen in
October of 1993).
The following is a tutorial in shell scripting. It relies heavily on
examples to illustrate features of the shell. As far as possible, the
example scripts have been tested, and some of them may actually be
useful in real life. The reader should cut out and save the examples,
assign them appropriate names, give them execute permission (chmod u+x
scriptname), then run them to see what happens. Note that some of the
scripts below introduce features before they are explained, and this
may require the reader to temporarily skip ahead for enlightenment.
Unless otherwise noted, the author of this document wrote the example
scripts that follow.
_________________________________________________________________
Chapter 2. Starting Off With a Sha-Bang
In the simplest case, a script is nothing more than a list of system
commands stored in a file. At the very least, this saves the effort of
retyping that particular sequence of commands each time it is invoked.
Example 2-1. cleanup: A script to clean up the log files in /var/log
# cleanup
# Run as root, of course.
cd /var/log
cat /dev/null > messages
cat /dev/null > wtmp
echo "Logs cleaned up."
There is nothing unusual here, just a set of commands that could just
as easily be invoked one by one from the command line on the console
or in an xterm. The advantages of placing the commands in a script go
beyond not having to retype them time and again. The script can easily
be modified, customized, or generalized for a particular application.
Example 2-2. cleanup: An enhanced and generalized version of above
script.
#!/bin/bash
# cleanup, version 2
# Run as root, of course.
if [ -n $1 ]
# Test if command line argument present.
then
lines=$1
else
lines=50
# default, if not specified on command line.
fi
cd /var/log
tail -$lines messages > mesg.temp
# Saves last section of message log file.
mv mesg.temp messages
# cat /dev/null > messages
# No longer needed, as the above method is safer.
cat /dev/null > wtmp
echo "Logs cleaned up."
exit 0
# A zero return value from the script upon exit
# indicates success to the shell.
Since you may not wish to wipe out the entire system log, this variant
of the first script keeps the last section of the message log intact.
You will constantly discover ways of refining previously written
scripts for increased effectiveness.
The sha-bang ( #!) at the head of a script tells your system that this
file is a set of commands to be fed to the command interpreter
indicated. The #! is actually a two byte " magic number", a special
marker that designates an executable shell script (man magic gives
more info on this fascinating topic). Immediately following the
sha-bang is a path name. This is the path to the program that
interprets the commands in the script, whether it be a shell, a
programming language, or a utility. This enables the specific commands
and directives embedded in the shell or program called.
#!/bin/sh
#!/bin/bash #!/bin/awk #!/usr/bin/perl #!/bin/sed
#!/usr/bin/tcl
Each of the above script header lines calls a different command
interpreter, be it /bin/sh, the default shell (bash in a Linux system)
or otherwise. Using #!/bin/sh, the default Bourne Shell in most
commercial variants of UNIX, makes the script portable to non-Linux
machines, though you may have to sacrifice a few bash-specific
features (the script will conform to the POSIX sh standard).
Note that the path given at the "sha-bang" must be correct, otherwise
an error message, usually Command not found will be the only result of
running the script.
#! can be omitted if the script consists only of a set of generic
system commands, using no internal shell directives. Example 2, above,
requires the initial #!, since the variable assignment line, lines=50,
uses a shell-specific construct. Note that #!/bin/sh invokes the
default shell interpreter, which defaults to /bin/bash on a Linux
machine.
_________________________________________________________________
2.1. Invoking the script
Having written the script, you can invoke it by sh scriptname, or
alternately bash scriptname. (Not recommended is using sh <scriptname,
since this effectively disables reading from input within the script.)
Much more convenient is to make the script itself directly executable
by
Either:
chmod 755 scriptname (gives everyone execute permission)
or
chmod +x scriptname (gives everyone execute permission)
chmod u+x scriptname (gives only the script owner execute
permission)
In this case, you could try calling the script by ./scriptname.
As a final step, after testing and debugging, you would likely want to
move it to /usr/local/bin (as root, of course), to make the script
available to yourself and all other users as a system-wide executable.
The script could then be invoked by simply typing scriptname [return]
from the command line.
_________________________________________________________________
2.2. Shell wrapper, self-executing script
A sed or awk script would normally be invoked from the command line by
a sed -e 'commands' or awk -e 'commands'. Embedding such a script in a
bash script permits calling it more simply, and makes it "reusable".
This also permits combining the functionality of sed and awk, for
example piping the output of a set of sed commands to awk. As a saved
executable file, you can then repeatedly invoke it in its original
form or modified, without retyping it on the command line.
Example 2-3. shell wrapper
#!/bin/bash
# This is a simple script
# that removes blank lines
# from a file.
# No argument checking.
# Same as
# sed -e '/^$/d $1' filename
# invoked from the command line.
sed -e /^$/d $1
# '^' is beginning of line,
# '$' is end,
# and 'd' is delete.
Example 2-4. A slightly more complex shell wrapper
#!/bin/bash
# "subst", a script that substitutes one pattern for
# another in a file,
# i.e., "subst Smith Jones letter.txt".
if [ $# -ne 3 ]
# Test number of arguments to script
# (always a good idea).
then
echo "Usage: `basename $0` old-pattern new-pattern filename"
exit 1
fi
old_pattern=$1
new_pattern=$2
if [ -f $3 ]
then
file_name=$3
else
echo "File \"$3\" does not exist."
exit 2
fi
# Here is where the heavy work gets done.
sed -e "s/$old_pattern/$new_pattern/" $file_name
# 's' is, of course, the substitute command in sed,
# and /pattern/ invokes address matching.
# Read the literature on 'sed' for a more
# in-depth explanation.
exit 0
# Successful invocation of the script returns 0.
Exercise. Write a shell script that performs a simple task.
_________________________________________________________________
Chapter 3. Tutorial / Reference
...there are dark corners in the Bourne shell, and people use all of
them.
--Chet Ramey
_________________________________________________________________
3.1. exit and exit status
The exit command may be used to terminate a script, just as in a C
program. It can also return a value, which is available to the shell.
Every command returns an exit status (sometimes referred to as a
return status ). A successful command returns a 0, while an
unsuccessful one returns a non-zero value that usually may be
interpreted as an error code.
Likewise, functions within a script and the script itself return an
exit status. The last command executed in the function or script
determines the exit status. Within a script, an exit nn command may be
used to deliver an nn exit status to the shell (nn must be a decimal
number in the 0 - 255 range).
$? reads the exit status of script or function.
Example 3-1. exit / exit status
#!/bin/bash
echo hello
echo $?
# exit status 0 returned
# because command successful.
lskdf
# bad command
echo $?
# non-zero exit status returned.
echo
exit 143
# Will return 143 to shell.
# To verify this, type $? after script terminates.
# By convention, an 'exit 0' shows success,
# while a non-zero exit value indicates an error or anomalous condition.
# It is also appropriate for the script to use the exit status
# to communicate with other processes, as when in a pipe with other scripts.
_________________________________________________________________
3.2. Special characters used in shell scripts
#
Comments. Lines beginning with a # (with the exception of #!) are
comments.
# This line is a comment.
Comments may also occur at the end of a command.
echo "A comment will follow." # Comment here.
Comments may also follow white space at the beginning of a
line.
# A tab precedes this comment.
;
Command separator. Permits putting two or more commands on the same
line
echo hello; echo there
Note that the ; sometimes needs to be escaped (\).
.
"dot" command. Equivalent to source, explained further on
:
null command. Exit status 0, alias for true
Endless loop:
while :
do
operation-1
operation-2
...
operation-n
done
Placeholder in if/then test:
if condition
then : # Do nothing and branch ahead
else
take-some-action
fi
Provides a placeholder where a binary operation is expected,
see [134]Section 3.3.1.
: ${username=`whoami`}
# ${username=`whoami`} without the leading : gives an error
Evaluate string of variables using "parameter substitution",
see [135]Example 3-6:
: ${HOSTNAME?} ${USER?} ${MAIL?}
Prints error message if one or more of essential environmental
variables not set.
${}
Parameter substitution.
See [136]Section 3.3 for more details.
()
command group.
(a=hello; echo $a)
Note: A listing of commands within parentheses starts a subshell
(see [137]Section 3.16).
{}
block of code. This, in effect, creates an anonymous function.
The code block enclosed in braces may have I/O redirected to
and from it.
Example 3-2. Code blocks and I/O redirection
#!/bin/bash
{
read fstab
} < /etc/fstab
echo "First line in /etc/fstab is:"
echo "$fstab"
exit 0
Example 3-3. Saving the results of a code block to a file
#!/bin/bash
# rpm-check
# ---------
# Queries an rpm file for description, listing, and whether it can be installed
