(GNU) Bash Scripting Introduction by Steve The Basics ---------- Help ---- man cmd -- Look at the MANual page for cmd help builtin_cmd -- Displays a brief help summary for Bash built in command, builtin_cmd. No argument lists builtins Redirection ----------- cmd > output.txt -- Send the output of cmd to output.txt cmd < input.txt -- Read the input of cmd from input.txt cmd >> appendedoutput.txt -- Append the output of cmd to appendedoutput.txt cmd 2> erroroutput.txt -- Send the error output of cmd to erroroutput.txt cmd > combined.txt 2>&1 -- Send the output, and the error output of cmd to combined.txt cmd1 | cmd2 -- Send the output of cmd1 to the input of cmd2 Script Files ------------ Comments - "#" -- An # (octothorp, commonly called a hash) starts a comment. command separator - ";" -- A ; (semicolon) separates two commands on one line. #!/bin/bash -- This 'magic comment' should be the first line of your Bash shell scripts. The "#!" prefix tells the shell to use the program directly following to interpret the contents of the file. Debugging -- It is often very useful to see how the shell expanded, and is running the commands in your shell script for debugging purposed. With the command "set -x", the shell will print each command it executes before it does. Look at "help set" for many other shell parameters you can manipulate. Tools ----- echo -- Echo takes its arguments, and prints them to its output. echo Hello World cat -- Cat opens the files listed as arguments, and prints their content to its output. cat helloworld.txt head -- Prints the beginning (head) of the file specified, or if no file is specified, the input to head. If no options are specified, 10 lines are printed. With the -n option, you can specify how many lines you would like printed from the beginning of the input. tail -- Just like head, but prints the last X lines. grep -- Grep takes one or two arguments. The first argument is a pattern, the optional second argument is a list of files. Grep matches the pattern against lines in the files, or its input if no file is specified, and then prints the matching lines to it's output. grep Hello helloworld.txt If the -v option is given to grep, it will print lines that do not match. grep -v zimbabway helloworld.txt The format for the pattern that grep accepts is a very extended form of regular expressions. Here are some basic concepts: a - match a (or any non-special character you choose) . - match any character * - match zero or more copies of the previous entity + - match one or more copies of the previous entity {m,n} - match at least m copies, but not more then n copies of the previous entity. Either argument may be omitted. ^ - match the beginning of a string $ - match the end of a string [] - match any character from within the []'s (you can do ranges of characters, i.e. [a-z], and negation [^a-z] ) () - make an entity. i.e. (abba)+ sed -- Sed is a stream editor, but it is most useful for doing substitutions on its input. Like grep, sed takes one or two arguments, a substitution pattern and an optional file. If no file is specified, its input is used. The result is printed to sed's output. The substitution pattern has three parts, "s/" to tell sed to do a substitution, "pattern/" to match part of the string, and "replacementstring/" to tell sed what to put in the place. echo Hello w0R1d. | sed s/0R1/orl/ The pattern part of the substitution accepts most of the same syntax as grep. However, ()'s save the value matched inside them for latter use in the replacement string. The ith set of ()'s in the pattern can be inserted into the replacement string with \i echo Hello w0R1d. | sed s/^(.*)0R1/\1orl/ Except this won't work -- we'll get to why in the section on quoting, but this what you need to do to actually make it work echo Hello w0R1d. | sed 's/^\(.*\)0R1/\1orl/' cut -- Cut extracts columns from lines. It accepts some options, and then a list of files. If no files are specified, it operates on the input to cut. The most typical options are -dX, and -fY. -dX changes the delimiter to X. For example -d" " changes the delimiter to space. -fY tells cut to print the Yth fields. Y may be a comma separated list, and may include ranges. echo "word1 word2 word3 word4 word5 word6" | cut -d" " -f1,3-4,6 tr -- Tr does translations. It will only read from its input, and write to its output. If two sets of characters are specified, it will change characters from the first set into characters in the second set. echo hELLO WORLD | tr A-Za-z a-zA-Z With the -d option, it will delete characters from the first set. And with the -s option, it will squeeze multiple instance of characters from the first set into a single instance. This is very useful when using cut with input that has a variable number of spaces between the columns. echo "Hello world" | tr -s " " find -- Find searches the file system for files. The find command is very extensive, it even has provisions for formatting its output. There are a few common ways in which I use find: find start/dir -name "*.ext" This will start searching in the directory start/dir, and search for any files that match *.ext. The quotes are needed because otherwise, the shell would expand the *.ext based on files from the current directory. find start/dir -type X This will start searching in the directory start/dir, and search for any files that match the given type. X can be: f=regular file d=directory l=symbolic link, etc. This (and the other find commands) can be combined with logical parameters, i.e. exp1 -and exp2 find start/dir -type f -and -name "*.mp3" Variables --------- Regular ------- Variables are defined with a statement such as variable=value. There must be no spaces on either side of the equal sign, otherwise the shell will try and execute one of the two words as a command. Variables can then later be used by prefixing the variable name with a dollar sign. a=abc echo $a Variables defined in this manner exist until the termination of the current shell, or until they are removed with the unset command. However, variables are not put into the environments of children of the shell unless they are exported. export b=def unset b Positional ---------- When a shell script is started, any additional arguments are assigned to the positional variables. For example, "script.sh arg1 arg2 arg3" would have "arg1" assigned to 1, referred to by $1, "arg2" assigned to 2, etc. Furthermore, these positional parameters are reassigned if functions within the shell script are called. Special ------- $0 -- The name of the script or function called (Think C style) $* -- All the positional parameters, i.e. $1 $2 $3 etc. $# -- The number of positional parameters $? -- Exit code of most recently executed process $$ -- PID of the current shell Quotes, Expansion, and Escaping ------------------------------- Quotes play a very important role in how *exactly* things are expanded in your shell. Sometimes it doesn't matter exactly what quotes are used, sometimes you can spend hours trying to achieve one effect or another, which is accomplished by changing what kind of quoting you are using. The set of metacharacters (characters that have a special meaning to the shell), must be quoted if you want their literal value in a string. Meta- characters include: | & ; ( ) < > space tab " ' ` \ $ \ -- Escape the next value. This tells the shell to ignore any special meaning that the character after the \ might have. With the one exception of \, which is treated as a line continuation. ' ' -- Single quotes preserve the literal value of all characters between them. A single quote may not appear in the string. " " -- Double quotes preserve the literal value of all characters except $, `, and \. $ and ` retain their special meaning (variable names and command substitution). The backslash retains its special meaning (escaping the next character) only when followed by one of: $, `, ", \, or . You can escape a double quote: "He said, \"Hi.\"" It's worth noting that if a variable contains multiple words, with multiple spaces, or other whitespace between them, you can preserve the white space by quoting the variable. a="abc 123" echo $a -> abc 123 echo "$a" -> abc 123 ` ` -- Back quotes do not act as quotes at all, but do command substitution. What ever command is between the `'s is executed, and the result is put in the place of the back quoted string. `echo cat helloworld.txt` { } -- Curly braces have a meaning in the context of file wild cards, but that is not covered here. {}'s are needed in a couple of contexts. If you want to use a positions variable with an index greater then 9, you need to use {}'s, i.e. ${13}. You also need {}'s around a variable name when you want to put it right next to another word, i.e. echo 123${middle}abc. Without the {}'s the variable would have been $middleabc, not $middle. $(( )) -- Double parenthesis mean math mode. Mathematical expressions are evaluated in $(( ))'s. echo 1+1 is $(( 1 + 1 )) Conditionals, Loops, and Functions ---------------------------------- Conditionals ------------ The Bash if statement is similar to most other languages, though may be a little more verbose. Aside from the IF keyword, the predicate, and the body, there's also a THEN keyword, and a FI termination keyword. The predicate is any arbitrary command. If the exit code of the predicate is zero the THEN (predicate is true) clause is executed. if COMMANDS then COMMANDS [ elif COMMANDS then COMMANDS ] ... [ else COMMANDS ] fi The test command is useful as a predicate. There are many test conditions, and on most systems the command [ is symlinked to test. The most useful test types are listed below. A full list can be obtained from man test or help test. The test command will expect a closing ] as the last argument if it was executed through the [ alias. -z STRING -- True if string is empty -n STRING -- True if string is not empty STRING1 = STRING2 -- True if the strings are equal number OP number -- Does arithmetic testings. OP is one of -eq -ne -lt -le -gt or -ge ! EXPR -- Negates the value of EXPR EXPR1 -a EXPR2 -- Returns the logical and of EXPR1 and EXPR2 EXPR1 -o EXPR2 -- Returns the logical or of EXPR1 and EXPR2 Pitfalls: Test expects exactly the number of arguments required. If you are using a variable as one or both arguments, it's best to quote the variable in case the variable ends up being empty, or contains a space. [ abc = $var ] -- will give an error if $var is empty, or has a space [ 5 -eq $var ] -- will fail if $var is not strictly a number [abc = "$var" ] -- will fail because "[abc" is not a command, similarly [$var = abc ] will also fail [ "$var" = abc] -- will fail because it considers abc] one world, and there is no closing ] [ abc= "$var" ] -- will fail because it only sees two parameters, "abc=", and the value of $var. Remember test is just a regular command, and relies on the shell to parse the arguments. Loops ----- while -- while COMMANDS; do COMMANDS; done The syntax for the while is straightforward. Though I haven't found the while loop to be very common. The only case when I use it is when I want to loop forever: while true; do COMMANDS; done for -- for NAME in WORDS ... ; do COMMANDS; done In the form above, the for construct is a bit different then typically thought of. In this form it's useful to iterate over a set of arbitrary items, instead of assigning consecutive integers to a variable. NAME is the variable that WORDS are bound to for each iteration of the loop executing COMMANDS. for i in A B C; do echo a${i}a; done It is often useful to put a command substitution (`cmd`) in the WORDS location. for i in `ls *.ext`; do mv $i `echo $i | sed "s/ext/newext/"`; done However, the more conventional usage is also available in conjunction with arithmetic evaluation for (( setup_expr; test_condition; increment_expr)); do COMMANDS; done for ((i=0; $i<10; i=$i+1)); echo $i; done Functions --------- While most quick shells scripts won't be terribly structured, if you find yourself using the same construct over and over, it may save typing to make it a function. There are two possible syntactic formats for a function, which are completely equivalent. function NAME { COMMANDS ; } NAME () { COMMANDS ; } As mentioned above, the positions parameters are reassigned when a a function is invoked. Conclusion ---------- Bash shell scripting is really just a little bit of syntax on top of the Unix tools. The more you can do with the tools, the more you can do in your shell scripts. Most people resort to a PERL script as soon as something as simple as a for loop is needed, but the interactive nature of shell scripting can still solve the problem faster in some cases. My advice for learning shell scripting is to first of all, actually use it, and once you are comfortable with most of the techniques presented in this tutorial, read through the Bash man page. You will pick up many new tricks. Once you're comfortable with them, read it again, etc. A good Bash scripting reference can be found here: http://www.tldp.org/LDP/abs