Differences between revisions 516 and 517
Revision 516 as of 2019-06-17 15:41:47
Size: 71998
Editor: GreyCat
Comment: pf59: i=$(( 10#$i ))
Revision 517 as of 2019-07-12 10:42:50
Size: 72163
Editor: 122
Comment: Add sudo tee file >/dev/null as alternative to wrapper
Deletions are marked like this. Additions are marked like this.
Line 1400: Line 1400:

Instead of a wrapper you can use `tee`:

sudo mycmd | sudo tee /myfile >/dev/null

This may be easier to write if `mycmd` has a lot of quoting.

Bash Pitfalls

This page is a compilation of common mistakes made by bash users. Each example is flawed in some way.

1. for f in $(ls *.mp3)


for f in $(ls *.mp3); do    # Wrong!
    some command $f         # Wrong!

for f in $(ls)              # Wrong!
for f in `ls`               # Wrong!

for f in $(find . -type f)  # Wrong!
for f in `find . -type f`   # Wrong!

files=($(find . -type f))   # Wrong!
for f in ${files[@]}        # Wrong!

Yes, it would be great if you could just treat the output of ls or find as a list of filenames and iterate over it. But you cannot. This entire approach is fatally flawed, and there is no trick that can make it work. You must use an entirely different approach.

There are at least 6 problems with this:

  1. If a filename contains whitespace, it undergoes WordSplitting. Assuming we have a file named 01 - Don't Eat the Yellow Snow.mp3 in the current directory, the for loop will iterate over each word in the resulting file name: 01, -, Don't, Eat, etc.

  2. If a filename contains glob characters, it undergoes filename expansion ("globbing"). If ls produces any output containing a * character, the word containing it will become recognized as a pattern and substituted with a list of all filenames that match it.

  3. If the command substitution returns multiple filenames, there is no way to tell where the first one ends and the second one begins. Pathnames may contain any character except NUL. Yes, this includes newlines.

  4. The ls utility may mangle filenames. Depending on which platform you're on, which arguments you used (or didn't use), and whether its standard output is pointing to a terminal or not, ls may randomly decide to replace certain characters in a filename with "?", or simply not print them at all. Never try to parse the output of ls. ls is just plain unnecessary. It's an external command whose output is intended specifically to be read by a human, not parsed by a script.

  5. The CommandSubstitution strips all trailing newline characters from its output. That may seem desirable since ls adds a newline, but if the last filename in the list ends with a newline, `...` or $() will remove that one also.

  6. In the ls examples, if the first filename starts with a hyphen, it may lead to pitfall #3.

You can't simply double-quote the substitution either:

for f in "$(ls *.mp3)"; do     # Wrong!

This causes the entire output of ls to be treated as a single word. Instead of iterating over each file name, the loop will only execute once, assigning to f a string with all the filenames rammed together.

Nor can you simply change IFS to a newline. Filenames can also contain newlines.

Another variation on this theme is abusing word splitting and a for loop to (incorrectly) read lines of a file. For example:

for line in $(cat file); do ...     # Wrong!

This doesn't work! Especially if those lines are filenames. Bash (or any other Bourne family shell) just doesn't work this way.

So, what's the right way to do it?

There are several ways, primarily depending on whether you need a recursive expansion or not.

If you don't need recursion, you can use a simple glob. Instead of ls:

for file in ./*.mp3; do    # Better! and...
    some command "$file"   # ...always double-quote expansions!

POSIX shells such as Bash have the globbing feature specifically for this purpose -- to allow the shell to expand patterns into a list of matching filenames. There is no need to interpret the results of an external utility. Because globbing is the very last expansion step, each match of the ./*.mp3 pattern correctly expands to a separate word, and isn't subject to the effects of an unquoted expansion.

Question: What happens if there are no *.mp3-files in the current directory? Then the for loop is executed once, with i="./*.mp3", which is not the expected behavior! The workaround is to test whether there is a matching file:

for file in ./*.mp3; do
    [ -e "$file" ] || continue
    some command "$file"

Another solution is to use Bash's shopt -s nullglob feature, though this should only be done after reading the documentation and carefully considering the effect of this setting on all other globs in the script.

If you need recursion, the standard solution is find. When using find, be sure you use it properly. For POSIX sh portability, use the -exec option:

find . -type f -name '*.mp3' -exec some command {} \;

# Or, if the command accepts multiple input filenames:


If you're using bash, then you have two additional options. One is to use GNU or BSD find's -print0 option, together with bash's read -d '' option and a ProcessSubstitution:

  some command "$file"
done < <(find . -type f -name '*.mp3' -print0)

The advantage here is that "some command" (indeed, the entire while loop body) is executed in the current shell. You can set variables and have them persist after the loop ends.

The other option, available in Bash 4.0 and higher, is globstar, which permits a glob to be expanded recursively:

shopt -s globstar
for file in ./**/*.mp3; do
  some command "$file"

Note the double quotes around $file in the examples above. This leads to our second pitfall:

2. cp $file $target

What's wrong with the command shown above? Well, nothing, if you happen to know in advance that $file and $target have no white space or wildcards in them. However, the results of the expansions are still subject to WordSplitting and pathname expansion. Always double-quote parameter expansions.

cp -- "$file" "$target"

Without the double quotes, you'll get a command like cp 01 - Don't Eat the Yellow Snow.mp3 /mnt/usb, which will result in errors like cp: cannot stat `01': No such file or directory. If $file has wildcards in it (* or ? or [), they will be expanded if there are files that match them. With the double quotes, all's well, unless "$file" happens to start with a -, in which case cp thinks you're trying to feed it command line options (See pitfall #3 below.)

Even in the somewhat uncommon circumstance that you can guarantee the variable contents, it is conventional and good practice to quote parameter expansions, especially if they contain file names. Experienced script writers will always use quotes except perhaps for a small number of cases in which it is absolutely obvious from the immediate code context that a parameter contains a guaranteed safe value. Experts will most likely consider the cp command in the title always wrong. You should too.

3. Filenames with leading dashes

Filenames with leading dashes can cause many problems. Globs like *.mp3 are sorted into an expanded list (according to your current locale), and - sorts before letters in most locales. The list is then passed to some command, which may incorrectly interpret the -filename as an option. There are two major solutions to this.


cp -- "$file" "$target"

There are potential problems with this approach. You have to be sure to insert -- for every usage of the parameter in a context where it might possibly be interpreted as an option -- which is easy to miss and may involve a lot of redundancy.

Most well-written option parsing libraries understand this, and the programs that use them correctly should inherit that feature for free. However, still be aware that it is ultimately up to the application to recognize end of options. Some programs that manually parse options, or do it incorrectly, or use poor 3rd-party libraries may not recognize it. Standard utilities should, with a few exceptions that are specified by POSIX. echo is one example.

Another option is to ensure that your filenames always begin with a directory by using relative or absolute pathnames.

for i in ./*.mp3; do
    cp "$i" /target


Finally, if you can guarantee that all results will have the same prefix, and are only using the variable a few times within a loop body, you can simply concatenate the prefix with the expansion. This gives a theoretical savings in generating and storing a few extra characters for each word.

for i in *.mp3; do
    cp "./$i" /target

4. [ $foo = "bar" ]

This is very similar to the issue in pitfall #2, but I repeat it because it's so important. In the example above, the quotes are in the wrong place. You do not need to quote a string literal in bash (unless it contains metacharacters or pattern characters). But you should quote your variables if you aren't sure whether they could contain white space or wildcards.

This example can break for several reasons:

  • If a variable referenced in [ doesn't exist, or is blank, then the [ command would end up looking like:

  • [ = "bar" ] # Wrong!
  • ...and will throw the error: unary operator expected. (The = operator is binary, not unary, so the [ command is rather shocked to see it there.)

  • If the variable contains internal whitespace, then it gets split into separate words before the [ command sees it. Thus:

  • [ multiple words here = "bar" ]
  • While that may look OK to you, it's a syntax error as far as [ is concerned. The correct way to write this is:

  • # POSIX
    [ "$foo" = bar ] # Right!
  • This works fine on POSIX-conformant implementations even if $foo begins with a -, because POSIX [ determines its action depending on the number of arguments passed to it. Only very ancient shells have a problem with this, and you shouldn't worry about them when writing new code (see the x"$foo" workaround below).

In Bash and many other ksh-like shells, there is a superior alternative which uses the [[ keyword.

# Bash / Ksh
[[ $foo == bar ]] # Right!

You don't need to quote variable references on the left-hand side of = in [[ ]] because they don't undergo word splitting or globbing, and even blank variables will be handled correctly. On the other hand, quoting them won't hurt anything either. Unlike [ and test, you may also use the identical ==. Do note however that comparisons using [[ perform pattern matching against the string on the right hand side, not just a plain string comparison. To make the string on the right literal, you must quote it if any characters that have special meaning in pattern matching contexts are used.

# Bash / Ksh
[[ $foo == "$match" ]] # Good! Unquoted would also match against the pattern b*r.

You may have seen code like this:

# POSIX / Bourne
[ x"$foo" = xbar ] # Ok, but usually unnecessary.

The x"$foo" hack is required for code that must run on very ancient shells which lack [[, and have a more primitive [, which gets confused if $foo begins with a -. On said older systems, [ still doesn't care whether the token on the right hand side of the = begins with a -. It just uses it literally. It's just the left-hand side that needs extra caution.

Note that shells that require this workaround are not POSIX-conforming. Even the Heirloom Bourne shell doesn't require this (probably the non-POSIX Bourne shell clone that's still most widely in use as a system shell). Such extreme portability is rarely a requirement and makes your code less readable (and uglier).

5. cd $(dirname "$f")

This is yet another quoting error. As with a variable expansion, the result of a CommandSubstitution undergoes WordSplitting and pathname expansion. So you should quote it:

cd -P -- "$(dirname -- "$f")"





[ bar = "$foo" ] && [ foo = "$bar" ] # Right! (POSIX)
[[ $foo = bar && $bar = foo ]]       # Also right! (Bash / Ksh)

(Note that we reversed the constant and the variable inside [ for the legacy reasons discussed in pitfall #4. We could also have reversed the [[ case, but the expansions would require quoting to prevent interpretation as a pattern.) The same thing applies to ||. Either use [[ instead, or use two [ commands.

Avoid this:


The binary -a and -o, and ( / ) (grouping) operators are XSI extensions to the POSIX standard. All are marked as obsolescent in POSIX-2008. They should not be used in new code. One of the practical problems with [ A = B -a C = D ] (or -o) is that POSIX does not specify the results of a test or [ command with more than 4 arguments. It probably works in most shells, but you can't count on it. If you have to write for POSIX shells, then you should use two test or [ commands separated by a && operator instead.

7. [[ $foo > 7 ]]

There are multiple issues here. First, the [[ command should not be used solely for evaluating arithmetic expressions. It should be used for test expressions involving one of the supported test operators. Though technically you can do math using some of [['s operators, it only makes sense to do so in conjunction with one of the non-math test operators somewhere in the expression. If you just want to do a numeric comparison (or any other shell arithmetic), it is much better to just use (( )) instead:

# Bash / Ksh
((foo > 7))     # Right!
[[ foo -gt 7 ]] # Works, but is pointless. Most will consider it wrong. Use ((...)) or let instead.

If you use the > operator inside [[ ]], it's treated as a string comparison (test for collation order by locale), not an integer comparison. This may work sometimes, but it will fail when you least expect it. If you use > inside [ ], it's even worse: it's an output redirection. You'll get a file named 7 in your directory, and the test will succeed as long as $foo is not empty.

If strict POSIX-conformance is a requirement, and (( is not available, then the correct alternative using old-style [ is

[ "$foo" -gt 7 ]       # Also right!
[ $((foo > 7)) -ne 0 ] # POSIX-compatible equivalent to ((, for more general math operations.

Note that the test ... -gt command will fail in interesting ways if $foo is not an integer. Therefore, there's not much point in quoting it properly other than for performance and to confine the arguments to a single word to reduce the likelihood of obscure side-effects possible in some shells.

If the input to any arithmetic context (including (( or let), or [ test expression involving numeric comparisons can't be guaranteed then you must always validate your input before evaluating the expression.

case $foo in
        printf '$foo expanded to a non-digit: %s\n' "$foo" >&2
        exit 1
        [ $foo -gt 7 ]


The code above looks OK at first glance, doesn't it? Sure, it's just a poor implementation of grep -c, but it's intended as a simplistic example. Changes to count won't propagate outside the while loop because each command in a pipeline is executed in a separate SubShell. This surprises almost every Bash beginner at some point.

POSIX doesn't specify whether or not the last element of a pipeline is evaluated in a subshell. Some shells such as ksh93 and Bash >= 4.2 with shopt -s lastpipe enabled will run the while loop in this example in the original shell process, allowing any side-effects within to take effect. Therefore, portable scripts must be written in such a way as to not depend upon either behavior.

For workarounds for this and similar issues, please see Bash FAQ #24. It's a bit too long to fit here.

9. if [grep foo myfile]

Many beginners have an incorrect intuition about if statements brought about by seeing the very common pattern of an if keyword followed immediately by a [ or [[. This convinces people that the [ is somehow part of the if statement's syntax, just like parentheses used in C's if statement.

This is not the case! if takes a command. [ is a command, not a syntax marker for the if statement. It's equivalent to the test command, except that the final argument must be a ]. For example:

if [ false ]; then echo "HELP"; fi
if test false; then echo "HELP"; fi

are equivalent -- both checking that the argument "false" is non-empty. In both cases HELP will always be printed, to the surprise of programmers from other languages guessing about shell syntax.

The syntax of an if statement is:

elif <COMMANDS> # optional
else <COMMANDS> # optional
fi # required

Once again, [ is a command. It takes arguments like any other regular simple command. if is a compound command which contains other commands -- and there is no [ in its syntax!

While bash has a builtin command [ and thus knows about [ it has nothing special to do with ]. Bash only passes ] as argument to the [ command, which requires ] to be the last argument only to make scripts look better.


The if compound command is made up of two or more sections containing lists of commands, each delimited by a then, elif, or else keyword, and is terminated by the fi keyword. The exit status of the final command of the first section and each subsequent elif section determines whether each corresponding then section is evaluated. Another elif is evaluated until one of the then sections is executed. If no then section is evaluated, then the else branch is taken, or if no else is given, the if block is complete and the overall if command returns 0 (true).

If you want to make a decision based on the output of a grep command, you do not want to enclose it in parentheses, brackets, backticks, or any other syntax! Just use grep as the COMMANDS after the if, like this:

if grep -q fooregex myfile; then

If the grep matches a line from myfile, then the exit code will be 0 (true), and the then part will be executed. Otherwise, if there are no matches, grep will return non-zero and the overall if command will be zero.

See also:

10. if [bar="$foo"]; then ...

[bar="$foo"]   # Wrong!
[ bar="$foo" ] # Still wrong!

As explained in the previous example, [ is a command (which can be proven with type -t [ or whence -v [). Just like with any other simple command, Bash expects the command to be followed by a space, then the first argument, then another space, etc. You can't just run things all together without putting the spaces in! Here is the correct way:


Each of bar, =, the expansion of "$foo", and ] is a separate argument to the [ command. There must be whitespace between each pair of arguments, so the shell knows where each argument begins and ends.

11. if [ [ a = b ] && [ c = d ] ]; then ...


If you want to express a compound conditional, do this:

if [ a = b ] && [ c = d ]; then ...


if test a = b && test c = d; then ...



if [[ a = b && c = d ]]; then ...

See pitfall #6 for a pitfall related to tests combined with conditional operators.

12. read $foo


  •  read foo

Or more safely:

  •  IFS= read -r foo




If you want to make a change to a file safely, other than appending to the end of it, use a text editor.

  •  printf %s\\n ',s/foo/bar/g' w q | ed -s file

If you are doing something that cannot be done with a text editor there must be a temporary file created at some point(*). For example, the following is completely portable:


The following will only work on GNU sed 4.x:



And the following equivalent command requires perl 5.x:





Rather than using a temporary file plus an atomic mv, this version "soaks up" (the actual description in the manual!) all the data, before opening and writing to the file. This version will cause data loss if the program or system crashes during the write operation, because there's no copy of the original file on disk at that point.

Using a temporary file + mv still incurs a slight risk of data loss in case of a system crash / power loss; to be 100% certain that either the old or the new file will survive a power loss, you must use sync before the mv.

14. echo $foo


     echo $msg


  •  Please enter a file name of the form freenfss.zip lw35nfss.zip

To demonstrate:

     echo "$var"   # writes *.zip


  •  printf "%s\n" "$foo"

15. $foo=bar


16. foo = bar


Likewise, the following are also wrong:

  •  foo= bar    # WRONG!
     foo =bar    # WRONG!
     $foo = bar; # COMPLETELY WRONG!
     foo=bar     # Right.
     foo="bar"   # More Right.

17. echo <<EOF


  •   # This is wrong:
      echo <<EOF
      Hello world
      How's it going?
      # This is what you were trying to do:
      cat <<EOF
      Hello world
      How's it going?
      echo "Hello world
      How's it going?"


      printf %s "\
      Hello world
      How's it going?


  •   printf %s \
      'Hello world
      printf %s 'Hello world

18. su -c 'some command'


  •  $ su -c 'echo hello'
     su: only the superuser may specify a login class




19. cd /foo; bar



  •  cd /foo && bar


  •  cd /foo || exit 1
     bat ... # Lots of commands.


  •  cd /net || { echo >&2 "Can't read /net. Make sure you've logged in to the Samba network, and try again."; exit 1; }

Note there's a required space between { and echo, and a required ; before the closing }. You could also write a die function, if you prefer.



Speaking of which, compare this:

       cd "$here"

With this:



Another approach is to cd unconditionally to where we're supposed to be, at the start of each loop iteration:

  •  here=$PWD
        cd "$here" || continue
        cd "$subdir" || continue

At least this way, we can continue to the next loop iteration and don't have to string an indefinite series of && together to ensure that we reach the cd at the end of the loop body.

20. [ bar == "$foo" ]

The == operator is not valid for the POSIX [ command. Use = or the [[ keyword instead.


In Bash, [ "$x" == y ] is accepted as an extension, which often leads Bash programmers to think it's the correct syntax. It's not; it's a Bashism. If you're going to use Bashisms, you might as well just use [[ instead.





  •  for i in {1..10}; do
       ./something &



22. cmd1 && cmd2 || cmd3

Some people try to use && and || as a shortcut syntax for if ... then ... else ... fi, perhaps because they think they are being clever. For instance,

  •  # WRONG!

However, this construct is not completely equivalent to if ... fi in the general case. The command that comes after the && also generates an exit status, and if that exit status isn't "true" (0), then the command that comes after the || will also be invoked. For example:

  •  i=0
     true && ((i++)) || ((i--))  # WRONG!
     echo "$i"                   # Prints 0


Another clever person thinks that we can fix it by using the pre-increment operator, since the exit status from ++i (with i initially 0) is true:

  •  i=0
     true && (( ++i )) || (( --i ))  # STILL WRONG!
     echo "$i"                       # Prints 1 by dumb luck

But that's missing the point of the example. It just happens to work by coincidence, and you cannot rely on x && y || z if y has any chance of failure! (This example still fails if we initialize i to -1 instead of 0.)


  •  i=0
     if true; then
     echo "$i"    # Prints 1


  •  # WRONG!


23. echo "Hello World!"

The problem here is that, in an interactive Bash shell (in versions prior to 4.3), you'll see an error like:

  •  bash: !": event not found



  •  $ echo "hi\!"




      mp3info -t "Don't Let It Show" ...
      mp3info -t "Ah! Leah!" ...


These solutions will work:

  •  echo 'Hello World!'


  •  echo "Hello World"!


  •  set +H
     echo "Hello World!"


  •  histchars=


Another solution is:

  •  exmark='!'
     echo "Hello, world$exmark"

In Bash 4.3 and newer, a double quote following ! no longer triggers history expansion, but history expansion is still performed within double quotes, so while echo "Hello World!" is OK, these will still be a problem:

  •  echo "Hello, World!(and the rest of the Universe)"
     echo "foo!'bar'"

24. for arg in $*


The correct syntax is:

  •  for arg in "$@"
     # Or simply:
     for arg


Here's an example:

  •  # Incorrect version
     for x in $*; do
       echo "parameter: '$x'"
     $ ./myscript 'arg 1' arg2 arg3
     parameter: 'arg'
     parameter: '1'
     parameter: 'arg2'
     parameter: 'arg3'

It should have been written:

  •  # Correct version
     for x in "$@"; do
       echo "parameter: '$x'"
    # or better:
     for x do
       echo "parameter: '$x'"
     $ ./myscript 'arg 1' arg2 arg3
     parameter: 'arg 1'
     parameter: 'arg2'
     parameter: 'arg3'

25. function foo()



  •  foo() {

26. echo "~"




27. local var=$(cmd)


Another problem with this syntax is that in some shells (like bash), local var=$(cmd) is treated as an assignment, meaning the right hand side is given special treatment, just like var=$(cmd); while in other shells (like dash), local var=$(cmd) is not treated as an assignment, and the right hand side will undergo word splitting (because it isn't quoted).

Quoting the right hand side will work around the word splitting issue, but not the exit status masking issue. For both reasons, it's best to use separate commands for this:

  •  local var

Both issues are also true of export.

The next pitfall describes another issue with this syntax:

28. export foo=~/bar


However, the export and local commands do not necessarily constitute an assignment. In some shells (like Bash), export foo=~/bar will undergo tilde expansion; in others (like dash), it will not.

  •  foo=~/bar; export foo    # Right!
     export foo="$HOME/bar"   # Right!



Change the quotes to double quotes:







     LC_COLLATE=C tr A-Z a-z


31. ps ax | grep gedit


The following is the quick and dirty stuff.


$ ps ax | grep gedit
10530 ?        S      6:23 gedit
32118 pts/0    R+     0:00 grep gedit



An alternative to this is to use:

ps ax | grep '[g]edit'              # quote to avoid shell GLOB



$ ps -C gedit
  PID TTY          TIME CMD
10530 ?        00:06:23 gedit

But why bother when you could just use pgrep instead?

$ pgrep gedit




$ ps -C gedit -opid=


$ pidof gedit



$ pgrep -fl firefox
3128 /usr/lib/firefox/firefox

Please read ProcessManagement. Seriously.

32. printf "$foo"


Always supply your own format string:

printf %s "$foo"
printf '%s\n' "$foo"

33. for i in {1..$n}


Do this instead:





if [[ $foo = "$bar" ]]





re='some RE'
if [[ $foo =~ $re ]]


The same problem occurs with pattern matching inside [[:

[[ $foo = "*.glob" ]]      # Wrong! *.glob is treated as a literal string.
[[ $foo = *.glob ]]        # Correct. *.glob is treated as a glob-style pattern.



[ -n "$foo" ]
[ -z "$foo" ]

[[ -n $foo ]]
[[ -z $foo ]]

Test follows symlinks, therefore if a symlink is broken, i.e. it points to a file that doesn't exists or is in a directory you don't have access to, test -e returns 1 for it even though it exists.


# bash/ksh/zsh

# POSIX sh+test
[ -e "$broken_symlink" ] || [ -L "$broken_symlink" ]



You can check that the following do work:



39. expr sub-string fails for "match"

This works reasonably well

most of the time


But WILL fail for the word "match"





$ word=match
$ echo "${word#?}"    # PE
$ echo "${word:1}"    # SE





41. content=$(<file)

There isn't anything wrong with this expression, but you should be aware that command substitutions (all forms: `...`, $(...), $(<file), `<file`, and ${ ...; } (ksh)) remove any trailing newlines. This is often inconsequential or even desirable, but if you must preserve the literal output including any possible trailing newlines, it gets tricky because you have no way of knowing whether the output had them or how many. One ugly but usable workaround is to add a postfix inside the command substitution and remove it on the outside:

absolute_dir_path_x=$(readlink -fn -- "$dir_path"; printf x)

A less portable but arguably prettier solution is to use read with an empty delimiter.

# Ksh (or bash 4.2+ with lastpipe enabled)
readlink -fn -- "$dir_path" | IFS= read -rd '' absolute_dir_path

The downside to this method is that the read will always return false unless the command outputs a NUL byte causing only part of the stream to be read. The only way to get the exit status of the command is through PIPESTATUS. You could also intentionally output a NUL byte to force read to return true, and use pipefail.

set -o pipefail
{ readlink -fn -- "$dir_path" && printf '\0'; } | IFS= read -rd '' absolute_dir_path
set +o pipefail

This is somewhat of a portability mess, as Bash supports both pipefail and PIPESTATUS, ksh93 supports pipefail only, and only recent versions of mksh support pipefail, while earlier versions supported PIPESTATUS only. Additionally, a bleeding-edge ksh93 version is required in order for read to stop at the NUL byte.


One way to prevent programs from interpreting filenames passed to them as options is to use pathnames (see pitfall #3 above). For files under the current directory, names may be prefixed with a relative pathname ./.

In the case of a pattern like *.* however, problems can arise because it matches a string of the form ./filename. In a simple case, you can just use the glob directly to generate the desired matches. If however a separate pattern-matching step is required (e.g. the results have been preprocessed and stored in an array, and need to be filtered), it could be solved by taking the prefix into account in the pattern: [[ $file != ./*.* ]], or by stripping the pattern from the match.

# Bash
shopt -s nullglob
for path in ./*; do

# Or even better
for file in *; do
    [[ $file != *.* ]] && rm "./$file"

# Or better still
for file in *.*; do
    rm "./$file"

Another possibility is to signal the end of options with a -- argument. (Again, covered in #pf3).

shopt -s nullglob
for file in *; do

43. somecmd 2>&1 >>logfile


somecmd >>logfile 2>&1




if cmd; then


case $status in
        echo success >&2
        exit 1
        echo "Unknown error $status, exiting." >&2
        exit "$status"

45. y=$(( array[$x] ))

The code given to an arithmetic expansion or compound command undergoes a pass of expansions and substitutions prior to evaluation. It is the text generated by these initial expansions that is ultimately evaluated as an arithmetic expression. This must be handled correctly or unintended code execution can easily result.

Here, an expression is stitched together by expanding one code fragment into another.

$ x='$(date >&2)'        # redirection is just so we can see everything happen
$ y=$((array["$x"]))       # Quotes don't help. The array doesn't even have to exist
Mon Jun  2 10:49:08 EDT 2014

The arithmetic processor needs to get a reference to this array variable in bash's internal symbol table, so it passes array[$(date >&2)] to a lookup function (just like e.g. read or printf -v do with variable names passed as arguments) which in turn expands the command substitution to resolve the index.

(For numeric indexed arrays, the lookup function next evaluates the expanded text of the index as an arithmetic expression. Consequently, mutually recursive variable lookups and arithmetic expansions can occur to any depth (up to Bash's defined limit), any of which can produce unintended side-effects.)

Most of the time, there is no need to use any kind of expansion within an arithmetic expansion. Use variable names directly in the expression (no $) wherever possible (i.e. except for positional parameters and POSIX "special variables"). If you validate variables before using them and guarantee that no expansion generates anything other than a numeric literal then most issues are automatically avoided.

Escape any expansions to pass them into the expression without expanding them first:

# Typical task reading some columns into an associative array.
typeset -A arr
printf -v offset '%(%s)T' -1

while IFS=' ' read -r x y; do
    [[ $x $y == +([0-9]) +([0-9]) ]] # validate input (see next pitfall)
    (( arr[\$(date -d "@$x" +%F)] = y - offset )) # Escaped substitution passes the entire expression literally.

Another option is to use let with single-quoted arguments. ((expr)) is equivalent to let "expr" (double-quoted args).

46. read num; echo $((num+1))

Always validate your input (see BashFAQ/054) before using num in an arithmetic context as it allows code injection.

$ echo 'a[$(echo injection >&2)]' | bash -c 'read num; echo $((num+1))'

47. IFS=, read -ra fields <<< "$csv_line"

Unbelievable as it may seem, POSIX requires the treatment of IFS as a field terminator, rather than a field separator. What this means in our example is that if there's an empty field at the end of the input line, it will be discarded:

$ IFS=, read -ra fields <<< "a,b,"
$ declare -p fields
declare -a fields='([0]="a" [1]="b")'

Where did the empty field go? It was eaten for historical reasons ("because it's always been that way"). This behavior is not unique to bash; all conformant shells do it. A non-empty field is properly scanned:

$ IFS=, read -ra fields <<< "a,b,c"
$ declare -p fields
declare -a fields='([0]="a" [1]="b" [2]="c")'

So, how do we work around this nonsense? As it turns out, appending an IFS character to the end of the input string will force the scanning to work. If there was a trailing empty field, the extra IFS character "terminates" it so that it gets scanned. If there was a trailing non-empty field, the IFS character creates a new, empty field that gets dropped.

$ input="a,b,"
$ IFS=, read -ra fields <<< "$input,"
$ declare -p fields

48. export CDPATH=.:~/myProject

Do not export CDPATH.

Setting CDPATH in .bashrc is not an issue, but exporting it will cause any bash or sh script you run, that happen to use cd, to potentially change behaviour.

There are two problems. A script that does the following:

cd some/dir || exit
cmd to be run in some/dir

may change directory to ~/myProject/some/dir instead of ./some/dir, depending on what directories exist at the time. So the cd may succeed and take the script to the wrong directory, with potentially harmful effects of the following commands which now run in a different directory than intended.

The second problem is when cd is run in a context where the output is captured:

output=$(cd some/dir && some command)

As a side-effect when CDPATH is set, cd will output something like /home/user/some/dir to stdout to indicate that it found a directory through CDPATH, which in turn will end up in the output variable along with the intended output of some command.

A script can make itself immune to a CDPATH inherited from the environment by always prepending ./ to relative paths, or run unset CDPATH at the start of the script, but don't assume every scripter has taken this pitfall into account, so don't export CDPATH.

49. OIFS="$IFS"; ...; IFS="$OIFS"

Directly assigning a variable's value to a temporary variable isn't alone enough to restore its state. The assignment will always result in a set but empty temporary variable even if the initial variable was unset. This is a particular problem for IFS because an empty IFS has a completely different meaning from an unset IFS, and setting IFS to a temporary value for a command or two is a common requirement.

An easy workaround is to designate a prefix to distinguish set from unset vars, then strip it when finished.

${oIFS:+'false'} unset -v IFS || IFS=${oIFS#_}

A local variable is usually preferable when possible.

f() {
  local IFS

Subshells are another possibility.


50. hosts=( $(aws ...) )

It is not safe to populate an array with a raw $(...) CommandSubstitution. The output of the command undergoes word splitting (on all whitespace, even ones that are inside quotes) and then globbing. If there's a word like * or eh? or [abc] in the result, it will be expanded based on filenames in the current working directory.

To select a replacement, you need to know whether the command writes its output on a single line, or multiple lines. If it's a single line:

read -ra hosts < <(aws ...)

If it's multiple lines (and you're targeting bash 4.0 or later):

readarray -t hosts < <(aws ...)

If it's multiple lines (and you want compatibility with bash 3.x, or want your command's exit status to be reflected in success or failure of the read operation without depending on behavior only available in bash 4.4 and newer):

IFS=$'\n' read -r -d '' -a hosts < <(aws ... && printf '\0')

This will prevent globbing. It still won't help you if you needed to avoid splitting on quoted whitespace, but unfortunately nothing bash can do handles that case. For generalized CSV (comma-separated value) file handling, you really need to switch to a language that has a dedicated CSV input library.

51. Non-atomic writes with xargs -P

GNU xargs supports running multiple jobs in parallel. -P n where n is the number of jobs to run in parallel.

seq 100 | xargs -n1 -P10 echo "$a" | grep 5
seq 100 | xargs -n1 -P10 echo "$a" > myoutput.txt

This will work fine for many situations but has a deceptive flaw: If $a contains more than ~1000 characters, the echo may not be atomic (it may be split into multiple write() calls), and there is a risk that two lines will be mixed.

$ perl -e 'print "a"x2000, "\n"' > foo
$ strace -e write bash -c 'read -r foo < foo; echo "$foo"' >/dev/null
write(1, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 1008) = 1008
write(1, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 993) = 993
+++ exited with 0 +++

Obviously the same issue arises if there are multiple calls to echo or printf:

slowprint() {
  printf 'Start-%s ' "$1"
  sleep "$1"
  printf '%s-End\n' "$1"
export -f slowprint
seq 10 | xargs -n1 -I {} -P4 bash -c "slowprint {}"
# Compare to no parallelization
seq 10 | xargs -n1 -I {} bash -c "slowprint {}"
# Be sure to see the warnings in the next Pitfall!

Outputs from the parallel jobs are mixed together, because each job consists of two (or more) separate write() calls.

If you need the outputs unmixed, it is therefore recommended to use a tool that guarantees output will be serialized (such as GNU Parallel).

For further details see a demonstration of the mixing problem.

52. find . -exec sh -c 'echo {}' \;

This command contains a CodeInjection vulnerability. The filename that is found by find is injected into a shell command and parsed by sh. If the filename contains shell metacharacters like ; or $( ... ) then the filename may be executed as code by `sh'.

The "slowprint" example in the previous Pitfall would have been a CodeInjection bug if the input weren't guaranteed to be integers.

To be more precise, POSIX find does not specify whether an argument which contains more than just {} is expanded. GNU find allows this CodeInjection to occur. Other implementations choose a safer path:

# uname -a
HP-UX imadev B.10.20 A 9000/785 2008897791 two-user license
# find /dev/null -exec sh -c 'echo {}' \;

The correct approach is to separate the filename argument from the script argument:

find . -exec sh -c 'echo "$1"' x {} \;

53. sudo mycmd > /myfile

Redirection is done before the command is executed. Usually that doesn't matter, but with sudo we have a command being executed as a different user than the redirection.

If the redirection must be executed with sudo-granted privileges, then you need a wrapper:

sudo sh -c 'mycmd > /myfile'

Instead of a wrapper you can use tee:

sudo mycmd | sudo tee /myfile >/dev/null

This may be easier to write if mycmd has a lot of quoting.

54. sudo ls /foo/*

This is very similar to the previous pitfall. Globbing is also done before the command is executed. If the directory isn't readable by your normal user privileges, then you may need the globbing to be done in a shell that has the sudo-granted privileges:

sudo sh -c 'ls /foo/*'

55. myprogram 2>&-

Do not close stdin, stdout or stderr as a "shorthand" for redirecting to /dev/null. Write it out correctly.

myprogram 2>/dev/null

Why? Consider what happens when your program tries to write an error message to stderr. If stderr has been redirected to /dev/null, the write succeeds, and your program is free to carry on, secure in the knowledge that it has diligently reported the error condition.

But if stderr has been closed, then the write will fail. At that point, your program may do something unpredictable. It may carry on and ignore the failure, or it may immediately exit, considering the execution environment so broken that it cannot safely continue. Or whatever else the programmer decided the program should do when its world has become a dystopian hell.

All programs are assured that stdin, stdout and stderr will exist and will be readable/writable in an appropriate and reasonable manner. By closing one of them, you have violated your promise to this program. This is not acceptable.

Of course, an even better solution would be to actually log the errors somewhere, so you can go back and read them and figure out what's wrong.

56. Using xargs without -0

xargs splits on whitespace. This is unfortunate because whitespace is allowed in filenames and commonly used by GUI users. xargs also treats ' and " specially, which can also lead to problems:

touch Dad\'s\ \"famous\"\ 1\'\ pizza.txt
touch Dad\'s\ 12\"\ records.txt
touch 2\"x1\'\ wood.txt
touch 2\"x4\"\ wood.txt

Here xargs warns:

# Do not do this
$ find . -type f | xargs wc

Here xargs does not warn at all:

# Do not do this
echo * | xargs wc
find *famous* -type f | xargs wc
find *4* -type f | xargs wc

Instead use xargs -0:

# Do this instead
printf '%s\0' * | xargs -0 wc
find . -type f -name '*famous*' -print0 | xargs -0 wc
find . -type f -name '*4*' -exec wc {} +

If using -0 is not simple, an alternative is to use GNU Parallel, which splits on \n. And while \n is also allowed in filenames they never occur unless your users are malicious. In any case: If you use xargs without -0 put a comment in your code explaining why that is safe in your particular situation.

57. unset a[0]

When passing an indexed array element to unset, it needs to be quoted. Otherwise, it may be treated as a glob, and expanded against the files in the current directory. If there happens to be a file named a0 then the glob is expanded to a0 and you end up executing unset a0.

unset 'a[0]'     # Always quote indexed array elements when unsetting.

58. month=$(date +%m); day=$(date +%d)

Calling date multiple times is a bad idea. Imagine what happens if the first call occurs a millisecond before midnight on April 30, and the second call occurs a millisecond after midnight on May 1. You would end up with month=04 and day=01.

It's better to call date one time, retrieving all of the fields you want in a single invocation.

A common idiom for that:

eval "$(date +'month=%m day=%d year=%Y dayname="%A" monthname="%B"')"

Or with bash's (4.2 or above) printf builtin:

printf -v d '%(month=%m day=%d year=%Y dayname="%A" monthname="%B")T'
eval "$d"

Remember things like month or day names are locale-dependent, hence the quotes around %A or %B to avoid problems in locales where day or month names contain spaces or other special characters for the shell.

Or, you may retrieve a timestamp in epoch format (seconds since the start of 1970), and then use that to generate human-readable date/time fields as needed.

# Requires bash 4.2 or above
printf -v now '%(%s)T' -1        # Or now=$EPOCHSECONDS in bash 5.0
                                 # -1 may be omitted in 4.3 or above
printf -v month '%(%m)T' "$now"
printf -v day '%(%d)T' "$now"

If your system's strftime() doesn't support %s, you can get the epoch time with:

now=$(awk 'BEGIN{srand(); print srand()}')

59. i=$(( 10#$i ))

Forced base 10 interpretation only works with signless numbers. As long as $i contains a string of digits with no leading - or +, everything is fine. But if $i might be negative, this conversion could fail, either noisily (with an error message), or even worse, silently (simply yielding the wrong result).

If there's any chance $i could be negative, use this instead:

i=$(( ${i%%[!+-]*}10#${i#[-+]} ))

For explanations, please see ArithmeticExpression.

CategoryShell CategoryBashguide

BashPitfalls (last edited 2019-07-13 22:25:38 by geirha)