Lessons on testing bash scripts

As people at work know, I’m very driven towards automated testing of software. For me, this is a no-brainer but of course within certain limits.

However, testing bash scripts remains problematic. I will share some of the techniques I use to pull this off.

As with a lot of testings, there are some requirements in order to easily test your code. I have the feeling this is doubly so in bash.

I use two techniques to ensure testing my bash scripts:

  1. load the script but don’t run it. This can be done by putting all the code in a main function and then call this function at the appropriate moment.
  2. use mock functions
    1. to overwrite functions in your application with a mock function.
    2. to replace external command calls with a mock function. This works because functions are searched first before external commands.

Some examples. Let’s say we want to test following bash script:

#! /bin/bash

# Gives some statistics about how many different applications are running under each user

function obtain_ps_list()
{
  ps -e -o user,cmd --no-headers | sort -u
}

function main()
{
  local ps_list

  ps_list=$(obtain_ps_list)
  echo "$ps_list" | awk '{print $1}' | uniq -c
}

if [[ $(readlink -f "${0}") == */tmp/bash_testing/example.sh ]]
then
  main
fi

Two thing will strike you right away:

  1. The main function is wrapped in an if construction to see whether the script is called directly or sourced. This is inspired on the way that Python does it.
    Beware, you will always need to change the path to the actual path. Currently it’s just a bogus entry.
    Also notice the * at the beginning of the path. That enables the code to run e.g. during package creation.
  2. The obtain_ps_list function seems ridiculous. There are two reasons why it exists:
    1. it makes the code more readable.
    2. it makes the code more testable. Especially if you have a larger application with several ps commands, it’s interesting to separate out their different usages.

Now let’s start testing. First we will test the obtain_ps_list function. We could mock it itself, but I would like to show how to mock shell commands.

#! /bin/bash

. ./example.sh

function ps()
{
  echo "root app1
user1 app3
user2 app4
user1 app1
root app1
user3 app5
root app2"
}

echo "Test obtain_ps_list with normal input"
expected_output="root app1
root app2
user1 app1
user1 app3
user2 app4
user3 app5"
actual_output=$(obtain_ps_list)
if [[ $expected_output != $actual_output ]]
then
  echo "FAIL:"
  diff -y <(echo "Actual output:"; echo "$actual_output") <(echo "Expected output:"; echo "$expected_output")
else
  echo "OK"
fi

Now let’s break this down.

First we source the script we want to test. Because of the if construction around the call to the main function, nothing is yet executed.

Next we write a ps function. This function will simulate output of the ps command. That way, we can ensure that our test will always give the same results.

Similarly, if there are corner cases we would like to test, it’s possible to code this into our test application.

For simplicity, I have hard-coded the output. Normally this is provided by variables that can be changed according to the test case that is being executed.

Secondly, we define a variable with the expected output and actual output. Again, for simplicity, only standard out is being checked. In real tests, also the error output is checked. The actual output is, of course, obtained by a call to the function to test.

Comparing the result is a simple if construction. Some people would like to see "x$expected_output" != "x$actual_output". Do what you think is best but I haven’t seen the need for this construction in bash yet (other shells might be more picky, especially if you use [ instead of [[).

In order to clearly show the deviations in case of failure, the differences are shown side-by-side with a headers saying which side is which. This is a personal preference. Using <( there is no need to create temporary files.

Doing a black-box testing of the application is similar. Just call the main function:

echo "Test complete app with normal input"
expected_output="      2 root
2 user1
1 user2
1 user3"
actual_output=$(main)
if [[ $expected_output != $actual_output ]]
then
  echo "FAIL:"
  diff -y <(echo "Actual output:"; echo "$actual_output") <(echo "Expected output:"; echo "$expected_output")
else
  echo "OK"
fi

As you can see, there is some boilerplate code that repeats itself. This would be good to put into a library. There are some testing libraries out there that I would have to check further.


Tags: , ,

Leave a Reply