Unit Testing: Table Driven Test

December 14, 2018

This is 2018, and almost 2019. If you are not unit testing your no-so-small project, then you should.

The whole usefulness of unit test is always debatable. Although most software engineers consider unit testing an important part of professional software development, there are notable engineers who would disagree. This is true of any discipline in the art of software development. However, in my personal experience, unit tests does help in many cases:

Unit tests helps testing the code quickly. Consider the case when you do not have unit tests: you need to build the project, and run it, then you need to manually invoke it, or read the logs that it generates, and then check if the outcome is the expected outcome. If not, you will stop, modify the code, and go through the same process. However, if this can be checked by a unit test, then you would have had to only run the test, and let the machine check the outcome. Unit tests are not meant to be integration tests, and hence would not help much if the function under question sends an email (you want to check that you received the email), but it can help test a large part of the system.
Unit tests helps in refactoring when your code becomes legacy code. When does your code become legacy code? Almost immediately! Without unit tests, it would be difficult, and sometimes dangerous to change and refactor due a behavior that could change. Unit tests will not help if it does not have enough cases to test. Michael Feathers goes further into defining legacy code as code without tests. (he is referring to tests in general rather than unit tests).
Unit tests helps preventing breaking the API. If your unit test is done correctly, it becomes an API tester, and any failed unit test means that the API contract has been broken. This prevents from accidentally shipping API breaking change.

This might not be your experience. That is fine.

The industry developed some patterns and anti-patterns around unit tests, what are as always up for debate. Should you unit test public methods only? Should you only use Arrange-Act-Assert style unit tests? etc.

In Go, the community has advocated some very useful patterns that I consider very helpful. Table driven testing, and golden files. I use both.

Table Driven Tests

Consider this scenario: you found a bug in your code, and you have identified the function that is causing this issue. The function amazing(i int, s string, p string, r string) should return the value 0 given some input, but it is in fact returning 1. You think to yourself: I know the fix. I will apply it and push it. “Why do you not add a unit tests first to check for this case, and then fix the code to confirm that it is fixing it?” your coworker asks. “Because that is too much work!” you respond.

A typical scenario. One of the main reasons people use for not writing unit tests is time-consuming. Indeed, setting up a unit test for a new use case takes time to think about the input, handle possible errors and exceptions, and then check that the output is correct. This can be overcome by a using the table driven pattern. In the table driven pattern, you spend the time setting up the pattern. But once that is done, there is no reason for not adding more unit tests. Using table driven makes adding unit tests as easy as adding a single line.

How does it work? First you define a list (table) of variables that a test might have. Then you iterate through the list and run the test. To give you an example, consider add(int i, int j) function. To test this function with table driven test you would do something like:

unit_test_add() {
	tests = [(i = 1, j = 1, result = 2), (i = 5, j = 2, result = 7), (i = -1, j = 1, result = 0)]
	for test in tests {
		got = add(test.i + test.j)
		if (got != test.result) {
			failed("add( {test.i} + {test.j} ) = {got}, but it should have been {test.result}")
		}
	}
}

As you can see, adding a new test would simply mean adding a new tuple. With this approach, there is no reason for not adding more tests.

Another advantage to this approach is that the developer is forced to focus only on the behavior of the function, then income and the outcome, rather than focusing on the behavior of the function under test. (you can still test the internals using this method as well).

Note that when unit_test_add runs, it runs all the test cases. This is usually a desirable feature, since you want to ensure the function acts correctly given any test case. When something fails, you would be able to tell which test case fails because you can read the input at which it fails, and the bad output as well as the expected output.

Sometimes you want to label a test, because the input and the output might not be simple values, they might be complicated classes. Or you want to label the behavior rather than the input, such as “the REST API should return NOT FOUND when the id of user does not exists”. In this case, you can simply add name field to the test cases, and output that as part of the output. In go, it is very common to see code such as:

func TestAPI(t *testing.T) {
	tests := []struct {
		name    string
		input   int
		output  ComplexStruct
		wantErr bool
	}{
		{
			name:   "when given 1, the result should be valid struct", 
			input:  1,
			output: ComplextStruct{
				// some valid complex structure
			},
		},
		{
			name:   "when given 4, the result should indicate warnings",
			input:  4,
			output: ComplextStruct{
				// some complex structure that indicates warnings
			},
		},
		{
			name:    "when given 6, it should error out",
			input:   4,
			wantErr: true,
		},
	}

	for _, test := range tests {
		t.Run(test.name, func(t *testing.T) {
			sut := NewStructUnderTest()
			got, err := sut(test.input)
			if (err != nil) == test.wantErr {
				t.Errorf("%s: got %v, but wantErr=%t", test.name, err, test.wantErr)
			}
			if !reflect.DeepEqual(got, test.output) {
				t.Errorf("got %v, but expected %v", got, test.output)
			}
		})
	}
}

In go, you can have sub tests as part of the main test using the function testing.Run(). This is very helpful for two reasons: (1) the test logs will include the test name without having to do it manually, and (2) you can target a specific sub-test to run without running the whole thing (generally you want to avoid this).

Table driven tests has been popularized by the go team, since it is used extensively in the standard library. An example is the strings package.

How would you test complicated behavior with table driven tests? Suppose that you want to mock a service, how would you do that with this pattern?

Good question, and there is no orthodox way to do such thing. I have seen two patterns, however. The first, which is the simpler, is to use the old method and create a function per test. This works effectively, and used very commonly. I use this approach when the setup is very complex and requires many different functions.

However, if the setup is small, I tend to use table driven tests for this. The idea is to pass the functions to mock as parameters, just like any other parameter. Consider this example:

type Item struct {
	// omitted
}

type Database interface {
	Get(int) (Item, error)
	Insert(Item) error
}

type API struct {
	db Database
}

func NewAPI(db Database) *API {
	return &API{db}
}

func (api *API) Create(i Item) error {
	if err := api.Validate(i); err != nil {
		return err
	}
	return db.Insert(i)
}

Now, in order to test the Create, you can create the mock struct first:

// mockDatabase mocks the database by ensuring that the methods are injectable
// by the caller
type mockDatabase struct {
	get    func(int) (Item, error)
	insert func(Item) error
}

func (m *MockDatabase) Get(id int) (Item, error) { return m.get(id) }
func (m *MockDatabase) Insert(i Item) error      { return m.insert(i) }

Now you can use table driven tests:

func TestCreate(t *testing.T) {
	var (
		// a generic get function that never returns an error
		noErrGet    = func(i int) { return Item{}, nil }

		// a generic insert function that never returns an error
		noErrInsert = func(i Item) { return i, nil }

		// a generic insert function that always returns an error
		errInsert   = func(i Item) { return Item{}, errors.New("mock failed") }
	)

	tests := []struct {
		name      string
		input     Item
		inserFunc func(i Item) error
		wantErr   bool
	}{
		{
			name:       "should fail valiation if the input is bad",
			input:      Item{}, // whatever the input is
			insertFunc: noErrInsert,
			wantErr:    true,
		},
		{
			name:       "should return error if the insertion fails in the database side",
			input:      Item{}, // whatever the input is
			insertFunc: errInsert,
			wantErr:    true,
		},
	}
	for _, test := range test {
		// create the mock
		// note that noErrGet is always passed since it is not used in Create
		// and that inserFunc is a parameter in the test table
		db := mockDatabase{noErrGet, test.insertFunc}
		api := NewAPI(db)
		if err := api.Create(test.input); (err != nil) != test.wantErr {
			t.Fatalf("got %v, but wantErr=%t", err, test.wantErr
		}
	}
}

I hope the example is clear! But I don’t recommend this approach if the setup is very complex. In this particular case, the setup requires a single method, and hence very easy to configure.

And I shall cover golden files in another post.