K8s Golang testing - Mocks, Fakes & Emulators

A lot of the Go code I write is developed against Google's Kubernetes API.
The API is fairly large and given that the code is mostly calling K8s then it inherently has a set of complex dependencies, these dependencies have time and costs associated to run up for real as K8s clusters in cloud providers data centres.

So how can we test K8s Go code ... or any Go code with significant dependencies. We must use substitute objects that simulate the dependency objects. There are three common terms for these substitutes known collectively as test doubles. They are mocks, stubs and fakes. Unfortunately these terms are all pretty interchangeable. So before I start bandying them about, I had better define what I mean by these and related terms for this blog post ...

Stub = A function, method or routine that doesn't do anything other than provide a shim interface. If a stub returns values then they are dummy values (possibly dependent on calling args, or call sequence) that are either fixed or generated for a fixed range.

Mock = An object which replicates all or part of the interface of another object, using stub methods.

Fake = An object that replicates all or part of the interface of another object, and has methods which are not all stubs, so some methods perform actions that simulate the actions of the real object's method.

Emulator (full fake) = A package that has a significant amount of faked methods (rather than stubbed ones). For example a database server will normally provide an in memory database configuration that will completely replicate the core functionality of the database but not persist anything after the test suite is torn down.
Normally an emulator is not part of the code base and may require service setup and teardown via the test harness. As such, use of emulators tends to be for integration tests rather than unit tests.

Spy = A stub, mock or fake that records any calling arguments made to it. This allows for subsequent test assertions to be made about the recorded calls.

K8s Go Unit Testing

Given that unit testing by definition should not need any dependencies, then the assumption might be that for dependency heavy code, most unit tests will require test doubles ... the follow on assumption is that double == mock.

Mocks

Hence a standard approach to this is to use one of the Go's many generic mocking libraries, such as https://github.com/vektra/mockery,  https://github.com/stretchr/testify or  https://github.com/golang/mock.

There are numerous tutorials and explanations available to get Gophers started with them, for example this walk through of testify and mock.

These mock tools all offer test generation from introspection of your API calls to the dependency, to reduce the maintenance overhead.

So for everything we write tests for we generate mocks that reflect our codes API and confirm that it works as we expect. However there is a problem with this, the problem is described in detail in this blog post by Seth Ammons or in summary by Testing on the Toilet

The issues with mocks are:

1. Mocking your code's calls to an API only models your usage and assumptions about it, it doesn't model the dependency directly. It makes your tests more brittle and subject to change when you update the code.
2. Mocks have no knowledge of the dependencies they are mocking so for example as Google's APIs change - your real code will fail, but your mocked tests will still pass.
3. Mocks may use call sequenced response values, so making them procedurally fragile, ie changing the order of your test calls can break mocked tests.
4. If you want to swap out one library with another for a component, then because your mocks are specifically validating that libraries API, your mocks of it will need to be regenerated or rewritten.

So what is the alternative... 

Fakes

Refactor your code to be testable by using interfaces.

Break things down into simpler interfaces and create fakes that implement the minimum methods for testing purposes. Those methods should perform some of the business logic in a simulated manner for them to better test the code and relation between method calls than pure stubs would. Your model of the dependency is direct rather than based just on your calls of its API, so arguably easier to debug when that model and the dependency (and your evolving uses of it) diverge. 

Use ready made Fakes

But back to K8s and Google APIs ... in some cases the Google component libraries already have fakes as part of the library. For example pubsub has pstest. So you can just add the methods required so that things work for your test. In which case faking can be simple ...


The client-go library has almost 20 fakes covering most of its components but the only other fakes already in the K8s go libs (that I could find!) are for pubsub and helm

cloud.google.com/go/pubsub/pstest/fake.go
k8s.io/helm/pkg/helm/fake.go
k8s.io/client-go/tools/record/fake.go
k8s.io/client-go/discovery/fake
k8s.io/client-go/kubernetes/typed/core/v1/fake
... etc

However there are also third party libs for fakes such as 
https://github.com/fsouza/fake-gcs-server

Use custom built Fakes
If there is not an existing fake or it doesn't fake what you need, then for Google libs the APIs you need to replicate are not simple and you may want to simulate a number of methods for your tests. So manually creating the fake and maintaining its API against the real Google API becomes too much work, compared with autogenerating mocks. 

Google have sensibly anticipated this and hence released google-cloud-go-testing

This package provides the full set of interfaces of the Google Cloud Client Libraries for Go
 there is no need to generate mock partial interfaces or maintain your own fake versions of its APIs.

As an example it can be used to create a fake GCS service, where data is just written to memory (in the global bucketStore variable)

The test substitutes the FakeClient for the real storage client. In order for the code to accept the real or fake client as the same type the library provides an AdaptClient method so both conform to the storage interface (stiface).
c, err := storage.NewClient(ctx, option.WithCredentialsFile(apiCredsFilename))

client = stiface.AdaptClient(c)

K8s Go integration Testing

For integration tests you ideally want to use the real dependencies, but if they are too slow or costly then they may well be best replaced by emulators.

Using gcloud emulators

Google also provides a number of full emulators to cater for speedy local integration testing,
https://cloud.google.com/sdk/gcloud/reference/beta/emulators which cover bigtable, datastore, firestore and pubsub.
So as part of your integration test setup you can can fire up the datastore emulator for example 

> export DATASTORE_EMULATOR_HOST=localhost:17067

> gcloud beta emulators datastore start --no-store-on-disk --consistency=1.0 
                                        --host-port localhost:17067 --project=my-project

The datastore client can then just be connected to the emulator for testing

client, err := datastore.NewClient(context.Background(), "my-project")

Using EnvTest and a local K8s API server

The EnvTest package creates a Kubernetes test environment that will start / stop the K8s control plane and install extension APIs. The K8s API server (and its etcd store) is by default a local emulator service (although it can also be pointed to a real K8s deployment for testing if desired).

EnvTest is wrapped up as part of Kubebuilder which is the primary SDK for rapidly building and publishing K8s APIs in Go. 

EnvTest caters for testing complex Kubernetes API calls of the type that might be required for testing a K8s operator for example. Hence when generating code for building an operator,  kubebuilder uses the controller-runtime in its boilerplate for running this up for a template integration test.

Summary

So if your K8s Go code is tested only by mocks for unit tests and running up a real Kubernetes cluster for integration tests, then maybe its time to re-evaluate your testing approach and start using the tools for fakes and emulators that are available. The only issue is that they are quite numerous with a mix of sources, so picking the right mix of Google internal lib, Google or 3rd party test package or custom built fakes and emulators becomes part of the task.