Docker Compose & Health Checks

Docker Compose is a very useful tool when used during the development, testing and continuous integration of a distributed application consisting of more than one service. Instead of having to manually manage the lifecycle of individual containers the tool docker-compose allows us to manage a whole multi-service application at once, with simple commands. To start such a distributed application it suffices to use the command docker-compose up, as an example. Similarly to tear down the application we can use docker-compose down.

The description of the application is kept in a yaml file called docker-compose.yml by default. Other names are possible but need to be communicated to the tool explicitly.

Let’s look at a simple sample application. I am going to use Apache Kafka as my engine for a simple streaming platform. Kafka also requires ZooKeeper to run. ZooKeeper is responsible to persist the topology of the Kafka cluster as well as security relevant information such as access control lists (ACLs). As a third service I will be running a tools container which will serve me as a bastion to Kafka, from where I can run various Kafka command line tools to write to and read from topics stored by the Kafka broker. My Docker Compose file looks like this:

If you want to follow along with this post and do the exercise yourself then start by creating a folder named, say healthchecks. Inside this folder create a file docker-compose.yml and add the above code snippet to and save. Now we’re ready to run the application. Open a terminal window and navigate to the healthchecks folder. From within this folder execute docker-compose up.

The first time you run this command, Docker will download the necessary images from Docker Hub. Once the images are downloaded the application services are started. Double check that all 3 services are up and running with docker-compose ps. The State of each service should be Up. You output should look similar to this:

So far so good. But we’re missing some important information. The fact that State is Up for all services only tells us that the corresponding container hosting the service is up and running, but we do not know anything about the health of the application that runs within the container. Worry not, we can change that. Health checks to the rescue!

We can e.g. define a health check for ZooKeeper. The health check that we define for this simple sample is, that the ZooKeeper service is listening at port 2181. We can use the netcat or nc tool to do this test. In your editor add line 9 to 14 right after line 8 of the docker-compose.yml file. The result should look like this:

Save your changes and back in the terminal execute once again the command docker-compose up. The ZooKeeper service, since it has changed, will be restarted. Use the command watch docker-compose ps to observe the state of the ZooKeeper. It should go through the phases Up (health: starting) to Up (healthy). Hit CTRL-c to stop watching.

Now we not only know that the zookeeper container is running but also that the ZooKeeper service is healthy. This is a definitive advantage.

Now, assume that Kafka should only ever start after(!) ZooKeeper is up and healthy. How can we do that? Well, nothing simpler than that. We can add a depends_on section to the definition of the kafka service in docker compose. Add lines 26, 27 and 28 to your docker-compose.yml file and save. The result should look like this:

In the terminal tear down the running application with docker-compose down -v. Open a second terminal window and navigate to the healthchecks folder. From within that folder run watch docker-compose up. Leave this watch command running and make sure the window stays visible for you to observe. Then switch back to your first terminal window. Execute the command docker-compose up -d to start our distributed application and observe how the 3 services are started. You will notice, that the tools and zookeeper container start immediately, but the kafka container is only started after zookeeper is up and healthy.

Conclusion

We have shown how to use health checks with Docker Compose to get more information about the health of the application services running inside a container, as well as how to use information gained through health checks to delay the start of services until all their dependencies are up and healthy.