Understanding Docker In Docker with Gitlab CI

Have been playing around with gitlab recently and trying to create a build pipeline for my golang app.

So thought of documenting my learnings visually about in docker in docker and interactions in the docker executor of the gitlab runners.

I recommend you read a little about gitlab CI and play around it with a bit to get a feel about the workflow in gitlab.

Lets say we have a project which basically does the following:

• connects to a httpd server and pulls information.
• it processes the information and pushes to kafka / mysql

Assume we write the code in golang and write normal tests along with integration tests.

The integration tests rely on a launching httpd server / kafka / zookeeper/mysql as docker containers.

 

For that lets say we do the following :

• we have a docker compose which launches kafka / zk only
• we manually launch the httpd server using ‘docker run’ (we can launch it in docker compose too. but lets do it separately as an exercise for the sake of learning to understand how dockers interact in gitlab ci).
• we manually launch mysql (we can launch it in docker compose too. but lets do it separately as an exercise for the sake of learning to understand how dockers interact in gitlab ci)

i.e.

• docker-compose up
• docker run -t httpd
• #start mysql somehow
• go test

 

Now we come up with a gitlab-CI definition file which specifies 2 stages – test and build docker image for our app.

Here is a sample GITLAB-CI file (you may ignore the details now) but remember we intend to do tests and build docker.

So lets focus on the TEST stage,

• Every build defined in gitlab-ci.yml file is allocated to a gitlab runner process.

 

• Each gitlab runner executes the build using an executor (many types of executors are there).

 

• We shall configure our runner to use the docker executor.

 

The following diagram illustrates what happens when we run the build in gitlab CI.

 

The above diagram focuses on the TEST_JOB of TEST stage.

Hope the diagram is self explanatory!

PS: hope you noticed the Docker in Docker In Docker

i.e. httpd container in dind container in docker executor container.

Any feedback is welcome.