Debugging a container that won't start

By Phil Helm

To implement an architecture that takes advantage of container technology, one way to persist the data has been to use a separate data container. In my case, I am using Jenkins in a container, and I want to save the configuration/jobs/history from my CI flow. Ideally, this container would a part of some kind of DR process, but in my case I simply wanted to show that the primary (jenkins process) container was disposable without the need to rebuild jobs.

During my most recent (of many) laptop restart, I noticed my jenkins data had disappeared. Actually, what I noticed was that there was a dummy “test” job that I had created as part of a previous attempt at building out Jenkins jobs. I wasnt sure why this suddenly happened, so I started debugging. I tried starting the data container (which I later realized did not need to be started in the first place) and it immediately failed and stopped. Although I can now move on to figure out why my data container is not working with my primary container, I went through some debugging steps that I thought may be of use to some.

1. Find out if the Docker daemon is running

Since I am running on Ubuntu, I built this VM to use Upstart as the boot time start mechanism. This means I need to use it’s utilities to check the status:

> sudo status docker

docker start/running, process 2841

so far so good.

2. Start your container, see if it’s running.

> sudo docker start xxx (where ‘xxx’ are there first 3 characters of the docker container)

> sudo docker ps

When I did this, nothing was listed..

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

jenkinsadmin@ubuntu:~/jenkins-data$

so that tells me the container didnt start correctly. Then I did a

> sudo docker ps -a

now i see my container:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

6a721ae2d232 myjenkinsdata “echo ‘Data container” 4 weeks ago Exited (0) About an hour ago jenkins-data

under the STATUS column, I see it is “Exited”. Not good.

3. Check the container log

> sudo docker logs xxx

where ‘xxx’ are the first three characters of the container. This should show you any message output during container initialization. In my case, it was

jenkinsadmin@ubuntu:~$ sudo docker logs 6a7

[sudo] password for jenkinsadmin:

Data container for Jenkins

this “data container for jenkins” message corresponds to the final entry in my Dockerfile,

CMD [“echo”, “Data container for Jenkins”]

that at least told me the commands in the Dockerfile were executing.

4. Commit the container changes to a new image

The idea here is that if there is corruption in the container, you can determine whether the image itself is also at fault by committing the changes (to a new image) and creating a new container from that new image.

>docker commit –change “debug container vs image” xxxxxxxxx <repo>/<image>:<version> where ‘xxxxxxxxxxx’ is the container id.

now you can execute the “docker run” command (as you did originally to create the container you are trying to debug)