Container detached from docker network

Introduction

One fine day you find out that one of your docker containers is running but is not currently attached to the container network that you created using docker network command. Ideally a container should be detached from the network when there is some error in it. However, if you set the restart: on-failure in the docker-compose.yml in that case the container should restart and still be attached to the network. In such a weird situation follow the steps documented in this blog.

Possible reasons

Listed below are the possible reasons for this issue

• Storage space issue
• Memory issue ( We will cover this in next blog )

Storage space issue

We need to check the space utilisation at the server level first then drill down to the docker engine.

Server level utilisation

In most cases the server is ubuntu system hence check the disk usage using the disk free command (i.e. df -h). It will show you the current storage space utilisation of the server.

df -h

Docker engine space utilisation

Docker engine will use the storage space for docker images, container internal storage, local volumes and build cache. We can see this summary using command mentioned below

docker system df

Docker system usage summary

To view detailed information at TYPE level, use verbose flag as shown in below command

docker system df -v

The output of the verbose flag includes

• Image space usage
• Container space usage
• Volumes space usage
• Build cache space usage

Analysis

Further analysis you need to do is :

• Image space usage could be reduced by pruning unused images. You should prune images more frequently so that unused images do not pile up.
• If you are using any storage containers like MinIO then check what kind of artefacts are stored and how to minimise their size or have some retention policy for them.
• Analyse databases like MongoDB, MySQL, etc. Check if there are any tables which store any kind of logs or some data that is added frequently. In such cases check if we could keep only the required range of data like past 2 days or 30 days as per the need.
• Container space is equivalent to the image size and it should be near about the same size. In this case you should try to optimise the image size during the docker image creation process.

Let us start reclaiming the space

Images pruning

Pre-requisites

Ensure the containers which you require are in stopped state ( i.e. stopped using command docker compose stop ).

Do not use down command : If you ran the docker compose down command for an app which you require then those containers will be unloaded from the docker engine hence associated images, volumes will be pruned too.

Pruning steps

Run the below command to prune images which are currently not attached to any container. Meaning the containers are still loaded on the docker engine but are in stopped state, the images used by these are retained however if you removed the containers using compose down command then such images are considered for pruning using below command.

docker image prune -a

Container space optimisation

Container space is directly proportional to the image size. Hence if you optimise the image size then container space will be reduced. I have seen examples where the images sizes of 500 MBs, post optimisation would be as small as 30-50 MBs.

Volume Pruning

Pre-requisites

Same as previous. Do not use compose down command instead use compose stop command to ensure the volumes which you require are attached to any containers. Even if they are in stopped state. Such volumes will not be pruned.

Pruning steps

Run below command to prune unused volumes.

docker volume prune -a

Build cache pruning

Build cache is used for speeding up the build process of docker images. For example if you build an image using some Dockerfile then each instruction in your Dockerfile is treated as a separate layer. Hence if you have 5 steps and you build an image. Next time if you update only the 5th step then layers till 4th step are reused and new layer is added for the 5th step. Overtime this cache is accumulated and we should clear it more frequently. Ideally you won't have this cache on any environment like staging or prod because there in you won't be building any images you will only download docker images from remote repository like NEXUS or something. But on your local machine if you are building docker images then this cache should be monitored.

Pruning steps

Run the below command to prune the build cache.

docker builder prune -a

Connect the container back

Do not forget to connect the container back into the network using below command. Assuming your network name is say local-net and container name is mongo.

docker network connect local-net mongo

Verify the same using command docker network inspect local-net.

Conclusion

I was able to cleanup my local system and staging server following above steps. Please see if you could optimise your environments and local machines too. Resources are valuable even if they are available at lower cost due to supply demand mismatch. Only thing which is unlimited in this world is misuse of any valuable resource. Save resources and eventually the environment.

BEFORE

AFTER

I was able to reclaim 40GB of my local machine space. Although I cannot share the optimisation I did on our staging server but it was significant.