DevOps - Importance of deleting container images by CI pipeline after pushing to registry

When we use self-hosted Azure pipeline agents, we may encounter the below issue during the build process. This is not a hard issue to troubleshoot. The reason is there in the error message.

Error processing tar file(exit status 1): open /root/.local/share/NuGet/v3-cache/670c1461c29885f9aa22c281d8b7da90845b38e4$ps:_api.nuget.org_v3_index.json/nupkg_system.reflection.metadata.1.4.2.dat: no space left on device

##[error]Error processing tar file(exit status 1): open /root/.local/share/NuGet/v3-cache/670c1461c29885f9aa22c281d8b7da90845b38e4$ps:_api.nuget.org_v3_index.json/nupkg_system.reflection.metadata.1.4.2.dat: no space left on device

##[error]The process '/usr/bin/docker' failed with exit code 1

Solution

Clean up the artifacts from the build server once it is uploaded to the permanent artifact's location. Also, clean up any residual that is produced as part of the build process such as unit test results. In our case, we faced this issue due to the leftover container images in a Linux build server. We have to delete the local images after pushing to Azure Container Registry.

Workaround

By default, Docker files (containers, images, etc) are located in /var/lib/docker which is part of the / filesystem. We can check how much space is left by using the df -h Linux command.

There are at least 2 workarounds

Option 1 - increase storage

Increase the size of the current / file system. This is easy if the build server is in the cloud environment. There will be some downtime of course.

Option 2 - change docker storage location

Mount new storage to the build server and point the docker to use that as the storage location. Below are the steps.

• The default location of docker images is specified in the /etc/docker/daemon.json file. The property name is "data-root". 
• That can be changed to the newly mounted location.
• Restart the Docker daemon.
• sudo systemctl daemon-reload 
• sudo systemctl restart docker

A totally different option is to get rid of self-hosted agents and use the Azure agents. This depends on the project requirements and enterprise policies.

References

https://docs.docker.com/engine/reference/commandline/dockerd/

https://docs.docker.com/config/daemon/systemd/#custom-docker-daemon-options