Debugging Docker Overlay2 Out of Space
“My docker container is using 80+ GB of disk space! My system is out of disk space! Help!” -a cry for help by someone experiencing a run away container eating up disk space. A common issue with using docker is the /var/lib/docker/overlay2 folder may take up gigabytes of space which will eventually cause an out of disk space issue on the host system. A quick google search reveals pieces of information but trying to debug this issue is difficult because it require understanding how the overlay2 file system works and tracing which container the overlay folder belongs to.
What is overlay2?
A detailed explanation of what the docker overlay2 can be found on docker’s website. The short explanation is it’s one of the choices for the container file system that uses a Union File System. In laymen terms, the /var/lib/docker/overlay2 folder represents a file system that every file that the container read or writes can be found within. Since it is a Union File System, the container does not make direct modifications to any original files so there may be two copies of the same file. The base image (original files) are stored in a “lowerdir”, the modified files are stored in “upperdir”, and finally combination of the two is the “merged” folder. For example, there is a file called foo that is 50mb in the “lowerdir”. The container writes to the file which…
