DevOps | Cloud | Analytics | Open Source | Programming





How To Fix Error - "Unable to Remove FileSystem" in Docker ?


In this post, we will explore How To Fix - "Error: Unable to Remove FileSystem" in Docker. There are various formats of the error based on the scenarios.


Error: Unable to remove filesystem for <XXXXXXXX\>
CAdvisor is one of the most common reason of this error. Utilities like CAdvisor mount the Docker system dirs in the container. You should run teh CAdvisor container as shown below -


VERSION=v0.36.0 # use the latest cadvisor release
sudo docker run \\
--volume=/:/rootfs:ro \\
--volume=/var/run:/var/run:ro \\
--volume=/sys:/sys:ro \\
--volume=/var/lib/docker/:/var/lib/docker:ro \\
--volume=/dev/disk/:/dev/disk:ro \\
--publish=8080:8080 \\
--detach=true \\
--name=cadvisor \\
--privileged \\
--device=/dev/kmsg \\
gcr.io/cadvisor/cadvisor:$VERSION
Subsequently CAdvisor will be running in the background on http://localhost:8080. The point is this setup contains directories having Docker state which CAdvisor needs to observe. If you bind-mount /var/lib/docker/, this would essentially mount all the resources of all other running containers as filesystems within the container which mounts /var/lib/docker/. In case you try to remove any of such containers, then the removal process will fail with the above error. The problem arises if the container which bind-mounts /var/lib/docker/ uses handles like statfs or fstatfs on filesystem within /var/lib/docker/. And if these are not closed and stay open.   Although CAdvisor requires such bind-mount, it is not a good practice to bind-mount /var/lib/docker in this approach. Proceed with the below checks first for the issue and then check the solution.  

Check :

Collect info of the container using the below steps -

  • Get container id

$ docker ps -a



CONTAINER ID    IMAGE             COMMAND                   CREATED             STATUS              PORTS                                                                                                                                                               NAMES
1a2163af19330    8wq7420301cb      "/bin/sh -c 'exec jav"    5 days ago          Dead                                                                                                                                                                                    4a1193af7710_4a1193af7710_4a1193af7710_xxxxxx2
  • Check the status of the container
 

  • Inspect more details of the container
 


$ docker inspect <container id>
     

Option 1:

To fix this issue -

  • First check the error log to note the dir name e.g. <XXXXXXXX> here as per the error log.
 

  • Find the process using below -

 $ sudo lsof /var/lib/docker/containers/<XXXXXXXX\>/shm
 

  • The above command will give the process name. That process is what is causing the  /var/lib/docker/containers/ path to be busy. And hence blocking the effort to remove it.
 

  • Then you can kill the process to release the resources.
 


$ kill -9 <PID>
   

Option 2:

  • Use below and find the PID of  <XXXXXXXX> (get this from the error message)
 


grep <XXXXXXXX> /proc/\*/mountinfo
 

  • Then you can kill the process to release the resources.
 


$ kill -9 <PID>
 

Option 3:

  • Reboot the server
    Hope this post helps.  

Other Interesting Reads -

   


docker rm unable to remove filesystem ,docker unable to remove filesystem device or resource busy ,docker unable to remove filesystem for ,docker unable to remove filesystem for ,docker error response from daemon unable to remove filesystem for , ,docker container is marked for removal and cannot be started , ,docker volume device or resource busy ,overlay device or resource busy ,rm: cannot remove device or resource busy ,docker compose down no resource found to remove ,docker failed to mount local volume ,refers to undefined volume invalid compose project ,unable to delete docker folder ,windows device or resource busy ,driver "overlay2" failed to remove root filesystem ,error response from daemon unable to remove filesystem for device or resource busy ,error response from daemon: container is marked for removal and cannot be started ,rm cannot remove device or resource busy docker ,driver "zfs" failed to remove root filesystem ,cannot delete docker image ,docker removal in progress ,driver "overlay2" failed to remove root filesystem ,error response from daemon: you cannot remove a running container ,error removing mounted layer ,driver "zfs" failed to remove root filesystem ,unable to delete docker folder ,docker removal in progress ,container is marked for removal and cannot be started ,rm cannot remove device or resource busy docker ,docker unable to remove filesystem ,docker unable to remove filesystem device or resource busy ,docker unable to remove filesystem for ,docker rm unable to remove filesystem ,docker failed to remove root filesystem unlinkat ,docker failed to remove root filesystem device is busy ,docker unable to remove filesystem for shm device or resource busy ,docker error response from daemon unable to remove filesystem for ,docker failed to remove root filesystem ,docker driver zfs failed to remove root filesystem ,docker driver devicemapper failed to remove root filesystem ,docker driver zfs failed to remove root filesystem exit status 1 , , , , ,error response from daemon unable to remove filesystem for device or resource busy ,docker prune ,driver "overlay2" failed to remove root filesystem ,docker container is marked for removal and cannot be started ,docker stop container ,docker remove all ,docker removal in progress ,docker failed to remove root filesystem unlinkat ,docker driver btrfs failed to remove root filesystem ,docker driver devicemapper failed to remove root filesystem ,docker driver windowsfilter failed to remove root filesystem ,docker driver zfs failed to remove root filesystem ,docker driver zfs failed to remove root filesystem exit status 1 ,docker error response from daemon unable to remove filesystem for ,docker failed to remove root filesystem ,docker failed to remove root filesystem device is busy ,docker failed to remove root filesystem unlinkat ,docker rm unable to remove filesystem ,docker unable to remove filesystem aix ,docker unable to remove filesystem command ,docker unable to remove filesystem command line ,docker unable to remove filesystem device or resource busy ,docker unable to remove filesystem for ,docker unable to remove filesystem for shm device or resource busy ,docker unable to remove filesystem git ,docker unable to remove filesystem group ,docker unable to remove filesystem group policy ,docker unable to remove filesystem grub ,docker unable to remove filesystem history ,docker unable to remove filesystem host ,docker unable to remove filesystem hostname ,docker unable to remove filesystem jar ,docker unable to remove filesystem java ,docker unable to remove filesystem job ,docker unable to remove filesystem jupyter ,docker unable to remove filesystem jupyter notebook ,docker unable to remove filesystem key ,docker unable to remove filesystem linux ,docker unable to remove filesystem list ,docker unable to remove filesystem log ,docker unable to remove filesystem mac ,docker unable to remove filesystem missing ,docker unable to remove filesystem module ,docker unable to remove filesystem name ,docker unable to remove filesystem namespace ,docker unable to remove filesystem nginx ,docker unable to remove filesystem not found ,docker unable to remove filesystem path ,docker unable to remove filesystem permission ,docker unable to remove filesystem permission denied ,docker unable to remove filesystem powershell ,docker unable to remove filesystem quarantine ,docker unable to remove filesystem quarkus ,docker unable to remove filesystem queue ,docker unable to remove filesystem quickly ,docker unable to remove filesystem quotas ,docker unable to remove filesystem temporarily ,docker unable to remove filesystem token ,docker unable to remove filesystem type ,docker unable to remove filesystem vcenter ,docker unable to remove filesystem version ,docker unable to remove filesystem vim ,docker unable to remove filesystem volume ,docker unable to remove filesystem windows ,docker unable to remove filesystem windows 10 ,docker unable to remove filesystem x86 ,docker unable to remove filesystem xcode ,docker unable to remove filesystem xcopy ,docker unable to remove filesystem xfs ,docker unable to remove filesystem xml ,docker unable to remove filesystem yaml ,docker unable to remove filesystem yet ,docker unable to remove filesystem yum ,docker unable to remove filesystem zero ,docker unable to remove filesystem zip ,docker unable to remove filesystem zip file ,docker unable to remove filesystem zone ,docker unable to remove filesystem zsh