No space left on device usually means that we run out of disk space. But in rare cases, we still have some space when we get this error. People will get confused when they get this error. Here is the troubleshooting process about this.
“No space left on device” even though there is space
# df -h Filesystem Size Used Avail Use% Mounted on
/dev/sda3 56G 26G 27G 50% /
/dev/sda1 99M 25M 70M 26% /boot
/dev/sdx1 50G 40G 7.3G 85% /mountpoint
$ cp /path/to/sourcefile.tar.gz /mountpoint/directory/
cp: overwrite `/mountpoint/directory/file.tar.gz’? y cp: writing `/mountpoint/directory/file.tar.gz’: No space left on device
Root Cause for “No space left on device”
It was found that directories contained millions of files, which ext filesystem driver cannot handle efficiently, and generally the ext3 and ext4 filesystems cannot handle efficiently. There are enough inodes available on the filesystem as well. So disk space and inode are not the root cause for this issue.
$ df -i Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 14942208 309407 14632801 3% /
/dev/sda1 26104 39 26065 1% /boot
/dev/sdx1 6553600 307 6553293 1% /mountpoint
The root cause is the unexpected directory size.
This directory has a lot of blocks allocated to it already but there are not a lot of files in it.ext3 filesystems do have a limit on the number of files that can be created in any one directory and after that you will see ENOSPC errors. At one stage this directory did have a lot of files in it, then the number of files was reduced. This may have created holes in the directory map. As a result, this new file, or any other sufficiently large file, can no longer be added.