What is IOwait on Linux
Updated: Sep 13
Linux IOwait is a common Linux performance issue. Today we will look at what iowait means and what contributes to this problem. Hope this can give you more ideas about high IOwait issue.
IOwait on Linux
The iowait column on top command output shows the percentage of time that the processor was waiting for I/O to complete. It indicates that the system is waiting on disk or network IO. Because the system is waiting on those resources, it can not fully utilize the CPU.
The process will get "stuck" in D state because of disk io or NFS io. Processes accessing hard NFS mounts will also be put into D state while they wait for the I/O to return - thus a hard NFS mount from a machine that has dropped off of the network can cause processes to go into D state.
High IOWait usually means that your disk subsystem or network storage is overworked.This can occur due to hardware problems (the kernel is waiting for something from a device that never comes) or from kernel-related issues (driver bugs that cause a system call to never return).
High iowaits on specific CPU cores
The io workload on the CPU cores are not evenly distributed. This is a expected behavior of a Linux kernel. When any cpu deals with any task and that task needs to do a IO transaction then cpu issues a request to a IO controller and now it is IO controllers responsibility to serve this request, so as much time the IO controller will take to accomplish the request that much of time that task will be in 'D' state and cpu will be just waiting for IO (called as IO_WAIT).
And if system has number of processors then the cpu which is serving to that particular task will wait for IO and will be idle for that amount of time, the other processors will be assigned to other running tasks, so seeing a IOWAIT for particular CPU's is expected behavior of a Linux kernel.
We can use iotop check which process is generating workloads to our disks. More info about iotop here.