Quick Guide to Fix Linux IOWait Issue

Updated: Sep 1

Linux high IO wait is a common Linux performance issue. Today we will look at what io wait means and what contributes to this problem. Hope this can give you more ideas about how to fix high IOwait issue.




  1. What is Linux IO wait?

  2. Determine if an IO problem is causing the system to slow

  3. Find which disk is being written

  4. Find processes that cause high IO wait

  5. Find which file caused the iowait


What is Linux IO wait?

The iowait column on top command output shows the percentage of time that the processor was waiting for I/O to complete. It indicates that the system is waiting on disk or network IO. Because the system is waiting on those resources, it can not fully utilize the CPU.


IO wait is related to the CPU resource on the server.


Learn more about Linux IO wait here.




Determine if an IO problem is causing the system to slow


Verify that the system is slow due to I/O we can use multiple commands, but the simplest is the Linux command top.


[~]# top
top - 15:19:26 up  6:10,  4 users,  load average: 0.00, 0.01, 0.05
Tasks: 147 total,   1 running, 146 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  96.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :   999936 total,   121588 free,   328672 used,   549676 buff/cache
KiB Swap:  2097148 total,  2095792 free,     1356 used.   450460 avail Mem 

From the CPU line we can see the percentage of CPU wasted on I/O wait; the higher the number, the more CPU resources are waiting for I/O permissions.

wa -- iowait
 Amount of time the CPU has been waiting for I/O to complete.

Check this post to understand CPU Usage us sy wa hi in Linux Top.




Find which disk is being written


The top command above explains I/O wait from a whole, but does not indicate which disk is affected, and to know which disk is causing the problem, we use another command Iostat command. Check this post to troubleshooting Disk Issues In Linux for more.


[[] ~]# iostat -x 2 5
Linux 3.10.0-514.el7.x86_64 (localhost.localdomain)     20170303_x86_64_    (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.34    0.00    0.31    0.01    0.00   99.33
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.05    100.16    400.17    3900.00    341000.38    84.60     0.00    2.17    0.87   11.14   0.65   111.41
scd0              0.00     0.00    0.00    0.00     0.00     0.00     8.00     0.00    0.64    0.64    0.00   0.64   0.00
dm-0              0.00     0.00    1.10    0.20    37.85    17.21    84.71     0.00    2.43    0.90   10.88   0.66   0.09
dm-1              0.00     0.00    0.01    0.02     0.07     0.08     9.70     0.00    1.42    0.27    2.05   0.09   0.00


In the example above, Iostat will be updated every 2 seconds, printing 5 times of information, and-X's option is to print out the extended information


The first Iostat report prints the statistics after the last boot of the system, which means that in most cases the first printed information should be ignored, and the remaining reports are based on the time of the previous interval.


For example, this command will be printed 5 times, the second report is a statistic from the first report, the third time is based on the second, and so on.




In the above example, the%utilized of sda is 111.41%, which is a good indication that a process is being written to the sda disk.


In addition to%utilized, we can get richer resources from iostat, such as read/write requests per millisecond (rrqm/s & wrqm/s)), read and write per second (r/s & w/s), and of course more. In the example above, our project seems to be reading and writing very much information. This is very useful for us to find the appropriate process.





Find processes that cause high IO wait

The simplest way to find the culprit is to [use Linux Iotop to check disk IO usage Per process] , by looking at iotop statistics, we can easily guide sshd as the culprit.


Although Iotop is a very powerful tool and is easy to use, it is not installed by default on all Linux operating systems.

[[email protected] ~]# iotop

Total DISK READ :    0.00 B/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:    0.00 B/s | Actual DISK WRITE:       0.00 B/s
   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND   
  1028 be/4 root        200.00 B/s    300.00 B/s  0.00 %  0.00 % sshd


Find which file caused the iowait

The lsof command can show all the files that a process opens, or all the processes that open a file. From this list, we can find out exactly what files are written, depending on the size of the file and the specific data of the IO file in/proc


We can use the-p <pid> method to reduce the output, PID is the specific process


[[] ~]# lsof -p 1028
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
sshd    1028 root  cwd    DIR  253,0      233      64 /
sshd    1028 root  rtd    DIR  253,0      233      64 /
sshd    1028 root  txt    REG  253,0   819640 2393730 /usr/sbin/sshd
sshd    1028 root  mem    REG  253,0    61752  180464 /usr/lib64/libnss_files-2.17.so
sshd    1028 root  mem    REG  253,0    43928  180476 /usr/lib64/librt-2.17.so
sshd    1028 root  mem    REG  253,0    15688  269136 /usr/lib64/libkeyutils.so.1.5
sshd    1028 root  mem    REG  253,0    62744  482870 /usr/lib64/libkrb5support.so.0.1
sshd    1028 root  mem    REG  253,0    11384  180425 /usr/lib64/libfreebl3.so
sshd    1028 root  mem    REG  253,0   143352  180472 /usr/lib64/libpthread-2.17.so



In order to further confirm that these files are read and written frequently, we can view them with the following command.

[[email protected] ~] # df /tmp
File system 1K-block used available used% mount point
/ dev / mapper / cl-root 17811456 3981928 13829528 23% /

From the results of the above command, we can determine that /tmp is the root of our environment's logical disk.


There are two main steps to fix the IOwait issue.


  • figure out why this process uses these IO resources, try to reduce the IO workload from application level

  • increase the disk or storage performance from the infra level.


Related:

Troubleshooting high iowait linux Sort Disk Read Write in Linux Iotop Command Debugging High CPU Utilization For a Process on Linux How to Check Disk usage Disk utilization on Linux What is High IOwait on Linux Using Linux Iotop to check disk IO usage Per process How to Check if a Disk is Busy in Linux Troubleshooting Disk Issues In Linux



870 views
屏幕快照 2021-08-08 下午5.16.32.png