How to identify what processes are generating IO wait load.
-------------------------------------------------------------
An easy way to identify what process is generating your IO Wait load is to enable block I/O debugging. This is done by setting /proc/sys/vm/block_dump to a non zero value like:
echo 1 > /proc/sys/vm/block_dump
This will cause messages like the following to start appearing in dmesg:
bash(6856): dirtied inode 19446664 (ld-2.5.so) on md1
Using the following one-liner will produce a summary output of the dmesg entries:
dmesg | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)' | sort | uniq -c | sort -rn | head
354 md
324 export
288 kjournald
53 irqbalance
45 pdflush
14 portmap
14 bash
10 egrep
10 crond
8 ncftpput
Once you are finished you should disable block I/O debugging by setting /proc/sys/vm/block_dump to a zero value like:
echo 0 > /proc/sys/vm/block_dump
References:
1) http://www.scriptbits.net/2009/07/how-to-identify-what-processes-are-generating-io-wait-load/
Thank you for posting this, I'm still debugging my iowait-problem but you got me a leap forward. Here is some awk magic to get the complete process names from dmesg:
ReplyDeletedmesg | awk '/READ|WRITE|dirtied/ {sub(/\(.*\):/,"",$2); print $2}' | sort | uniq -c | sort -rn | head
Hajo Skwirblies
I wrote a small program:
Delete--------------------------------------
#!/usr/local/bin/perl
use strict;
sub usage
{
warn("Usage:
$0
$0 {start|stop}
\n");
exit 2;
}
if (defined $ARGV[0] && $ARGV[0] ne ''){
if ($ARGV[0] eq 'start'){
my $cmd = 'echo 1 > /proc/sys/vm/block_dump';
warn("Execute: $cmd\n");
qx($cmd);
}elsif ($ARGV[0] eq 'stop'){
my $cmd = 'echo 0 > /proc/sys/vm/block_dump';
warn("Execute: $cmd\n");
qx($cmd);
}else{
&usage;
}
exit;
### NOT REACHED
}
unless (open(FD, q(dmesg | egrep 'READ|WRITE|dirtied' | awk '{print $2}' | sort | uniq -c | sort -rn |))){
die("Cannot open cmd: $!");
}
my @p = ();
chomp(@p);
close(FD);
my $i;
foreach my $f (@p){
my $pid = $f;
$pid =~ s/^.*\((\d+)\).*$/$1/;
if (defined $pid && $pid =~ m/^\d+$/){
#warn("kill -0 $pid\n");
if (kill(0,$pid)){
$pid = `ps -p $pid -o user,pid,ppid,stime,time| tail -1`;
}else{
$pid = "No process\n";
}
}else{
$pid = "No PID\n";
}
printf("%-40s %s", $f, $pid);
last if (++$i > 35);
}