Monday 4 April 2011

IO wait load tracking to a process.


How to identify what processes are generating IO wait load.
-------------------------------------------------------------

An easy way to identify what process is generating your IO Wait load is to enable block I/O debugging. This is done by setting /proc/sys/vm/block_dump to a non zero value like:

echo 1 > /proc/sys/vm/block_dump
This will cause messages like the following to start appearing in dmesg:

bash(6856): dirtied inode 19446664 (ld-2.5.so) on md1
Using the following one-liner will produce a summary output of the dmesg entries:

dmesg | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)' | sort | uniq -c | sort -rn | head
    354 md
    324 export
    288 kjournald
     53 irqbalance
     45 pdflush
     14 portmap
     14 bash
     10 egrep
     10 crond
      8 ncftpput
Once you are finished you should disable block I/O debugging by setting /proc/sys/vm/block_dump to a zero value like:

echo 0 > /proc/sys/vm/block_dump


References:

1) http://www.scriptbits.net/2009/07/how-to-identify-what-processes-are-generating-io-wait-load/

2 comments:

  1. Thank you for posting this, I'm still debugging my iowait-problem but you got me a leap forward. Here is some awk magic to get the complete process names from dmesg:

    dmesg | awk '/READ|WRITE|dirtied/ {sub(/\(.*\):/,"",$2); print $2}' | sort | uniq -c | sort -rn | head

    Hajo Skwirblies

    ReplyDelete
    Replies
    1. I wrote a small program:
      --------------------------------------
      #!/usr/local/bin/perl

      use strict;

      sub usage
      {
      warn("Usage:
      $0
      $0 {start|stop}
      \n");
      exit 2;
      }

      if (defined $ARGV[0] && $ARGV[0] ne ''){
      if ($ARGV[0] eq 'start'){
      my $cmd = 'echo 1 > /proc/sys/vm/block_dump';
      warn("Execute: $cmd\n");
      qx($cmd);
      }elsif ($ARGV[0] eq 'stop'){
      my $cmd = 'echo 0 > /proc/sys/vm/block_dump';
      warn("Execute: $cmd\n");
      qx($cmd);
      }else{
      &usage;
      }
      exit;
      ### NOT REACHED
      }

      unless (open(FD, q(dmesg | egrep 'READ|WRITE|dirtied' | awk '{print $2}' | sort | uniq -c | sort -rn |))){
      die("Cannot open cmd: $!");
      }

      my @p = ();
      chomp(@p);
      close(FD);

      my $i;
      foreach my $f (@p){
      my $pid = $f;
      $pid =~ s/^.*\((\d+)\).*$/$1/;
      if (defined $pid && $pid =~ m/^\d+$/){
      #warn("kill -0 $pid\n");
      if (kill(0,$pid)){
      $pid = `ps -p $pid -o user,pid,ppid,stime,time| tail -1`;
      }else{
      $pid = "No process\n";
      }
      }else{
      $pid = "No PID\n";
      }
      printf("%-40s %s", $f, $pid);
      last if (++$i > 35);
      }

      Delete