IO service time differences: gstat vs. iostat

[bestboy]

I just noticed that on my FreeNAS 9.3.1 iostat and gstat do report dramatically different service times. Here is a screenshot of my monitoring tmux session while I do a CIFS read and write operation to display the issue.
While the latencies reported by gstat seem to be cogent, it seems that the svc_t values of iostat are way off.

According to this probably outdated article about iostat on Solaris 9, svc_t is supposed to display reliable IO service times in milliseconds when the device is at least 5% busy.

I also found another related thread here on the forums about a similar issue with iostat, but the weird svc_t anomalies have not been cleared.

So does anyone know what's up with iostat?

 

[bestboy]

The disk latency dtrace script doesn't seem to agree with iostat neither

Code:

max da(0), us: 50592
max da(1), us: 212027
max da(2), us: 203019
max da(3), us: 52573
max da(4), us: 81536
max da(5), us: 59848

avg da(0), us: 1455
avg da(1), us: 2017
avg da(2), us: 1822
avg da(3), us: 1793
avg da(4), us: 1840
avg da(5), us: 1508

stddev da(0), us: 3510
stddev da(1), us: 9541
stddev da(2), us: 7879
stddev da(3), us: 3883
stddev da(4), us: 4765
stddev da(5), us: 3933

 

[bestboy]

Is no one else having those weird svc_t values when running iostat -dx 5? Is it just me?!

 

[cyberjock]

I just checked on my FreeNAS Mini and I can confirm the issue. I've used iostat on Linux (Mint and Ubuntu precisely) and it worked fine for me. I've always used gstat on FreeBSD/FreeNAS though.

I asked a FreeNAS developer if he's aware of any reason why you wouldn't want to use iostat over gstat and he said "no idea. I use gstat".

So my recommendation is to use gstat over iostat because it works. :P

 

[bestboy]

Thanks for checking up on this Cyberjock.
Well, ok. I'm totally fine with using gstat instead of iostat. It's just that - as you mentioned - on all the other *nix systems iostat is used for this usecase. So it is likely that sysadmins will probably be presented with the weird iostat values first before they eventually get into gstat. Not nice, but also not a big problem (especially since it is unlikely that anyone in the right mind would even consider those values to be true).

 

[cyberjock]

Just for those that are curious, another option is "zpool iostat". You can add a zpool name and even add a -v parameter to see it broken down to a "per disk" basis. Very handy at times.