Since update to 9.2-release seeing slowdowns to 100mb/s

[didolgi]

I have a FreeNAS server that I know is low on RAM for its storage only 8gb of RAM for 15TB, but its worked well for my needs so far. I run zraid2 pools with a single large ZFS volume.

I just updated to 9.2.0-Release and I'v noticed that at seemingly random times during large reads that are running along at 400-600mb/s it will suddenly drop to less than 100mb/s and stay there. When it happens the only way I've found to restore performance is to reboot the FreeNAS. All of this IO is taking place on an iSCSI connection to a single Windows 7 box also with an Intel NIC connected to the same switch. I've had this setup in place for well over a year and never noticed any slowdowns until I updated to 9.2.0.

I've tried various things to get performance back including 'netif restart' and stoping/starting the iSCSI service. Short of a reboot nothing seems to work.

Does anybody have any ideas of how I can investigate further? What info I should post to help see if this is a configuration/hardware issues vs. some bug.

Thanks for any advice.

 

[rm-r]

gstat will tell you what the drives are up to - but yeah the ram is v low - likely causing issues

also depending on how far you have come up from - what version were you before? things change...

 

[didolgi]

rm-r -

Thanks for the reply. I think I found my issue, and it was gstat that told me the problem. It looks like one of my drives is spending an insane amount of time randomly going to 99% busy and staying there. I've just finished copying the data off the volume that included that disk and I'm going to replace it. I'm guessing the drive is failing in a weird way that its not throwing and specific error, but is probably retrying over and over to read some block. I think the slowdown just after the upgrade was a coincidence. The marginal disk caused it.

Thanks again!

 

[rm-r]

Sweet, you could also have a look at "smartd" for the drives smart information to give you a clue. And check out the other drives ;-)

 

[didolgi]

Yep - I did look at the SMART data for the drives, and I do gave it enabled. It never threw any errors. I removed the disk about 10 minutes ago, and I'm going to put it in another system for testing. Now I'm in resliver mode for for a while.

Thanks again.