Freenas Server Pausing frequently. Don't know how to diagnose.

[JKaniarz]

When accessing a share on my FreeNAS 8.0.0 box, the transfer will pause for 15-30 seconds, sometimes more than once. This happens via iSCSI and CIFS. While this is happening everything seems to be frozen including SSH login. However if I have an existing session, it works until I try to access the file system. ("touch some.file" works, but "cat some.file" will freeze the session until everything unpauses)
While watching "top" during an iSCSI transfer the cpu usage is around 25% when it's working, and 0% when paused.
Same for CIFS. All in all It seems like a file system issue to me.

I ran zpool status, and saw that I had a large number of write errors on one of my drives. I zpool cleared them, and then did a zpool scrub. A follow up zpool status after the scrub quoted me an estimated completion time of 400 hours and rising. (this is while the pausing is still happening intermittently). I put my desktop in sleep mode and went to bed. (effectively stopping all share traffic) This morning I ran zpool status and it said that it finished in just under one hour (around the same amount of time a full drive backup takes). There were no errors after the scrub, but it did have to resliver about 250MB.
I tried upgrading to 8.0.1 b4 (not realizing that RC1 was out) and this did not fix the problem either.

So long story short. iSCSI and CIFS transfers and directory listings can cause the pauses. The pauses block all disk IO (as best as I can tell), zpool scrub does not trigger the pauses.

Where do I proceed from here?

 

[JKaniarz]

Update: So I've upgraded to RC1, no change. All services (but ssh) are disabled.
I've opened a second SSH session and I'm running zpool iostat tank 1.
There's exactly 30 seconds between every 1-2 seconds of read write activity.
When I cat a file, it freezes until the start of the next 30 second interval.
During every burst of activity it reads at approximately full disk speed.

 

[matthewowen01]

hardware specs? network specs?

 

[JKaniarz]

Intel core2 duo e4300 (1.8ghz, 64bit)
Realtek gigabit lan (onboard)
4gb ddr2 800
cheap motherboard (intel g31)
3x 1.5TB Seagate harddrives.
Corsair Voyager GT 4gb USB drive

 

[JKaniarz]

Another quick update: i stated doing an zpool iostat -v and i noticed that the third disk in my raidz doesn't get updated at the same time as the first two.

When I try to do a SMART test (smartctl -t short), it always says that the test was interrupted. Do I need to offline the disk first?
How do I match up gpt/disk# from zpool status to the actual /dev/ada# ?

 

[JKaniarz]

So I moved the drives to another computer and booted off the same hard drive, and everything is working great. So while the problem is solved. I still don't know how I would have detected this failure via the command line. Are there any tools to detect a bad board?

 

[jgreco]

Are there any tools to detect a bad board?

Look in a mirror. ;-)