FreeNAS does freeze on large NFS file copies

[ibmg]

We have done some testing on the follwing hardware.
sidenote: we are not able to install Freenas using the CD to an attached USB Stick

First Server

Dell PowerEdge 2900
6x 73 gig 15k 3.5" drivesm, RAID 10 (software) {Does not count system drives, no spares} {15k 3.5" ZIL drive}
16 gig RAM
Dual dual core 3.2 Ghz Core2Duo CPUs


The t610 hardware configuration-
dual hex core 2.8 ghz CPUs (HT enabled)
144 gig 1333 mhz RAM
h200 controller
4x 7.2k ES drives, software RAID 0

We are using Raid 0 only for testing to emulate a possible Raid10 config for the final server
maybe a R720xd from Dell

First test with the PE2900 was saw the following

Copying a 2 gig file over the network to the lab SAN and to the FreeNas box shows they run at the same speed, or at least close enough (~100-120 meg/sec).
Copying a 2 gig file file from the lab san over the network goes at about 65 meg/sec. The same file copying from the freenas box over the network to the same server goes MUCH slower- I stopped the first attempt because it was running in kb/sec. The second run started at 5 meg/sec but sped up as it got through the file to 36/meg sec, almost like it had to load the file in to a buffer before it could serve it properly
Likewise copying files between the two SANs seems inconsistent- it will also be fast copying files to the lab SAN but not always fast going to the Freenas server. But sometimes the performance is the same; This seems to be related to the size of the data set- under 2 gig is fast in both directions, but much over 2 gig is slow going to the Freenas system.
Deleting a large file from both units- the lab san deleted a 2 gig file pretty much instantly, but it took 15 seconds on the freenas box.
Some of the Intel tests show that while the lab SAN has consistent performance, the raw throughput to the freenas box is faster for many workloads but is very bursty- e.g., it will move a lot of data and then do nothing for a bit, then move another large amount of data.
Running an iometer over the network shows that the freenas box is 4x faster than the lab SAN in terms of raw IOPS.
I tried copying very large files, though, and that's where we see the biggest difference; a 7 gig file copies at ~90 meg/sec to the lab SAN but after the first gig or two of the file to the freenas box starts slowing down, and keeps getting slower. It looks like a buffer can fill up and take the system to it's knees. At 40% of the way through the copy I stopped it because it was down to 17 meg/sec and dropping.

All of this, by the way, is using NFS so this isn't samba tuning.

on the T610

Copying our 7 gig reference file does basically the same thing in the same place as the 2900 based Freenas server- it gets a little more than 2 gig in to the file and the copy speed drops precipitously. Left to itself, by the time it finishes copying it is running slower than a copy to a slow USB thumb drive (10 meg/sec)

any Idea what is going wrong here, that Freenas slows down ?

 

[jgreco]

You've made some reasonable observations though a bit short on hard data. My suspicion? Your system is kind of a mishmosh of variables. 16GB and a moderately decent system can combine with a slower storage subsystem to cause interesting performance issues.

Try this:

Run "gstat" on the FreeNAS console. Try bursting a 2GB write to the FreeNAS. See if your I/O pegs out for a period of time after it is supposedly completed.

If that's the case, you may find some help in understanding what's happening in bug #1531 (which has nothing specifically to do with iSCSI and pretty much everything to do with ZFS write scheduling/sizing/etc). You really need to walk through the whole thing to see how I ran across a rather pathological configuration and the ways both to CAUSE it and also mitigation steps to address it. If your system behaves in a similar fashion, then I suggest you try the mitigation steps suggested.