Peculiar FreeNAS File Transfer Patterns

[Hanakuso]

Hi all,

My setup consists of:
HP DL180 G6
12 Slot LFF Backplane
2x Intel L5520 Xeon 4 Core 2.26Ghz
6x Seagate 7.2K SATA 3TB HDDs
48GB RAM
LSI 9211-8i

configured thusly:

NAME STATE READ WRITE CKSUM
RAID10 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/de76695d-ec29-11e2-9211-f4ce46bd81ae ONLINE 0 0 0
gptid/df1b2423-ec29-11e2-9211-f4ce46bd81ae ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/dfc8a9c9-ec29-11e2-9211-f4ce46bd81ae ONLINE 0 0 0
gptid/e07bb44e-ec29-11e2-9211-f4ce46bd81ae ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
gptid/e12ffb20-ec29-11e2-9211-f4ce46bd81ae ONLINE 0 0 0
gptid/e1e8012e-ec29-11e2-9211-f4ce46bd81ae ONLINE 0 0 0
It is serving as an iSCSI target for an ESXi datastore. When I begin transferring large files (in this case, a single 45GB file) to the VM, the graphs look like this:

I have run this test many times and consistently get the same results. I've attempted to do some tuning, but it doesn't seem to make much difference. Can anyone offer some direction as to what might be causing this pattern? I know this setup is capable of sustaining wirespeed (1Gbit) transfers.

 

[survive]

Hi Hanakuso,

Take a look at this post by jgreco:

http://forums.freenas.org/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

Particularly the section titled "The ZFS write cache. (No, it's not the ZIL.)", I think that might maybe explain your problem.

As I understand it freenas\zfs needs some tweaking when you have a lot of memory and a (relatively) slow pool. ZFS uses 1/8th of your RAM for a transaction group (zfs organizes writes into what's called a transaction group & flushes them to disk when it reaches a certain size or after a certain amount of time) so it can commit your data in one giant sequential write instead of a bunch of discrete individual writes. If you have enough RAM & a slow pool the system may not be able to flush the writes to disk before the next transaction group is ready to be committed, which stalls pretty much everything. The tweaking basically adjusts how much data zfs should cache to be more in line with how fast your pool can actually write the data out.

That said, I could be completely wrong but given your hardware that's the first thing I though of.

While you are waiting for a more definitive answer you might want to run some benchmarks on your pool just to confirm that it's performing as expected. I see you have a RAID-10ish pool, so I would expect that you can move data at an impressive rate, but it could just be that the pool is jacked up in some way and isn't as fast as you expect. Here are some ways to test throughput:

1) The old reliable "dd" command where you blast a bunch of "0000000"s out to a file. Here is an example: dd if=/dev/zero of=testfile bs=4096k count=200
notes: you want to be sure what this command is doing..."dd" does exactly what you tell it, blast those zeros at the wrong thing and you can toast your pool. Change to your pool and run this on something (a dataset)
that's uncompressed. Be sure to change the "count=" to give you something bigger than your installed RAM size
2) iozone here's an example command: "iozone -a -s 10g -r 4096" change the "10g" to something bigger than your RAM.

-Will