ESXi + NFS + ZFS = Bad Performance?

[Nate W]

I still don't see how this suboptimal IO pipeline can give such bad overall results. I mean, yes the IO performance may be bad considering the array, but the network performance is just abyssal. He can read with 1200 MB/s from ARC, but just gets 300 MB/s on the wire...

In order to make sure 10 GbE transfer rates can be met, I'd

• iperf that sucker really hard
• and then try a test with smaller files that would be read entirely from ARC to eliminate the IO pipeline from the equation

I'd not think about getting a new HBA before those tests are green. The current rig really should be able to pass those simple tests.

BTW: Maybe this DTrace script can provide some insights into the IO problem. Although not being designed for SSDs, I'd still give it a try. You know, just to make sure that the latencies are as low expected.

Thanks a ton for these suggestions! I did a dd test while forcing a sync write, the speeds were nearly as fast as without the forced sync write, so it seems that is not the issue (Since NFS forces sync write, this seemed to be a valid test)

I'll give iperf a go and see what happens there. If I *can* narrow it down to a piece of hardware, that would be nice as this is driving me crazy :)

Nate

 

[Nate W]

I went into the 3Ware controller and tweaked the per-drive settings. Sadly, it cannot do HBA mode, but I *did* disable the RAID Card's read and write cache for each drive in my ZFS array.

I am now seeing the following performance (Still not as mind blowing as I was hoping, but much much better). This test was done mounting the zpool over NFS on an Ubuntu VM.

Enabled read/write cache on the 3Ware 9750:

dd if=/dev/zero of=temp.dat bs=4M count=5k
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 136.16 s, 158 MB/s

dd if=temp.dat of=/dev/null bs=4M
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 68.4528 s, 314 MB/s


Disabled read/write cache on the 3Ware 9750:

dd if=/dev/zero of=temp.dat bs=4M count=5k
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 49.9979 s, 430 MB/s

dd if=temp.dat of=/dev/null bs=4M
5120+0 records in
5120+0 records out
21474836480 bytes (21 GB) copied, 42.7172 s, 503 MB/s

A marked improvement, perhaps a straight HBA card would be even better?

 

[cyberjock]

I went into the 3Ware controller and tweaked the per-drive settings. Sadly, it cannot do HBA mode, but I *did* disable the RAID Card's read and write cache for each drive in my ZFS array.

And that right there is why you shouldn't be using ZFS. Not only does it hurt performance but you abstraced the drives away in a very dangerous method.