ZFS + iSCSI performance diff between FreeNAS and EON: How to improve FreeNAS perf?

[Got2GoLV]

Hello Everyone,

(I'm posting here as I think this is a iSCSI specific issue...I might be wrong though)

I am doing some ZFS/iSCSI performance testing between FreeNAS and EOS connecting to standalone ESXi 5.1 server (running a Win7 VM + CrystalDiskMark).
It seems that EON test results are a bit different than the FreeNAS test results, using the same hardware and same benchmarks.

Specifically, the 4KB random write/read tests.

Ex:

5 passes, 1,000MB file test
2 drive zfs mirror (2TB seagate 7200RPM drives)

Code:

FreeNAS:
           Sequential Read :   107.546 MB/s
          Sequential Write :    95.909 MB/s
         Random Read 512KB :   100.575 MB/s
        Random Write 512KB :    94.183 MB/s
    Random Read 4KB (QD=1) :    10.981 MB/s [  2680.8 IOPS]
   Random Write 4KB (QD=1) :     6.097 MB/s [  1488.6 IOPS]
   Random Read 4KB (QD=32) :    85.672 MB/s [ 20915.9 IOPS]
  Random Write 4KB (QD=32) :     5.221 MB/s [  1274.6 IOPS]

Code:

EON:
           Sequential Read :   106.239 MB/s
          Sequential Write :    92.353 MB/s
         Random Read 512KB :    97.753 MB/s
        Random Write 512KB :    92.485 MB/s
    Random Read 4KB (QD=1) :    12.146 MB/s [  2965.4 IOPS]
   Random Write 4KB (QD=1) :     9.872 MB/s [  2410.2 IOPS]
   Random Read 4KB (QD=32) :   114.754 MB/s [ 28016.1 IOPS]
  Random Write 4KB (QD=32) :    80.932 MB/s [ 19758.8 IOPS]

Both EON and FreeNAS are base installs, and the only settings changed are to configure the ethernet IPs, ZFS pool, and iSCSI.

EON: eon-1.0b-151-64-cifs-min.iso
FreeNAS: FreeNAS-8.3.0-BETA3-x64 (r12468M) (I dont recall the random write 4KB QD=32 being so slow on 8.2 and previous versions)


The NAS server is a super micro 12 bay chassis, with an X7DGU MB, 2 Xeon 5310s, 20GB ECC RAM, and a SAT2-MV8 PCI-X SATA card, 2 Broadcom based NIC ports, and 4 Intel based NIC ports. Two flash USB drives are used for switching between FreeNAS and EOS on this same server (ZFS pools are destroyed and created after the OS is changed)

The ESXi 5.1 server is a Supermicro 1U chassis with an X8DTU-F MB, 24GB ECC RAM, 2 Xeon E5520s, 2 Broadcom NIC ports, and 2 Intel NIC ports.

iSCSI is setup directly between the two servers with ethernet cables, using 2 ports on each server (for MPIO). Each one on a separate subnet.

dd tests seem consistent between EON and FreeNAS, and on both, the results are much higher than the iSCSI tests.
So, it would seem that the issue is not with the disk system, but perhaps specifically with the different iSCSI implementation of both OSs.

Ideally, I would like to see FreeNAS perform as good as EOS on the 4k random tests.

Any suggestions on what to check/change ?



Thanks in advance!

 

[Got2GoLV]

Not sure if it's related...

I see that aoyama has a new istgt version out: 20120901
(http://shell.peach.ne.jp/aoyama/archives/2120)

And the current FN 8.3b3 seems to have 20111008.

Would there be an easy way to update the FN version to test ?


Thanks!

 

[Got2GoLV]

Here is another test with the same setup, except I used FreeNAS 8.2 release p1 x64

How can I get this performance out of 8.3 beta3 ?
Is it perhaps some default setting that changed between 8.2 and 8.3 ?

Code:

FreeNAS 8.2 release p1

           Sequential Read :   107.392 MB/s
          Sequential Write :   105.885 MB/s
         Random Read 512KB :   100.658 MB/s
        Random Write 512KB :    97.283 MB/s
    Random Read 4KB (QD=1) :    11.015 MB/s [  2689.3 IOPS]
   Random Write 4KB (QD=1) :     9.250 MB/s [  2258.4 IOPS]
   Random Read 4KB (QD=32) :    97.136 MB/s [ 23714.9 IOPS]
  Random Write 4KB (QD=32) :    63.174 MB/s [ 15423.3 IOPS]