Excellent performance on CIFS, much lower on iSCSI and miserable on NFS

Status
Not open for further replies.

Sbi

Cadet
Joined
Sep 17, 2016
Messages
2
My current setup is based on:
Dell T410
LSI SAS 9211-8i
32 GB RAM
2x Xeon E5620 2.4GHz
RAIDZ2 (4x SAS seagate constellation es.3 drives) with lz4 commpression enabled
4x 1 Gbe NICs (Intel + Broadcom)
1x 10 Gbe NIC (Intel X540-T2)

10 NETGEAR XS708T 10GbE switch
Cat6A Cables

3x ESXi hosts with Intel X540-T2 for iSCSI/NFS purpose

FreeNAS-9.10.1
----------------
With iSCSI share i got approximately 600 Mbits/sec and NFS showed even less - 300 Mbits/sec max.
In the same time i've managed to push CIFS with 1 Gbit/sec which is the limit of my client workstation (not ESXi hosts).
Iperf test between FreeNAS and ESXi host showed 8 Gbit/sec.
FreeNAS is set with default settings.

Interesting thing happened when I decided to test same setup on NAS4free iSCSI share. I've got 2.2-2.5 Gbit/sec stable result. NFS share gave me the same as on FreeNAS.

Btw result was the same with jumbo frames and without them as well as with 1 Gbe NICs.

Here is some dd testing:

Code:
dd if=/dev/zero of=/mnt/primvol/testfile bs=1024k count=25k
^C21606+0 records in
21605+0 records out
22654484480 bytes transferred in 68.953910 secs (328545320 bytes/sec)



Code:
dd if=/dev/zero of=/mnt/primvol/testfile bs=4M count=10000
^C6153+0 records in
6152+0 records out
25803358208 bytes transferred in 86.836757 secs (297147880 bytes/sec)


300 MB/s is not the same i can get from NFS share:( It's up to 10 times lower than the system can give.

From this perspective I'd like to know "What I'm doing wrong???".
 
Last edited by a moderator:

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
You want to do the DD Testing with Compression Off

Performance suggestions:
  • Use Mirror vDevs
  • Get a proper SLOG
  • Toss in more RAM
  • For iSCSI you want to use zVol instead of DataSet
Some test results I posted for the OP are in this thread
 
  • Like
Reactions: Sbi

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Performance is always going to suck when using block devices for sharing. Especially with a single vdev.
 
Status
Not open for further replies.
Top