Slow performance with SSDs over 10gig

[dpskipper]

I've recently put together an all SSD pool containing 6 Segate 1200 SAS3 drives which i'll be using as a really fast scratch disk over the network. Network hardware is a 10gig switch and SFP+ fiber to my PC and the FreeNAS VM inside of an ESXi host.
Freenas has 32GB ram and 4 cores two sockets. 8 cores total.
The drives are running in an MD1220 diskshelf through a 9207-8e. The HBA is passed to FreeNAS. I understand that i'm running SAS3 drives at SAS2 speeds however I should have the bandwidth between the disk shelf and the host system to get much better speeds.

Problem is I'm seeing terrible numbers over SMB.

Crystal disk mark gives me Sequential read speeds of ~800MB/s and write speeds of no more than 400MB/s. Considering each drive has a stated sequential write of 750MB/s i'm a little confused why it so low.

I ran this command in fio to benchmark the writes on the pool:

Code:

fio --name=seqwrite --rw=write --direct=1 --bs=256k --numjobs=8 --size=256G --runtime=600 --group_reporting

and the results are as follows:

Code:

WRITE: bw=1039MiB/s (1089MB/s), 1039MiB/s-1039MiB/s (1089MB/s-1089MB/s), io=609GiB (654GB), run=600003-600003msec

I've done no performance tweaking at all because I really have no idea what to tweak. My network is on MTU 1500. I have confirmed that I can I can get about 6-7Gbps from the freenas ARC to my PC. So i know my network can do at least 700MB/s. Iperf tests have been conducted on windows but I only see about 4Gbps with 1 thread. I've heard that iperf on windows is not reliable though.


I guess that my first step should be diagnosing why my network isn't getting at least 9Gbps over iperf? I don't have another physical PC to test on. Just my PC on Windows 10 with a Mellanox connect x3 card. The other end is a Mikrotik 10gig SFP+ switch and an Intel 10gig NIC going into a Dell R720 server. Freenas has the correct 10gig vNIC.

 

[Kcaj]

Someone smarter than me will be better at helping you, but what is your pool arrangement? Also this thread may also be of interest

 

[dpskipper]

Someone smarter than me will be better at helping you, but what is your pool arrangement? Also this thread may also be of interest

I've got all 6 in raid 0. Average speeds over smb are around 300 with the odd burst to 400 mb per second.

I've also put a single ssd in raid 0 in its own pool and got average speeds of 400 with bursts to 550.. I then extended the same pool and added a second ssd in a stripe and saw the numbers return to 300. Something seriously spooky is going on.

 

[Kcaj]

Are you still using the RAID controller on the MD1220?

 

[dpskipper]

Are you still using the RAID controller on the MD1220?

No according to my original post you'll see I'm using an LSI card not a Dell PERC raid card. The MD1220 is just a JBOD chassis.

 

[Kcaj]

Sorry but I am still confused about your topology. Im guessing the 9207-8e is in your server, is that correct? So what are your physical disks connected to? According to this spec sheet the MD1220 has the PERC H800

 

[dpskipper]

Do you have much experience with enterprise storage equipment? Essentially the MD1220 is just a chassis. The drives have to be connected to a HBA or Raid card in a host system. The MD1220 is not a self contained ISCSI SAN.

The MD1220 has two controllers (called EMMs) that talk to the disks and pass the communications off to a raid card or HBA via SAS SFF8087. The HBA in question is in my Dell R720 on a gen 3 x8 slot.

 

[Kcaj]

Do you have much experience with enterprise storage equipment?

Nope, thank you for enlightening me. Do you actually have your disks in RAID0 or are you using ZFS?

 

[dpskipper]

Nope, thank you for enlightening me. Do you actually have your disks in RAID0 or are you using ZFS?

I'm using ZFS. When I mean raid 0 I mean all the drives in a stripe in one vdev which is effectively raid 0.

 

[dpskipper]

Update:

I've enabled jumbo frames (MTU 9000) on all points in the link between my PC and the FreenNAS VM. This did improve write speeds however I think given its 6 SSDs in RAID 0 that the numbers are still too slow.

 

[Johnnie Black]

Iperf tests have been conducted on windows but I only see about 4Gbps with 1 thread. I've heard that iperf on windows is not reliable though.

I'd say that's likely your main issue, in my expericnce, and for a single SMB tranfer, you usually max out at around the same speed you get with a single iperf thread, I get around 9Gbps/9.5Gbps with a single thread on iperf, and around 800/900MB/s max an SMB trasnfer, also using Mellanox.

 

[dpskipper]

I'd say that's likely your main issue, in my expericnce, and for a single SMB tranfer, you usually max out at around the same speed you get with a single iperf thread, I get around 9Gbps/9.5Gbps with a single thread on iperf, and around 800/900MB/s max an SMB trasnfer, also using Mellanox.

Doing 5 threads in iperf and I can max out my connection. I've also tried simultaneous SMB transfers from the same PC (don't have multiple 10gig machines) and the max write speed of about 550 MB/s just gets spread between each transfer.

Next i might try NFS but NFS on windows is a real pain and i've not had any success with it previously

 

[dpskipper]

Update:

Mounted the share using Windows NFS client. Considerably worse speeds. Copying the same file over SMB (previously got over 550MB/s) gets me about 160MB/s.

 

[Mirfster]

Couple suggestions:

• Use Striped Mirrors instead of a single Striped vDev (Increased IOPS and has redundancy - you may not care about redundancy since you said it was "scratch" though)
• In a single vDev you will only operate at the speed of a single disk
• Could always add more RAM or even a dedicated SLOG (should be a decent one with PLP)
• Could always do iSCSI instead of NFS, but in either case I would think that performance increase/decrease is negligible depending on settings/configuration. Personally I prefer iSCSI over NFS.

 

[dpskipper]

Couple suggestions:

• Use Striped Mirrors instead of a single Striped vDev (Increased IOPS and has redundancy - you may not care about redundancy since you said it was "scratch" though)
• In a single vDev you will only operate at the speed of a single disk
• Could always add more RAM or even a dedicated SLOG (should be a decent one with PLP)
• Could always do iSCSI instead of NFS, but in either case I would think that performance increase/decrease is negligible depending on settings/configuration. Personally I prefer iSCSI over NFS.

• Tried striped mirrors. Same performance
• Tried 6 vDevs 1 disk wide. Same performance.
• Could add some more RAM but looking at netdata I don't see lack of RAM being an issue.
• I doubt SLOG would speed up the pool at the pool would be the same speed as the slog. I've also got the pool set to sync=disabled which added less than 20MB/s extra performance.
• Can't use ISCSI as the share has to be accessible by multiple PCs eventually.

 

[Mirfster]

Any difference if you bypass the Mikrotik 10gig SFP+ and direct connect (if possible)? Also, I am assuming that you have a separate vSwitch in ESXi just for the 10GB Network (that is just how I do it)?

While I do not have an all SSD Pool to test, I would still tend to think that a SLOG could provide an increase since it would offload the ZIL. However, I guess that could be for consideration later.

 

[dpskipper]

Any difference if you bypass the Mikrotik 10gig SFP+ and direct connect (if possible)? Also, I am assuming that you have a separate vSwitch in ESXi just for the 10GB Network (that is just how I do it)?

While I do not have an all SSD Pool to test, I would still tend to think that a SLOG could provide an increase since it would offload the ZIL. However, I guess that could be for consideration later.

Haven't tried that yet I might as a last resort. Iperf tests aside I can do about 700mb/s reads from ARC. So realistically my writes should be the same. If the switch was to blame I'd be seeing slowdowns across the board and not just reads.

Don't have a dedicated vSwitch just for Free as (all the VMS share it) but the other network traffic they generate is miniscule in comparison and I doubt it has any effect.

 

[Silverstar24]

Try to disable and enable LRO support with "ifconfig [interface] -LRO" and "ifconfig [interface] LRO" after that i ghet 1.1GB/s in both direction.

 

[dpskipper]

Tried that. No improvement.