10Gb slower than 1Gb from ESXi

Status
Not open for further replies.

JWCombs

Cadet
Joined
Mar 26, 2018
Messages
6
Posting here to see if anyone has seen anything similar. I have two FreeNAS servers running 11.1-U1 which I use for ESXi. I have both iSCSI and NFS shares configured as datastores. I have both SSD and HDD volumes. I've been using this for a long time over 1Gb connections and it's worked great. Since faster is better and 10Gb has gotten a lot cheaper, I added a 10Gb switch and 10Gb nics to both FreeNAS and ESXi. iPerf shows the 10Gb connection working great from ESXi to the NAS. However, disk IO for VMs is horrible over the 10Gb. From within a VM it took just 10.125s to DD 1GB synchronously over 1Gb. It took 4174.08s to do the same over 10Gb! Same SSD volume in both tests.

I've gone over the nic settings and can find nothing wrong. The MTU is 1500 all around.
 
Joined
Apr 9, 2015
Messages
1,258
Full hardware specs will be needed and are a part of the forum rules for help. I am also guessing the NIC on both ends as well as the switch and cabling used will also come into question.
 

JWCombs

Cadet
Joined
Mar 26, 2018
Messages
6
FreeNAS
SuperMicro X8 board in a 4U 24 drive chassis, 3xLSI9300 HBA, 2x 6 core Zeon, 64GB ram
Chelsio 110-1088-30 dual port SFP+ adapter, 1Gb is an intel quad port server card
test volume is 5 Intel SSDs in Z1 but the same issue is seen with all volumes
ESXi
Dell R710, 2x 6 core Zeon, 256GB ram
Intel X520-DA2 10Gb SFP+, onboard 1 Gb
Switch
FS.com S3800-24T4S all 10Gb connections are SFP+ DAC.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
FreeNAS
SuperMicro X8 board in a 4U 24 drive chassis, 3xLSI9300 HBA, 2x 6 core Zeon, 64GB ram
Chelsio 110-1088-30 dual port SFP+ adapter, 1Gb is an intel quad port server card
test volume is 5 Intel SSDs in Z1 but the same issue is seen with all volumes
ESXi
Dell R710, 2x 6 core Zeon, 256GB ram
Intel X520-DA2 10Gb SFP+, onboard 1 Gb
Switch
FS.com S3800-24T4S all 10Gb connections are SFP+ DAC.
Check the specs on your X8-series motherboard and the CPUs you're using -- my X8SIE system w/ 3400-series Xeons is limited to 2.5GT/s DMI, while most 10G NICs require 5GT/s capability. More detail and discussion here:

Will this gear do 10 gigabits?

Also, I found it necessary to use jumbo packets (MTU=9000) to get 10G to play well on my network...
 

JWCombs

Cadet
Joined
Mar 26, 2018
Messages
6
Check the specs on your X8-series motherboard and the CPUs you're using -- my X8SIE system w/ 3400-series Xeons is limited to 2.5GT/s DMI, while most 10G NICs require 5GT/s capability. More detail and discussion here:

Will this gear do 10 gigabits?

Also, I found it necessary to use jumbo packets (MTU=9000) to get 10G to play well on my network...

Thanks for the reply. I plan on trying MTU=9000 but changing the MTU on ESXi is painful. The thing is, I am seeing 8GB+ speeds doing iPerf from ESXi to the NAS. The slow speed only seems to affect NFS and iSCSI. The board is supposedly "up to 6.4 GT/s" and I'm using Xeon 5650s which are also supposed to do 6.4 GT/s.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Thanks for the reply. I plan on trying MTU=9000 but changing the MTU on ESXi is painful. The thing is, I am seeing 8GB+ speeds doing iPerf from ESXi to the NAS. The slow speed only seems to affect NFS and iSCSI. The board is supposedly "up to 6.4 GT/s" and I'm using Xeon 5650s which are also supposed to do 6.4 GT/s.
Sounds like you're over the biggest hurdle, then. Perhaps the larger packet size will help with NFS/iSCSI. Good luck!
 

tangles

Dabbler
Joined
Jan 12, 2018
Messages
33
I got a two Chelsio adapters in my old X58 chipset running 11.1U4
Chelsio T520-BT (10GB dual copper)
Chelsio T520-SO (10GB dual fibre)
These serve up goodies from 10 x 4TB SATA connected to a hpt RR2744.

I just installed them without any tuning whatsoever and so I'm guessing they're still MTU=1500.
The reason I haven't tinkered is because I consistently get around 500MB/sec and the best I get is ~700MB/sec over SMB from a stupid MacMini with an ATTO 10GB coper TB box attached...

Perhaps give SMB a go to test to see if it's a protocol v protocol issue..
 

JWCombs

Cadet
Joined
Mar 26, 2018
Messages
6
I finally figured out the issue was the switch. Transfer rates would drop to around 100Kb in one direction at times. Directly connecting the ESXi box to the NAS resulted in 270MB/s writes from a VM to VHD. Writing from a real machine I have now seen writes as fast as 1.3GB/s
 

Nanosynth

Cadet
Joined
May 22, 2018
Messages
9
I have the same problem, 10gE back to back, NO SWITCH though. Horrible thruput and I tried everything suggested here. No matter what I do, MTU adjustments, tunings, nothing, the speed still is 32 minutes to transfer a 25gig VM from Esxi 5.1 server to the Freenas server, and servers are identical Dell R410's with 3 feet of twinax between them in my bedroom. With all the tweaks and changes nothing gets better OR worse, it stays the same, 32 minutes for 25gigs of data. I got this same kind of post going on somewhere else on this board, with all my specs. I have just come to accept the speeds, as it was just to cheap not to do and test out.
 

JWCombs

Cadet
Joined
Mar 26, 2018
Messages
6
Have you tried using dd from the vmware console to test raw performance? What is your disk configuration?
 

Nanosynth

Cadet
Joined
May 22, 2018
Messages
9
I have not tried dd, yes, would give a more granular look at things. In the freeNAS Dell r410, there are 4 drives, all SAS 15k. Various GB sizes, no RAID at all, just all individual. FreeNAS is directly installed on its own dedicated harddrive, it is not a VM. The other 3 drives in the FreeNAS are just for iSCSI storage (Datastores) for the Esxi 5.1 on the other Dell r410. I do not even run the VM's on the FreeNAS box, just for backups of the complete VM's. The ESXi 5.1 r410 also has 4 SAS 15k drives of various sizes and no RAID. Esxi 5.1 is on one dedicated drive and the working VM's are on the other 3 drives. Really basic setup, back to back with the 2 10gE cards. I didn't know what to expect on the transfer rates. Its ok though, FreeNAS is doing what it does just perfect for me.
 

Nanosynth

Cadet
Joined
May 22, 2018
Messages
9
I figured this thread is about 10gE, so maybe this help someone. Well, I got my 2 Qlogic 10g cards and they don't work in either my FreeNAS Dell r410 nor do they work in my Esxi 5.1 Dell r410. Maybe they would work with further investigation or drivers loaded but I just wanted to tell anyone, if they are interested, what 10gE cards are truly plug in and it works. So far, in my setup, the Mellanox MNPA19-XTR worked in the FreeNAS box (just plug it in and it works) but not in the Esxi5.1 box. The 10gE card I just happened to find at work in a locker, it is an HP card but is model NC552SFP. It just "worked" when I plugged it into the Esxi5.1 box but I didnt try it in the FreeNAS box. Both of my Qlogic QLE8152 did not work at all in either box. I would gladly give them to whoever wants them, and I would even pay the $7 USPS priority shipping, but it would just take me a day or 2 to get to the post office.
 
Joined
Dec 29, 2014
Messages
1,135
I have the same problem, 10gE back to back, NO SWITCH though. Horrible thruput and I tried everything suggested here.

I think the MTU is is a red herring here. The SFP+ modules you use and/or the DAC/Twinax cables make a difference. The modules that one card/switch like may not work at all on another brand. The DAC/Twinax cables seem to be the most touchy, escpecially if you are mixing brands. I have 11.1-U5 with Chelsio NIC's (and Chelsio SFP+ modules) in my FreeNAS, and a variety of network cards. My Cisco ESXi hosts have Cisco 10G NIC's (the baseboard management controller doesn't run the fans as high when the NIC card is one is recognizes), and My HP DL360 G7 has the 522 (aka Mellanox) card. The Mellanox card works great with ESXi, but FreeNAS/FreeBSD don't like it much. If you are using NFS (as I am), you most likely culprit is the synchronous writes from ESXi. They will make your throughput writing to FreeNAS terrible. If you add some kind of decent SSD (search the forum for the recommenced list), that will help a lot. I went from ~= 500M write throughput to ~= 4G write throughput by adding an Intel Optane 900P as an SLOG (ZFS intent log). You also have to watch flow control settings on the 10G side. Those are usually more obvious on the switch side (assuming it is a managed switch). Increasing the speed of the NIC makes the slowest component something besides the network. Across a variety of drives and controllers, I have gotten close to 8G read throughput from FreeNAS. That assumes that where you are writing can do it that fast which is rarely a given. It is tedious, but you have to find tests that can individually prove out the the different moving parts until you find what is slowing you down. I would start with the SFP+ or DAC//twinax cables first. I don't think MTU is going to get you much until your overall throughput is much higher.
 
Joined
Dec 29, 2014
Messages
1,135
I have not tried dd, yes, would give a more granular look at things.

That is an excellent place to start. If you read from your primary hard drive but write to /dev/null, you can get an idea of read performance without complicating it with writes. Using FreeBSD as an example, I used the following command.

dd bs=256k if=/dev/da0 of=/dev/null

You have to increase the block size, or your throughput will be terrible. You have to play around to find the sweet spot for the. 256K seemed to be about there for me. You could even try bs=1M and see if that gets even higher. Strangely enough, this didn't even get as high as a storage Vmotion for me. I got up to about 2.1G of read throughput, but it will give you some idea if your network is a problem.
 
Status
Not open for further replies.
Top