Performance issue between TN Scale and VMWare

[Janus0006]

HI,

I’m currently having an issue with my environment and I’m not sure where is my problem exactly. The problem is with read/write performance from my 2 VMWare servers to my TrueNAS Scale NAS.

As I said, I have 2 VMWare Server both connected directly via 10Gb FCoE to my TrueNAS Scale server. But when I move a VM, to a datastore on my NAS, it’s very slow. And when I say slow, it slow to often having an error and not completing the transfert.

When I test the network with iperf3, I have good result. I have 9.69GB/s. But when I monitor the move/copy of a WM, it’s much slower, I cannot see better than 10 Mbits/s.

I tried a datastore configured with NFS or iSCSI, same result.

I have dedicated interfaces on FC to connect ESX and NAS. All FC interfaces have an MTU of 9000

When I copy file from CIFS or NFS from my Windows machine (with my 1Gb interface) I can see copy of 150mb/sec, so the bottleneck is not the disk.

Really not sure where the issue is; on vmware or on my Scale server.

Any ideas ?

 

[diogen]

All FC interfaces have an MTU of 9000

Have you tried without jumbo frames?
I had cases where turning them on did more harm than good. Especially on 10Gbe... And iperf numbers will not be affected...

...I have 2 VMWare Server both connected directly via 10Gb FCoE to my TrueNAS Scale server.

What network adapter is used on the ESXi boxes to connect to TrueNAS datastore - e1000 or VMXNet3?
What version of ESXi? Is the 10G NIC supported natively or do you sideload the drivers?

Can you list the hardware of your ESXi boxes?

EDIT
You use an Emulex 10G NIC on the TrueNAS box... That is far from ideal to put it mildly

Emulex 10Gb SFP+ "works" in 11.3-U4

So I'm on U4.1, I throw an Emulex 49Y4972 OCE11102 in and it's up... Out of the box. I give it an IP and remap the LUNs from previous FC target setup and I'm connected. Then I go to use it... Works fine, I load a 50mb video file into premier and make some changes then I add two more and it...

www.truenas.com

Also

TrueNAS on an Emulex LPE12000 not showing up

So when I pourchased my server (Dell R520) it came with two emulex LPe12000 8gb HBAs with fiber transceivers. I was able to find windows drivers for my windows machine from the dell website and dell lists this card as being compatible with FreeBSD on there website so I'm wondering what I'm...

www.truenas.com

 

[Janus0006]

Have you tried without jumbo frames?
I had cases where turning them on did more harm than good. Especially on 10Gbe... And iperf numbers will not be affected...

What network adapter is used on the ESXi boxes to connect to TrueNAS datastore - e1000 or VMXNet3?
What version of ESXi? Is the 10G NIC supported natively or do you sideload the drivers?

Can you list the hardware of your ESXi boxes?

Thanks for your reply.

I looked at what you sent me. Here are the informations
For the adapter used, here is the list:
ESX1 / Dell R720 / VMWare 7.0.3
Dual QLogic Inc. QLogic 57810 10 Gigabit Ethernet Adapter
ESX2 / Dell R710 / VMWare 6.7u3
Dual QLogic Inc. QLogic 57810 10 Gigabit Ethernet Adapter
TrueNAS Scale / Custom build
Dual Emulex Corporation OneConnect 10Gb NIC

As I can see with your advise, I should try to swap my Emulex card from my TN to an ESXi and use a QLogic in TN ... ?

For the drivers used in my ESX, I used the native ones

 

[diogen]

The Qlogic cards are also not TrueNAS friendly (the whole FC family is not officially supported)

FreeNAS with QLogic 8200 dual 10G card only see single port

FreeNAS-11.3-U5, with HP Microserver Gen 10, and QLogic 8262 10G card. Issue: The card comes with dual 10G SFP+ port, but somehow, I only see a single port showing under FreeNAS. both ports have no SFP+ connected, but i think it should not matter and at least it should show 2 x ports. I...

www.truenas.com

But trying it wouldn't hurt...

From my experience, Broadcom on ESXi 7 and SolarFlare on TrueNAS work well together (with proper cooling).
I'd use SolarFlare on both ends but the latest versions of ESXi dropped support

Reddit - Dive into anything

www.reddit.com

The most recommended (and most expensive) 10G NICs are Chelsio and Intel X520 (or later). Boils down to quality of Linux drivers.

EDIT
Just for reference, Qlogic bought Broadcom a few years ago, so there is confusion sometimes between model numbers.
I use Broadcom 57810S that is probably now the same as yours Qlogic 57810... I use the SFP+ versions with DAC cables...

 

[morganL]

HI,

I’m currently having an issue with my environment and I’m not sure where is my problem exactly. The problem is with read/write performance from my 2 VMWare servers to my TrueNAS Scale NAS.

As I said, I have 2 VMWare Server both connected directly via 10Gb FCoE to my TrueNAS Scale server. But when I move a VM, to a datastore on my NAS, it’s very slow. And when I say slow, it slow to often having an error and not completing the transfert.

When I test the network with iperf3, I have good result. I have 9.69GB/s. But when I monitor the move/copy of a WM, it’s much slower, I cannot see better than 10 Mbits/s.

I tried a datastore configured with NFS or iSCSI, same result.

I have dedicated interfaces on FC to connect ESX and NAS. All FC interfaces have an MTU of 9000

When I copy file from CIFS or NFS from my Windows machine (with my 1Gb interface) I can see copy of 150mb/sec, so the bottleneck is not the disk.

Really not sure where the issue is; on vmware or on my Scale server.

Any ideas ?

There are no claims of FC support on TrueNAS SCALE.

It might appear to work, but there has been no testing by iXsystems. Many other functions are a higher priority.
This should only be done by developers.

 

[Janus0006]

There are no claims of FC support on TrueNAS SCALE.

Thanks for your answer. Event if it's not good news ;)

But knowing this, is there a way to use my TN Scale as a "fast" VMWare Datastore for my 2 ESXi hosts ? Or Scale has been designed only for small home solution ? But, I think Scale is planning the market of large enterprise (Scalable), and this solution should be available, no ?

I think to migrate to Core and configure a VM to run dockers images; to get almost the same result I have curently.

 

[HoneyBadger]

Question of clarity - you're running FCoE on your existing cards. Is there a specific reason you can't run regular Ethernet mode (no "FC" component?) and iSCSI?

 

[diogen]

But knowing this, is there a way to use my TN Scale as a "fast" VMWare Datastore for my 2 ESXi hosts ?

Start with getting two of these and replace your fiber transceivers/cables with them

Amazon.com

If this does not help, replace the Emulex card in your TrueNAS box with something like SolarFlare or Intel.

When using supported hardware TrueNAS as a NFS/iSCSI datastore (connected over 10G) for vSphere/vCenter works quite well...

 

[morganL]

Thanks for your answer. Event if it's not good news ;)

But knowing this, is there a way to use my TN Scale as a "fast" VMWare Datastore for my 2 ESXi hosts ? Or Scale has been designed only for small home solution ? But, I think Scale is planning the market of large enterprise (Scalable), and this solution should be available, no ?

I think to migrate to Core and configure a VM to run dockers images; to get almost the same result I have curently.

We recommend iSCSI at the moment.. that is well tested.
NFS can be used.. not quite as fast.