10Gbe - 8Gbps with iperf, 1.3Mbps with NFS

[LearnLearnLearn]

I'm at a loss of where to look for this problem but it seems to be with the TN NFS service itself.

I have a TrueNAS instance running on an old Dell R620 with 10Gbe card installed.
An NFS share is set up on the server solely to provide storage/backups etc to ESX 6.7 hosts.

After setting everything up with shares on both ESX hosts, I moved a vm from one of the hosts to the NFS share.
The speed was terrible, reaching a max of 1.3Mbps. I then tried moving the vm to the second host and the same thing happened.

Then I ran iperf on the TN server using different sizes from 128k to 1024k. As you can see, the transfer speeds are pretty much where they should be I think.
So, what is going on? Is NFS not able to keep up? Is there some additional configuring I should be doing on the TN server?

BTW, the 10Gbe cards are also iSCSI but I've read that iSCSI is only slightly faster so have not bothered to go through all that change/test.

All of the cards are HP 634026-001 554FLR 10/40G 2P SFP and the switch is an old NETAPP NAE-1101. The TN box would have a QLogic QLE8152.

 

[Samuel Tai]

How is your pool configured?

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files. ZFS will allocate long, contiguous stretches of disk...

www.truenas.com

Also, how much RAM do you have?

 

[Samuel Tai]

Also, have you applied the recommended network tuning?

10GbE performance issue in one direction

what is your MTU size? And are you seeing a lot of retries?

www.truenas.com

TCP congestion control algorithm selection in TrueNAS Core/FreeNAS

Networking-minded TrueNAS Core/FreeNAS admins may need to change the TCP congestion control algorithm for their specific installations from the default TCP new Reno to achieve the best performance. (For reference, see http://intronetworks.cs.luc.edu/current/html/newtcps.html.) There are several...

www.truenas.com

 

[LearnLearnLearn]

The system has 32GB which seems to be fine based on the iperf tests and dashboard not showing any low numbers at least. What other info can I provide?

 

[LearnLearnLearn]

>Also, have you applied the recommended network tuning?

No, I didn't know about this but can look at it.
But, why does iperf show such good results while NFS has such poor performance? Sorry, just learning about TN.

 

[Samuel Tai]

The system has 32GB which seems to be fine based on the iperf tests and dashboard not showing any low numbers at least. What other info can I provide?

View attachment 51455
View attachment 51456

No, please provide the output of zpool status -v r6pool or click the gear, and select Status. We need to know your pool topology. Also 32 GB is very low for serving to ESXi hosts.

 

[Samuel Tai]

Also please provide your NFS mount options.

 

[LearnLearnLearn]

There are no mount options in ESX that I know of. You just point to the IP and the name of the NFS share.

Code:

# zpool status -v r6pool
  pool: r6pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:14 with 0 errors on Sun Dec  5 00:00:14 2021
config:

        NAME                                          STATE     READ WRITE CKSUM
        r6pool                                        ONLINE       0     0     0
          gptid/1a688784-35d2-11ec-b4a3-90b11c1dd891  ONLINE       0     0     0

errors: No known data errors

BTW, I'm not serving the hosts from storage, just using the storage as backup, to move vms around and other such things. vms are on host storage directly.

Reading the 10Gbe tuning post you showed me, I'm not sure what I'm looking for. Even before any tuning, I'm getting over 8Gbps with iperf. Only the NFS sharing seems slow. I'll try the tune and see what happens with ESX.

 

[HoneyBadger]

I'm at a loss of where to look for this problem but it seems to be with the TN NFS service itself.

I have a TrueNAS instance running on an old Dell R620 with 10Gbe card installed.
An NFS share is set up on the server solely to provide storage/backups etc to ESX 6.7 hosts.

After setting everything up with shares on both ESX hosts, I moved a vm from one of the hosts to the NFS share.
The speed was terrible, reaching a max of 1.3Mbps. I then tried moving the vm to the second host and the same thing happened.

Do you have an SLOG device in this TrueNAS host? ESXi sends all NFS traffic as synchronous.

 

[HoneyBadger]

There are no mount options in ESX that I know of. You just point to the IP and the name of the NFS share.

Code:

# zpool status -v r6pool
  pool: r6pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:14 with 0 errors on Sun Dec  5 00:00:14 2021
config:

        NAME                                          STATE     READ WRITE CKSUM
        r6pool                                        ONLINE       0     0     0
          gptid/1a688784-35d2-11ec-b4a3-90b11c1dd891  ONLINE       0     0     0

errors: No known data errors

Hold up a minute. That's "r6pool" as in "RAID6"? As in "two parity drives"? As in "are you putting this on a single virtual device that's using a hardware RAID card"?

 

[Samuel Tai]

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

 

[LearnLearnLearn]

Let me see if I can reply to everything.

The chassis is a ten slot one.
The first two slots are two 15K 146GB SAS drives configured as RAID1 for the OS only. I didn't have anything smaller at the time.
The rest of the drives are eight 600GB 15K SAS drives and yes, good catch, they are configured as a single RAID6 array.

 

[Samuel Tai]

Please read the note about RAID controllers. Although this may work in the short term, you WILL lose data eventually. Also read the notes about pool construction. For your case, it would've been better for you to have constructed the pool as a 4-way stripe of 2-way mirrors. As it is, you're crippled by the RAID controller restricting you to the IO of a single disk, as every write has to be written both to a data disk and 2 parity disks at the same time.

 

[NugentS]

Classic example of how NOT to do a TrueNAS Server

 

[HoneyBadger]

Let me see if I can reply to everything.

The chassis is a ten slot one.
The first two slots are two 15K 146GB SAS drives configured as RAID1 for the OS only. I didn't have anything smaller at the time.
The rest of the drives are eight 600GB 15K SAS drives and yes, good catch, they are configured as a single RAID6 array.

So that means you have a PERC H710(P) and are definitely doing Bad Things in relation to a healthy system. Thankfully you can easily swap this out for an H310 (flashable to LSI IT mode) or flash the H710(P) if desired.

There's been links posted already about why hardware RAID cards don't play nice with ZFS, as well as the "path to success for block storage" involving a strong preference for mirror vdevs. But what you'll also need, if you intend to run NFS or synchronous iSCSI, is an SLOG or Separate LOG device. This will serve as a safe write log for ZFS and greatly accelerate the ESXi storage behavior.

 

[NugentS]

An Optane is an ideal SLOG - wicked fast, long life and PLP (Power Loss Protection) - you need all three. Preferably mirrored of course for data safety.

 

[Kailee71]

Also, even though it's not directly relevant in this case... Be careful with Scale as an ESXi filer as its nfs implementation (ganesha) is user space and can throttle performance noticably. Just a small heads-up in case someone is looking to fix Scale nfs performance issues.

A small side-note to NugentS' remark re: optane - this is absolutely true, and explicitly also holds true for the little 32Gb sticks - a very cost effective way to boost sync performance, especially if you combine 2 or 3 of them for a pool slog (they get striped automatically, and no, in this use case that's not a problem).

My apologies if this is considered cross posting in the wrong forum.

Kai.

 

[LearnLearnLearn]

Ok, I can barely keep track of all this. First I was told to try the tune so it seems there is no point in doing that since it sounds like I have to start over.
Second, I don't have time to order more parts, this was supposed to be a quick inexpensive solution mainly to move vms between esx hosts and back them up off of the hosts.

The system is to be installed at a remote location which I won't easily have access to other than remote.
I decided to use two SAS drives as RAID1 for the OS because I have dozens collecting dust and want the OS to be redundant.

I read countless posts from folks that said don't trust flash based devices not to eventually fail. I do have some smaller SSD drives, like 128GB but none which are matched if I wanted to use them in some redundant way.

So, thanks for the popcorn, sad that this is a classic mistake but wanting to learn to work with this correctly so appreciate the help.
It also shows there is a lot to storage, always changing and it's hard to keep up when you aren't working on this all the time.

I guess my first step and my fastest option is to re-flash this H710 controller but what do I need for this? The shared thread doesn't mention what I'm looking for in terms of this controller so I'll search for the H710 specifically.

I do already better understand now why it's not a good setup to use a RAID controller with the storage devices. Mainly it's because ZFS IS the controller and handle problems, heal them when it has full control of the storage. I think what I'm looking for is some sort of pass-thru method.

 

[HoneyBadger]

Do you have iDRAC configured for OOB remote access? If so, you should be able to do all of this remotely.

Ok, I can barely keep track of all this. First I was told to try the tune so it seems there is no point in doing that since it sounds like I have to start over.
Second, I don't have time to order more parts, this was supposed to be a quick inexpensive solution mainly to move vms between esx hosts and back them up off of the hosts.

Sorry to be the bearer of bad news here. Thankfully you should be able to do all of this without buying new hardware; although you will have to destroy the existing pool/storage configuration.

The system is to be installed at a remote location which I won't easily have access to other than remote.
I decided to use two SAS drives as RAID1 for the OS because I have dozens collecting dust and want the OS to be redundant.

It's possible to install TrueNAS to a ZFS mirror during installation. You do have to take action to change the boot device if one enters a "sort of failed state" though, it's not as automated as a HW RAID card here I admit.

I read countless posts from folks that said don't trust flash based devices not to eventually fail. I do have some smaller SSD drives, like 128GB but none which are matched if I wanted to use them in some redundant way.

Regular consumer SSDs will not provide adequate performance for SLOG responsiveness or have the endurance to stand up to SLOG duties long-term. Optane is a different beast though as it's a bit-addressable NAND design with significantly more endurance. The 32GB M.2 cards on a carrier or even the P1600X "boot device" would be a solid option. If this was purely for offline backups (no running VMs, at all, EVER) you might be able to get away with disabling sync on the dataset.

So, thanks for the popcorn, sad that this is a classic mistake but wanting to learn to work with this correctly so appreciate the help.
It also shows there is a lot to storage, always changing and it's hard to keep up when you aren't working on this all the time.

No worries. If you're here to learn, we're here to help.

I guess my first step should be to re-flash this H710 controller so need to do some searching as the shared thread doesn't mention what I'm looking for, for this controller.

Check the resource here, and double-check your hardware models:

Cross-flashing Dell H310 & H710 Mini to IT Mode

You might remember fourlynx's guide on flashing the H310 Mini Mono to IT Mode - been working with him on coming up with a similar guide for all variations of the H710 Mini as well so you no longer have to pay 4x the card price to get one...

www.truenas.com

Order of operations, assuming you have remote iDRAC, would be:

1. Destroy the existing pools and virtual disks on the PERC H710.
2. Use Fohdeesha's guide and ISO to flash your H710 to an LSI IT mode HBA
3. Reinstall TrueNAS to a mirror of the small boot devices
4. Configure a RAIDZ2 (if you won't be running VMs from there and/or don't care about performance) or a set of mirror vdevs (better for VMFS performance)
5. If you won't be running live machines or can't get there to install a proper SLOG device, disable sync on the dataset. This introduces a non-trivial risk of data corruption if you run live machines without SLOG. Do not run live machines like this unless you have no other alternative and understand the risk entirely.

 

[LearnLearnLearn]

No issues at all, the hardware is still physically here and there is nothing on it I cannot destroy since I only started testing which is why I ended up finding this problem.

Just to be clear, the vms are running on storage on the hosts.
The storage is mainly to have a central place to store vms, ISO's and other large files off the hosts. Backups if you will but not live backups of the vms which would actually be nice to do and I know that TN has this option even if I've never used it.
The storage will also be used as shared NFS space for vms across all of the esx hosts.

The chassis has 10 slots. I don't need massive amounts of storage so used the first two slots with SAS drives to run the OS.
What I have on hand that could run the OS separately from the storage are m-sata devices, samsung ssd sm841 128g, lite-on lmt-256m6m and a m.2 2242 Kingdata 64g solid state drive. I also have various SSD (none matched) 2.5 drives and lots of SAS drives.

So, first step is to go check on that ISO and update the HBA.
I guess I have to return the new battery I just ordered too as it sounds like the cache will no longer be used.

On item 3, I assume you mean that during the install, I will be offered an option to have a mirrored copy of the TN OS. I think I've used that when building pfsense firewalls.
On item 4, I assume another option while installing.