10Gbe - 8Gbps with iperf, 1.3Mbps with NFS

[LearnLearnLearn]

Ok, let's see how the speed changes after this and go from there. No tune yet unless you feel I should add one of the two links you shared.

Code:

# zpool status -v pool01
  pool: pool01
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        pool01                                          ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/84e7ceac-606e-11ec-80b2-90b11c1dd891  ONLINE       0     0     0
            gptid/850f39c4-606e-11ec-80b2-90b11c1dd891  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/8488311e-606e-11ec-80b2-90b11c1dd891  ONLINE       0     0     0
            gptid/84acdd11-606e-11ec-80b2-90b11c1dd891  ONLINE       0     0     0
          mirror-2                                      ONLINE       0     0     0
            gptid/8422d86d-606e-11ec-80b2-90b11c1dd891  ONLINE       0     0     0
            gptid/843010bf-606e-11ec-80b2-90b11c1dd891  ONLINE       0     0     0
          mirror-3                                      ONLINE       0     0     0
            gptid/84c64c61-606e-11ec-80b2-90b11c1dd891  ONLINE       0     0     0
            gptid/847dd20b-606e-11ec-80b2-90b11c1dd891  ONLINE       0     0     0

errors: No known data errors

 

[Samuel Tai]

Yes, that looks correct. Did you set sync=disabled?

 

[LearnLearnLearn]

Without the CUBIC tune, just the other that I didn't remove, I've getting around 100Mbps.

 

[Samuel Tai]

Try enabling jumbo frames, and disabling offload on your 10G NICs.

 

[LearnLearnLearn]

Jumbo and offload... as in network settings or solely on the TN server?

 

[Samuel Tai]

Jumbo frames are interface MTUs > 1500 bytes. Both settings are found in TN network interface settings. Typically, the max MTU is 9000 bytes, but you need to enable it at both the client and server, and every network device in between.

 

[LearnLearnLearn]

That won't be an option then because servers are all over the data center and we don't control any of the switches.

 

[Samuel Tai]

Give it a shot, anyway. Most data centers I know support jumbo frames as a standard offering, because it's requested and/or needed so often.

 

[LearnLearnLearn]

I can enable jumbo frames on the local switch but once beyond that, unless as you said, the DC already handles this, that's it.
Before doing that, you mentioned I should do the tune but I asked which one? I already have the tunables one but did not do the kernel one.

 

[Samuel Tai]

The 10G and cubic tunables, please. Also, disable offload in your interface.

 

[LearnLearnLearn]

I have a new problem. I cannot get into the switch. Once I can, then I'll continue.

 

[LearnLearnLearn]

I've spent hours on this and cannot do it. It's too much work, too many switches I need access to even if I can set this up solely for a test.
That said, I can connect at least eight of the servers to the same switch as the TN is on and that could work for those machines and none else.

Problem is, I don't know if we are just testing right now or if you are helping me to work toward a working configuration.

Using the mirrors we've set up, there is not enough storage at 3.5TB and I don't have time to order larger drives. Maybe I can replace them one at a time over time and of course, they would run as 1TB drives until all eight are changed out then I would rebuild the server.

Next, if I set this up, it might be overlooked by admins even if documented.
A couple of those servers are stand alone and the others are esx. I'm not sure if that means I just need to change the esx hosts to handle jumbo or if I have to change the vms also to handle that.

 

[NugentS]

Next, if I set this up, it might be overlooked by admins even if documented.
A couple of those servers are stand alone and the others are esx. I'm not sure if that means I just need to change the esx hosts to handle jumbo or if I have to change the vms also to handle that.

ESX has to handle the jumbo, not the guests as far as I am aware as it storage, not network. Of course we are assuming good practise of seperating the storage adapter from the VM Network adapter

 

[LearnLearnLearn]

Yes, they are separated.

 

[LearnLearnLearn]

Looks like in ESX, you don't simply enable but have to specify the size as well. What size is being suggested?

 

[Samuel Tai]

9000

 

[LearnLearnLearn]

Jumbo frames set to 9000 on TN and with ESX. Switch has jumbo enabled, no option for size.
Testing moving a vm to tn and there is nothing good to report.
There was a short spike to around 1Gbps.
Must be missing something.

 

[Samuel Tai]

Hmm. That's unexpectedly disappointing. Is it possible to try it with the ESX hosts on the same switch as the TN server? What NFS mount options are in use on the ESX side?

 

[LearnLearnLearn]

The ESX host I am testing is on the same switch as TN and both are 10Gbe.
But, something just dawned on me. I was focusing on getting into the switch but that's a 1Gbps switch :).
I need to set MTU on the 10Gbe switch and re-test.

 

[Samuel Tai]

Also, can you report the output of midclt call tunable.query | jq?