Copy TO server = 10gbps NIC while copy FROM server = 1gpbs NIC

[tool_462]

Good evening/afternoon/morning! I don't know when exactly this started happening, but I just noticed it today. When doing network transfers (mostly over Windows SMB share) I have behavior that SCALE uses my 10Gbps NIC to transfer FROM the server to computers on the network. I'm able to saturate 10gbps as well as 2.5gbps NICs no problem.

When copying TO the same shares, SCALE is using the 1Gbps motherboard NIC that I leave connected for ASMB9 iKVM management. I have confirmed by watching NIC activity that it truly is using separate NICs for read/write operations to the share.

My SMB shares are all bound to the 10Gb NIC IP address.

Any ideas?

Specs:
TrueNAS Scale 22.12.2:
Asus WS c621e SAGE
2 x Xeon Gold 6130 ES 16c / 32t 2.1GHz - 3.7GHz - 35480 Passmark
12 x 32GB (384GB) DDR4-2400 ECC
GTX 1050 Low Profile
4 x WD Red 12TB (onboard chipset SATA)
4 x HGST 10TB SAS
2 x 128GB Samsung SSD boot (Asmedia SATA)
512GB Hynix m.2 (onboard m.2, Plex media cache)
4 x Intel P4510 1TB U.2
4 x 1TB SM2258HX SATA SSDs (on LSI HBA in IT)
Intel X540-T2 NIC
LSI 9300 8i

Network:
Ubiquiti UDM SE (10gig SFP to switch)
Ubiquiti Enterprise 8 PoE - can copy FROM at 2.5Gbps (RJ45) and 10Gbps (SFP+ to RJ45 transceiver)
CAT6a all runs less than 10'

Copy from:

Copy to (this one is from 2.5Gbps NIC):

 

[Arwen]

Multiple NICs on the same sub-net does not work as most people would expect, for Unix;

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

These will help in tuning:

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

 

[NugentS]

The IPMI port should NOT be sending traffic from the NAS. It SHOULD be IPMI only
[Stating the bleeding obvious]

What happens if you remove the 1Gb port do things speed up?

 

[tool_462]

The IPMI port should NOT be sending traffic from the NAS. It SHOULD be IPMI only
[Stating the bleeding obvious]

What happens if you remove the 1Gb port do things speed up?

This board doesn't have a dedicated IPMI onboard NIC, it just ties ASMB9 to either of the Intel 1g ports that are plugged in. Either one or both.

Unfortunately it doesn't seem to work with management over the X540. It worked with an old 4 port i340 though so maybe that's a different angle to approach this problem.

Removing the 1g seems to work, I'll have to try it for a few days to see if I hit any speedbumps. No bridging or anything in this setup so I think it'll be fine.

 

[tool_462]

Multiple NICs on the same sub-net does not work as most people would expect, for Unix;

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

These will help in tuning:

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

Thank you! I had read the primer and a few dismissive posts without much explanation. I'll have to dig into the tuning and other threads to understand it better.

I'm just a little baffled since it's been happy to send/receive a steady ~980MB/s in the past. No updates or changes to anything except some Apps like Plex and Minecraft servers.

 

[tool_462]

Update: If I drop the 1g port I lose some Apps (Valheim server, PiHole DNS) but not other Apps (MC servers, Plex)

 

[MrGuvernment]

Then you got apps tied to the IP of the 1G NIC likely.

what do you have for DNS entries on your network and how do you copt to your server? to the IP or Hostname?

Could do 2 DNS records for your network..

truenas.mydomain.com --> 192.168.1.2
truenasadmin.mydomain.com ---> 192.168.1.3

Then use truenas.mydomain.com to connect to your truenas for file transfers and usage ?

 

[tool_462]

Then you got apps tied to the IP of the 1G NIC likely.

what do you have for DNS entries on your network and how do you copt to your server? to the IP or Hostname?

Could do 2 DNS records for your network..

truenas.mydomain.com --> 192.168.1.2
truenasadmin.mydomain.com ---> 192.168.1.3

Then use truenas.mydomain.com to connect to your truenas for file transfers and usage ?

You inspired me to revisit ideas. I'd been out of town a lot lately and since things were "working" for the family I let it lie.

I was going to utilize a combination of subnet and DNS entries as you suggested.

Ended up removing the PiHole Docker app from all DNS and Secondary DNS entries on my other equipment. Stopping all Apps and force re-initialization of the kube cluster then picked up the correct NIC (10 gig) so I was able to disable the old 1 gig and utilize that for iKVM only!

 

[MrGuvernment]

You inspired me to revisit ideas. I'd been out of town a lot lately and since things were "working" for the family I let it lie.

I was going to utilize a combination of subnet and DNS entries as you suggested.

Ended up removing the PiHole Docker app from all DNS and Secondary DNS entries on my other equipment. Stopping all Apps and force re-initialization of the kube cluster then picked up the correct NIC (10 gig) so I was able to disable the old 1 gig and utilize that for iKVM only!

BOOM!

Sometimes we just need to step away from a problem, and then start from scratch to see the light!