weird 1gb/s limit on read only-write is 8gb/s

[aaronski1]

Hey, new server new weirdness.
Supermicro dual 6 core xeons, 96gb ram
Truenas 12
24x16tb exos drives. setup a pool as 4 raid z2's and 4 spares.
setup a simple smb share, connecting using a fast, modern pc with 10gb ethernet.

uploading to the smb share, i get around 8gb/s which is fine.
when I download the same file back i get 990mb/sec which is pretty clearly bottlenecking at gigabit speed.

anyone know why this would be the case?

I thought it could be smb related, so i setup an iscsi target, mounted that within windows and still get 8000 up, 990 down. so it doesn't appear to be smb related.
i ran atto disk bench on the iscsi share and have about 20k iops/sec write and 14k read, so that should be more than enough to get over the 1gb/sec. is there some network tuning i need to do?

 

[NugentS]

Please post your hardware spec as per forum rules

 

[aaronski1]

dual xeon 2680's, 96gb ddr3, LSI controller in IT mode, supermicro motherboard.
However, I found the issue, it had 2 IP's, one for the 1gb nic that is also used for IPMI, and one for the 10gb nic. I was connecting using the IP of the 10gb nic, but for some reason, the traffic was flowing back over the 1gb one. maybe the routing table prefers to use the 1gb nic for outbound transfers? I dunno. I'll disable it and move on. strange.

 

[HoneyBadger]

If both of your NICs are on the same subnet, this is expected.

See the following resource for details.

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

 

[jgreco]

Also, dear lord, please use proper abbreviations.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to translate terms on the reader's side when reading posts...

www.truenas.com