Slow speeds over 40GB/s connection

[Gagik]

Hello,

I have built a new TrueNAS Core server with 60X 14TB drives in a RAIDZ2 configuration within a pool. Additionally, I have installed a 40GB network card in both the server and my PC, establishing a direct 40GB connection between the NAS and the PC. However, I am experiencing slow speeds over SAMBA. Currently, I am getting approximately 900MB/s for write operations and 600MB/s for read operations. On my PC, I have enabled Jumbo frames, and I have done the same on the NAS side. Any assistance or suggestions would be greatly appreciated. I plan to test a 25GB connection soon.


Thanks,

 

[Arwen]

First, per Forum rules, conveniently linked at the top of every forum page, please describe your hardware completely. Including vendor and model of your 40Gbps Ethernet card.

Next, include your software version.

Last, how do you have your 60 x 14TB disks setup. You specify RAID-Z2, but not how many disks per vDev, and how many vDevs. (Hopefully you don't have have all 60 disks in a single RAID-Z2 vDev!!! Shudder...)

 

[Gagik]

System is

Intel 2U server.
Motherboard is S2600WF with the latest BIOS / Firmware
Dual intel CPU Silver 4108
NVME Boot drive
192GB RAM
40Gb card is Chelsio T580-LP-CR

TrueNas core version TrueNAS-13.0-U5.1


I have 5 Vdevs @ 12 drives in each and all are added to a single pool. let me know if you need any additional information.

Hope this helps..

 

[Arwen]

Thank you. I don't have any suggestions, as high speed networking is outside my area. But, any who could help, would need the information I requested. (And probably more... that I don't know what.)

Good luck.

 

[Gagik]

I have set up a similar configuration with 60 drives in a RAID 6 on my self-built Linux CentOS server, serving out Samba, and I am achieving a read and write speed of 2000MB/s.

 

[NickF]

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com