Dear community members,
first off I want to say thank you for all the great stuff I was already allowed to read on this forum. It helped me a lot in many cases, but now unfortunately I am stuck and need to ask you directly for help.
In short: I can upload stuff to my FreeNAS-server with almost 10Gbit/s but when downloading stuff I never exceed 3Gbit/s.
Note: Is seems to be most likely no ZFS or zpool issue since i am testing with iperf.
Hardware of the FreeNAS-server
Even though it turned out that the pool is not involved in my network issues I wanna give you my config here. It is a Raid-10 configuration, which i chose because i want to have fast resilvering times.
Hardware of the surrounding
After a bit or research I read that the FreeNAS 11 tunables might not be suited for optimal performance of a 10GbE network connection (which i cannot really believe but, still). I tried this (source: https://gist.github.com/bjornbouetsmith/2c8201c9e71ef33c0087010a3cf1e186):
In addition I have this one active as well:
But that didn't change a thing either.
As you can see I am running out of ideas.
Can anybody please help me?
In order to help me, do you need any further informations (e.g. config-files)?
Thanks in advance and kind regards
mimesot
first off I want to say thank you for all the great stuff I was already allowed to read on this forum. It helped me a lot in many cases, but now unfortunately I am stuck and need to ask you directly for help.
In short: I can upload stuff to my FreeNAS-server with almost 10Gbit/s but when downloading stuff I never exceed 3Gbit/s.
Note: Is seems to be most likely no ZFS or zpool issue since i am testing with iperf.
Hardware of the FreeNAS-server
- Intel Xeon E3-1245 v6 with 4x 3.7GHz plus Hyperthreading
- 64GB DDR4 RAM in 4 slots
- Supermicro X11SAT-F board
- LSI 9211-8i SATA-Controller in IT-Mode
- 9x WD Red 4TB ATA WDC WD40EFRX-68N harddrives with CMR (approx 135MB/s)
(run as my main data-pool "hdd-pool" in a RAID 10 configuration plus one hot spare) - 1x Optane 900P as a SLOG device (once I have my virtual maschines and or databases on the pool)
- 1x Samsung NVME SSD 256GB (as my test-pool just for performance tests)
- 1x Samsung SM951 SATA SSD 240GB as my OS-drive
- Intel X550-T2 ethernet card
- Seasonic Platinum Series 660W ATX power supply
- 4U Chassis with abundant cooling
- Operating System: Freenas 11.3-U3.2
- Active Services: Active Directory, SSH, SMB, SMART
- Networking: using ethernet port ix0, created vlan interface, vlan called "Server-VLAN", vlan interface has a static IP, ix0 has no configuration
Even though it turned out that the pool is not involved in my network issues I wanna give you my config here. It is a Raid-10 configuration, which i chose because i want to have fast resilvering times.
- mirror
- ATA WDC WD40EFRX-68N
- ATA WDC WD40EFRX-68N
- mirror
- ATA WDC WD40EFRX-68N
- ATA WDC WD40EFRX-68N
- mirror
- ATA WDC WD40EFRX-68N
- ATA WDC WD40EFRX-68N
- mirror
- ATA WDC WD40EFRX-68N
- ATA WDC WD40EFRX-68N
- spare
- ATA WDC WD40EFRX-68N
- log
- NVME Intel Optant 900P
Hardware of the surrounding
- Cabeling: Cat6A S/FTP
- Switch: TP-Link T1700X with 12x 10GbE and 4x SFP+
- Server1 (Virtualization): Similar to the FreeNAS-Server, far less storage but including NVME, Xeon E3-1240L v5, 32GB RAM, Supermicro X11SSH-CTF, comes with 2x Intel X550T, using VLAN 802.1Q tagged "Server-VLAN", operating system Proxmox 6.2 with Debian 10 as a basis
- Server2 (Workstation): Similar to the FreeNSA-Server, far less storage but including NVME, Xeon E3-1245 v6, 32GB RAM, MSI 236a Workstation, Intel X550-T2, using VLAN 802.1Q tagged "Server-VLAN", operating system Proxmox 6.2 with Debian 10 as a basis
- Server3 (FreeNAS-Test): This is essentially the same hardware like the original FreeNAS-Server but with no pool. Purely set up for Testing.
- Started with a simple SMB share on the FreeNAS-Server. Tested the download from the FreeNAS-servers hdd-pool to my Server2 (Workstation). Therefore I created a 10GByte random bytes file.
Downloaded it with max. 150MB/s, uploaded it with 850MB/s. Weird! - Repeated this with the nvme-pool ... same result.
- Repeated it with the srv1 ... same result.
- Repeated it over SSH (with scp) ... same results.
- I made an internal copy from the hdd-pool to the nvme-pool. Worked perfectly. So it was not the pools.
- Switched to iperf for further testing, because as far as I know that doesn't involve the storage. Guess what ... same results:
Upload to the FreeNAS-Server: around 9,3Gbit/s
Download from the FreeNAS-Server: 1-3Gbit/s ... we are definitely exceeding 1Gbit/s here. - Checked the Ports: No failed auto-negotiation. All say 10GbE full-duplex
- Checked the system load: the cpu cores do not shed a single droplet of sweat
- Performed iperf in both directions between Server1 and Server2. They managed about 9.3 to 9.4 Gbit/s in both directions.
--> The problem has to be in between the FreeNAS-Server and the Switch. - Changed the switch port
- Changed the cable
- Changed the network card port
- Changed the network card (to another X550-T2)
- Changed the PCIe port of the network card
... no changes in the results - Installed Debian 10.4 on the OS-SSD ... i get 9.3 Gbit/s up and dows
--> it has to be the FreeNAS-OS itself or in combination with my hardware - Update: Tried to enable Jumbo Frames MTU 9000 at all devices involved. Performance increased by a factor of 2-3, but the core problem persisted. It hints at the package processing rate at the server is limited by some unknown factor.
- Update: I created Server3 with basically the same harware and default config and tested FreeNA-to-FreeNAS. The same asymmetry occurs -> it is not an Issue in between different OSes.
After a bit or research I read that the FreeNAS 11 tunables might not be suited for optimal performance of a 10GbE network connection (which i cannot really believe but, still). I tried this (source: https://gist.github.com/bjornbouetsmith/2c8201c9e71ef33c0087010a3cf1e186):
Code:
#Add these lines to the file /etc/sysctl.conf #10GBE tuning kern.ipc.soacceptqueue=2048 kern.ipc.somaxconn=2048 kern.ipc.maxsockbuf=33554432 net.inet.tcp.recvbuf_inc=2097152 # (default 16384) net.inet.tcp.recvbuf_max=16777216 # (default 2097152) net.inet.tcp.recvspace=4194304 # (default 65536) net.inet.tcp.sendbuf_inc=2097152 # (default 8192) net.inet.tcp.sendbuf_max=16777216 # (default 2097152) net.inet.tcp.sendspace=4194304 # (default 32768) net.inet.tcp.sendbuf_auto=1 net.inet.tcp.recvbuf_auto=1 #Add these lines to /boot/loader.conf net.isr.maxthreads="-1" net.isr.bindthreads="1"
In addition I have this one active as well:
Code:
net.route.netisr_maxqlen=2048
But that didn't change a thing either.
As you can see I am running out of ideas.
Can anybody please help me?
In order to help me, do you need any further informations (e.g. config-files)?
Thanks in advance and kind regards
mimesot
Last edited: