Slow file transfer speed

matt998

Dabbler
Joined
Aug 14, 2021
Messages
16
So i have a bit of an issue. I have a 10Gbs network on this network i have two computers one is a windows computer and one is a Truenas computer. This issue is when ever i upload to the TrueNas computer i get on at top end 300MB's< i have tested the cables the NIC switch all of it. so i tried a file transfer with in the same computer. The windows computer was fine GB + with no issue However the truenas system was like 150MB's. I have looked and looked and i cant find anything that could be causing this. Has any one else come across this issue.
 

c77dk

Patron
Joined
Nov 27, 2019
Messages
467
You need to list the specs of the system and how pool/datasets are setup to give some info to work with
 

matt998

Dabbler
Joined
Aug 14, 2021
Messages
16
OK not a problem. On top of the picture i am running a
MOB: B450 AORUS PRO WIFI
NIC: Mellanox ConnectX-2
Cable OM3 LC to LC Multimode Fiber
Network Switch: TL-SX2008P V1.0
SATA Expander: Ziyituod SATA Card, 4 Port with 4 SATA Cables, 6 Gbps SATA Controller PCI Express Expression Card with Low Profile Bracket, Boot as System Disk, Non Raid, Support 4 SATA 3.0 Devices
I am running it on RAIDZ2.


If there is any thing else let me know and I will get you the information.


1635348992115.png


1635349040845.png


1635349637573.png
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
So you have 8 hard drives in a RAIDZ2 pool? Of the top of my head the speed you are getting sounds about right for the pool you have constructed. In order to troubleshoot network speed issues you should use the proper tools, for example to test network speed use iperf. Once you are satisfied with the iperf results, then you can look into your pool configuration if you need to speed things up.
 

matt998

Dabbler
Joined
Aug 14, 2021
Messages
16
So this is my iperf test. I believe they are good at that speed. Can i change the RAIDZ2 pool to a better and much faster. Or to be able to have redundancy I have to have it at this pool setup.



1635357713554.png
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
I noticed "SATA Expander" - would you care to clarify as to exactly what that is please?
 

matt998

Dabbler
Joined
Aug 14, 2021
Messages
16
Not a problem. I have added the amazon link for it.

Ziyituod SATA Card, 4 Port with 4 SATA Cables, 6 Gbps SATA Controller PCI Express Expression Card with Low Profile Bracket, Boot as System Disk, Non Raid, Support 4 SATA 3.0 Devices

 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Yeesh. This is a bit of a hardware catastrophe. Between what appears to be a PCIe x1 crappy SATA controller and a crappy Mellanox ConnectX-2 card, ... shudder

You need to figure out your network problems first. You should be able to get better than the ~1Gbps that you are seeing on the Mellanox, but maybe not a ton better.
 

matt998

Dabbler
Joined
Aug 14, 2021
Messages
16
Yeesh. This is a bit of a hardware catastrophe. Between what appears to be a PCIe x1 crappy SATA controller and a crappy Mellanox ConnectX-2 card, ... shudder

You need to figure out your network problems first. You should be able to get better than the ~1Gbps that you are seeing on the Mellanox, but maybe not a ton better.

What suggestion would you give for the Sata controller and the NIC.
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
NIC: Chelsio or Intel
SATA Controller: No - Use a SAS Controller - with an LSI chipset flashed to IT Mode. Look on ebay for a system dismantler (do not buy one from China - there is a significant chance they will be fake) or goto The Art of Servers Ebay store: https://www.ebay.com/str/theartofserver?_trksid=p2047675.l2563 which will be preflashed
 

Hinterwaeldler

Dabbler
Joined
Sep 13, 2021
Messages
11
I recently spent some time tuning the performance of my homelab. The system is for research and has only cheap consumer grade hardware in it, that was mostly selected to be cheap and not make noise (i5-9500T, Asus XG-C100F).

1635784306600.png


Transfer speeds with smb were really shitty, like around 200mb onto a nvme drive. I did some probing around, but what finally made a huge impact was allowing the samba server to use multi channel support with Receive Side Scaling.

Essentially two additional lines in the auxiliary parameters of the smb service:
server multi channel support = yes
interfaces = "192.168.82.100;capability=RSS,speed=10000000000"

(obviously replace your ip)

1635783995233.png


This change alone allowed smb to essentially saturate the 10gbs connection.

1635784338701.png


Note that with the smb version deployed with truenas right now, multi channel support ist sill experimental.
 

matt998

Dabbler
Joined
Aug 14, 2021
Messages
16
I recently spent some time tuning the performance of my homelab. The system is for research and has only cheap consumer grade hardware in it, that was mostly selected to be cheap and not make noise (i5-9500T, Asus XG-C100F).

View attachment 50408

Transfer speeds with smb were really ****ty, like around 200mb onto a nvme drive. I did some probing around, but what finally made a huge impact was allowing the samba server to use multi channel support with Receive Side Scaling.

Essentially two additional lines in the auxiliary parameters of the smb service:
server multi channel support = yes
interfaces = "192.168.82.100;capability=RSS,speed=10000000000"

(obviously replace your ip)

View attachment 50405

This change alone allowed smb to essentially saturate the 10gbs connection.

View attachment 50409

Note that with the smb version deployed with truenas right now, multi channel support ist sill experimental.
thank you for the information when i get the new card in and if i have the same issue i will use this.
 

nullane

Cadet
Joined
Nov 2, 2021
Messages
4
@Hinterwaeldler I am having the same issue where it starts fast and drops down to 200-300mb like you. I tried the parameter you put in here, but it is still affecting me. Do you have any other tunables in place on TrueNAS?
 

matt998

Dabbler
Joined
Aug 14, 2021
Messages
16
@Hinterwaeldler I am having the same issue where it starts fast and drops down to 200-300mb like you. I tried the parameter you put in here, but it is still affecting me. Do you have any other tunables in place on TrueNAS?
I have kinda put it to the side for now because I am waiting for some parts to come in.
 

erdas

Dabbler
Joined
Dec 13, 2022
Messages
10
I recently spent some time tuning the performance of my homelab. The system is for research and has only cheap consumer grade hardware in it, that was mostly selected to be cheap and not make noise (i5-9500T, Asus XG-C100F).

View attachment 50408

Transfer speeds with smb were really shitty, like around 200mb onto a nvme drive. I did some probing around, but what finally made a huge impact was allowing the samba server to use multi channel support with Receive Side Scaling.

Essentially two additional lines in the auxiliary parameters of the smb service:
server multi channel support = yes
interfaces = "192.168.82.100;capability=RSS,speed=10000000000"

(obviously replace your ip)

View attachment 50405

This change alone allowed smb to essentially saturate the 10gbs connection.

View attachment 50409

Note that with the smb version deployed with truenas right now, multi channel support ist sill experimental.
I tried that on a 100gbe 455a with 4x gen3 in raid-0 and get only this:
1671949871205.png



using richcopy which is like robocopy, maybe i need to restart the smb... lemme try that... done... no difference.
and it's got rdma (not used on freeBSD) and rss on... weird
1671950083312.png
 

erdas

Dabbler
Joined
Dec 13, 2022
Messages
10
server multi channel support = yes
for me it drops the speed from 1.5GB/s to 400MB/s
interfaces = "192.168.82.100;capability=RSS,speed=10000000000"
this turns speed into a yoyo with some nice peaks
1671952246040.png

and CPU monitor in truenas shows that more cpu cores are engaged. it used to be one core peaking and the rest sleeping
1671952323866.png

Still don't know why the transfer is so mountainous, ssd transfer is far from peaking like ssd to ssd

1671952445893.png

I'd think it's the pcie, but no, the mellanox is in a cpu gen5 x8 pcie so it's got about 5GB/s leg room
SSD can read at 3.5GB/s via windows copy so let's say that this thing is about 1.5GB/s short of its peak.
The server and desktop are beefy, the server raid-0 is 4x gen3 which push raw speed of 12GB/s in crystalmark 8 threads and 8GB/s 1 thread sustained (when the server used to run windows 10)
i dunnot know
 

ankit.irse

Cadet
Joined
Jan 6, 2023
Messages
2
Screenshot 2023-01-07 at 12.13.18 AM.png
I am getting a max of 7 Mb/s ........ have a 250Gb cache nvme on B550i aorus pro ax. using all mother board sata ports only. in and out data both at the speed. tried the code but it says
 

Mdean211

Cadet
Joined
Mar 3, 2019
Messages
1
View attachment 62010 I am getting a max of 7 Mb/s ........ have a 250Gb cache nvme on B550i aorus pro ax. using all mother board sata ports only. in and out data both at the speed. tried the code but it says
Make sure you are pasting this in services>SMB (edit)>Advance Options and paste in the auxillery perameters at the bottom right.

This error usually shows if you are trying to put his in the pool options and not the share.
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,904
B550i aorus pro ax
It is likely the RealTek NIC on this board. See link in the "Recommended readings" in my signature for more information.
 
Top