10gbit network speed slow

[jakeblues]

Hi There,

I bought two 10Gbit cards (Intel X520-DA2 with AFBR-703SDZ-IN2 Transceiver) and I am getting only 1 Gbits.
In TrueNAS I use a bridge between the 1gbit and 10gbit NIC as I only have one PC where I need the fast connection to the NAS.

I've read several posts according to this topic but nothing helped so far.
The cards are original Intel with yotta-marks. And I placed it in the correct PCIe slot of the Supermicro board.
Both cards are being recognized as 10gb in Windows10 and in TrueNAS Core.

When I copy one single 1,5GB file from Windows (NVME) to NAS I get around 600MB/s.
When I copy it back to Windows there are only 130MB/s.
A mixed folder (mainly big videocam files) from NVME to NAS starts with 600MB/s and after a few seconds drops to 250MB/s

Iperf Bandwith says 1.47 Gbits/sec but I get a strange failure at the beginning:

tcp write failed: Broken pipe
shutdown failed: Socket is not connected

I ran "iperf3.exe -s" on Windows and "iperf -c x.x.x.x" on TrueNAS both listening on the same port

I would be very glad if someone could help me out with this!

 

[artlessknave]

a single raidz2 of spinning disks is unlikely to give you 10gbe speed. what you are seeing is likely 600MB/s until the ARC is full and starts having to write everything to disk. any file that can completely fit into RAM will show 10gbe speeds, and then be written to disk in the background. as you only have 32GB of RAM, this is not going to include many media files.

without more vdevs you are going to be limited to speed of the raidz2, which is the speed of 1 disk irrc. spinners rarely make it much over 150MB/s. the 250MB/s is probably ZFS helping.

if you want nvme speeds, you need to have NVME on both sides.

a mirror based pool will give you better speeds, as both writes will be simpler and you would have more vdevs.

When I copy one single 1,5GB file from Windows (NVME) to NAS I get around 600MB/s.
When I copy it back to Windows there are only 130MB/s.

file going to ARC will be fast (slowest of: RAM, Network)
file reading from disk will be the speed of the pool (slowest of Pool/RAM/Network) UNLESS the file is cached in ARC, but your ARC is kinda small.

 

[jakeblues]

Thank you for your insight! You stated a lot of interesting facts I didn't consider!

Then the good news are, my cards work perfectly fine. The bad news is I think I can hand them back as there won't be any speed upgrade to 1gbit. As I understand going for more RAM would only speed up transfers temporarly or help with files up to 128GB (if RAM maxed out). Or do you think upgrading to 64GB would be a big improvement (more would blow my budget . Would a cache SSD speed transfers up (would only speed up read speed as it is no ARC chache)? If yes, wich minimum size would you suggest? I don't want to do a mirror based pool as my main concern is data safety and (one big) diskspace. But perhaps I'll invest in a 2 x 2TB SSD mirrored vdev Pool in the future (for video editing). Would you have any further suggestions? Thanks a lot!

 

[chuck32]

In TrueNAS I use a bridge between the 1gbit and 10gbit NIC as I only have one PC where I need the fast connection to the NAS.

So basically my setup. I would advise you to not bridge the two interfaces. Just use the 10g NIC on truenas without a bridge and assign it to a new subnet (e.g. 10.5.10.10/24). Assign an IP in windows to your 10g NIC in the same subnet. Mount your shares using the new IP. This makes sure that the 10G will be used exclusively for your file transfers.

When I copy it back to Windows there are only 130MB/s.

It should be in ARC and therefore a lot faster when you accessed it prior to copying,

The bad news is I think I can hand them back as there won't be any speed upgrade to 1gbit.

No, even as raidz is slow, the write / read performance of a single disk will saturate a 1g connection. A single HDD will give you read write speeds around 200 MB/s (+-), so double 1g speeds.

But perhaps I'll invest in a 2 x 2TB SSD mirrored vdev Pool in the future (for video editing).

Keep the cards and do that.

Would a cache SSD speed transfers up (would only speed up read speed as it is no ARC chache)?

L2ARC I'd increase memory to at least 64GB before considering then L2ARC device. If I was you: stay at 32 GB (if you don't need more memory for other reasons like VMs anyway) and invest in the SSD pool for video editing, this should give you best performance boost in my opinion. With ARC you are always dependent on the files beeing stored in ARC vs. if your pool is fast enough, you don't need to pay attention to that. But this is just my personal opinion.

Also since you are using windows, look at automatic metrics for the network interfaces. Windows will determine which interface to use based on its speed. Personally I found with multiple interfaces (VMware from some testing also added new network adapters) my browsing experience took a hit, since my 1g ethernet connection was not the preferred NIC anymore. For your file sharing it does not matter, because you access it over the 10G only anyway (as mentioned in my first paragraph).

 

[jakeblues]

So basically my setup. I would advise you to not bridge the two interfaces. Just use the 10g NIC on truenas without a bridge and assign it to a new subnet (e.g. 10.5.10.10/24). Assign an IP in windows to your 10g NIC in the same subnet. Mount your shares using the new IP. This makes sure that the 10G will be used exclusively for your file transfers.

I wanted to do exactly that but failed as I found a post, that stated it is not possible to perform a static route on a seperate NIC. The post said that static routes are host-based not interface-based. My initial plan was to configure the internal two NICs as LACP and the 10g NIC as an exclusive static subnet-route between NAS and PC. Could you share how you accomplished that?

 

[chuck32]

You don't need a static route.

I assume your 1 Gb connection / home network is correctly configured and does not need any change. Remove the 10Gb interface from the bridge.
I also assume your PC is directly connected to your truenas machine via the two 10 Gb cards.

Something like this:

This is my 10Gb interface on truenas:

In windows I assigned a static IP in the same subnet:

And then just mount your shares using the same subnet

My local homenetwork lives in the 192.168.X.X space.

 

[jakeblues]

Holy cow! That was simple and it worked right away! Thanks a lot!!!!

But perhaps I'll invest in a 2 x 2TB SSD mirrored vdev Pool in the future (for video editing).

Keep the cards and do that.

Would you set this up as iSCSI or SMB to access it from Windows?

 

[chuck32]

No problem!

I haven't looked into is iSCSI at all in my homelab journey so far, I use SMB for everything.

The good thing is (my wife's computer is not on a 10g link) for all other machines you can still mount the shares with your other home Network IP. So this solution does not limit you at all as far as I can tell.

 

[jakeblues]

I encountered another problem: Since I only tested 10gb transfer-connections I ran into a problem with the 1gb connection. Write speeds dropped to 0. Read speeds were fine at 120 MBs. So I turned off Sync and write speeds reached also 100. I read that it is not recommended to turn off sync. Is there another possibility to increase speed with sync on? Otherwise I have to give up ZFS cause I need the 1gb shares.

 

[chuck32]

Are we talking SMB? I have it set to Standard (Inherit) and 0 problems.

Please share your Network config in truenas and show what is configured in Windows.

 

[jakeblues]

Sorry, yes. SMB.What TrueNAS network config do you mean? Global config? In windows I didn't alter any default settings. What screen do you want to see?

 

[chuck32]

The Network tab where you define your interfaces.
Did you check the link about Windows metrics?

10g is working fine?

I just realized, I didn't check your signature, are those SMR drives? Quick research says yes, that's not good.. check the forums for info on that.

Do you have the same problems over 1g when accessing the ssd pool?

 

[jakeblues]

Do you mean MTU? I left it blank so it should be at 1500. 10g is working fine. I have no SSD share. Yes, they are SMR. That are really bad news! Flicked through some posts. Overheating, data-loss... hurray. I guess that was a short trip to ZFS. As new drives are no option I guess I'll have to move to unraid with XFS than...

 

[artlessknave]

I read that it is not recommended to turn off sync.

where did you read that? sync is rarely needed. sync is off by default, and smb cannot interface with sync at all.

you can improve speeds with sync with SLOG, but should be a whole other thread.
the quick version:
SLOG will never be as fast as async, nor is it a cache as is often mistakenly stated. SLOG is a backup of the ARC write cache, and will still slow your write speeds down overall, but be fast then sync to spinners, as all sync writes are written 3 times. once to ARC, once to SLOG/ZIL (ideally this is in parallel to the ARC write), once to the pool.
to get sync with SMB you must turn it on for the dataset shared via smb.
enabling sync without a SLOG means that every write MUST be written to the pool NOW, interrupting anything else the pool is doing to write to the disk in a temporary location, THEN write that again to the correct location in the pool when there are no SYNC writes being requested. with everything as sync writes, you are tanking all writes.
sync is only used for things like databases and VM storage, where any lost write is catastrophic.

 

[jakeblues]

Thank you for the insight! Then I'm not worrying about turning it off any more. But the SMR problem still remains. Would you have any suggestions how to deal with that without buying new drives and leaving the truenas environment?

 

[artlessknave]

deal with that without buying new drives and leaving the truenas environment?

leaving truenas will not help with the SMR drives. they are junk in any kind of disk arrays. they arent great drives to begin with, but anything requiring timely return of data that isnt cached they can be pretty horrible. I'll buy SSDs before SMR.

the only way to fix SMR drives is to not have SMR drives.