10GB transfer

[BRIO]

I am not the 1st one to post on the subject but since my specs differ from other people I wanted to share my case precisely.

Basically, I just finished build and setup my first NAS with TrueNas but for some reason my write transfer on the NAS won't go over around 113MB/s speed.

NAS specs:
Motherboard: ASUS Prime X570-Pro
CPU: AMD Ryzen 5 5600G
RAM: T-Force Vulcan DDR4 32GB (2X16GB)
10GB PCIe Card: Intel X540-T2 Dual 10GB port ( https://www.ebay.com/itm/143592050461?var=442552216136 )

Computer:
Motherboard: MSI Prestige X570 Creation ( https://www.newegg.ca/msi-prestige-x570-creation/p/N82E16813144258?Item=N82E16813144258 )
CPU: AMD Ryzen 9 5900X
RAM: T-Create DDR4 64GB (2X32GB)

Wires: Amazon basic Cat-6 ( https://www.amazon.ca/-/fr/gp/produ...le_10?smid=A3DWYIK6Y9EEQB&th=1&language=en_CA )

My NAS is connected through the 10GB PCIe card ---> to the cat-6 wire going ---> in the 10GB port of the Prestige X570 motherboard

Do I need to adjust some settings somewhere to enable the 10GB transfer? Specify to transfer through the 10GB port? Adjust the bios of the NAS? Change something in my TrueNas settings?

Anyway let me know if you guys have any idea of how I could make it work. Thanks in advance!

 

[BRIO]

I forgot to mention that I have an M.2 1TB cache drive on the nas as well

 

[jgreco]

I forgot to mention that I have an M.2 1TB cache drive on the nas as well

Well that's inappropriate. Why do you have this?

With 32GB of RAM, we don't even recommend L2ARC at all; but the recommended ratio of L2ARC to ARC starts at 5:1 and shouldn't exceed 10:1. And that's ARC. If you are using SCALE with its typical 1/2 ARC limit, that means that you need 256GB of RAM to hit an 8:1 ratio (or 128GB of RAM on CORE for the same).

10GB PCIe Card: Intel X540-T2 Dual 10GB port

Do I need to adjust some settings somewhere to enable the 10GB transfer?

Well, two or three things at least.

One, copper 10GBase-T is horribly bad and gives newbies lots of trouble. You should have followed the recommendations in the 10 Gig Networking Primer

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

Intel ARK indicates that the Twinville cards are discontinued, so this is a super-strong indication that a card sold as "new" on eBay from China is a knock-off or complete fake.

Fake server cards

Some time ago, forum frequenter @artlessknave kindly sent me a pair of LSI HBA cards that had failed to work out in a FreeNAS build. Having suspected that these were fakes, I asked for them to be shipped here for comparison. However, COVID...

www.truenas.com

Your card is clearly transferring at 1Gbps, which is why you get that flatline 113MBytes/sec. The Chinese knock-off cards often work fine but sometimes don't, so you might pursue some testing to determine why this isn't working. However, since you may only have that one card available and then the built-in on the other mainboard, that's going to be a difficult proposition.

In general, copper is finicky and any loss, crosstalk, etc., will cause the signal to degrade. Since your signal may be subpar to begin with due to what I expect is a knockoff card with substandard parts, you may not be able to get this to work reliably. Check what link speed is reported on the network cards on both ends and see if you can get it to link at 10GBase-T.

Otherwise, eat the loss and instead get yourself some inexpensive fiber stuff following the guidance in the 10 Gig Networking Primer.

 

[BRIO]

Well that's inappropriate. Why do you have this?

With 32GB of RAM, we don't even recommend L2ARC at all; but the recommended ratio of L2ARC to ARC starts at 5:1 and shouldn't exceed 10:1. And that's ARC. If you are using SCALE with its typical 1/2 ARC limit, that means that you need 256GB of RAM to hit an 8:1 ratio (or 128GB of RAM on CORE for the same).




Well, two or three things at least.

One, copper 10GBase-T is horribly bad and gives newbies lots of trouble. You should have followed the recommendations in the 10 Gig Networking Primer

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

Intel ARK indicates that the Twinville cards are discontinued, so this is a super-strong indication that a card sold as "new" on eBay from China is a knock-off or complete fake.

Fake server cards

Some time ago, forum frequenter @artlessknave kindly sent me a pair of LSI HBA cards that had failed to work out in a FreeNAS build. Having suspected that these were fakes, I asked for them to be shipped here for comparison. However, COVID...

www.truenas.com

Your card is clearly transferring at 1Gbps, which is why you get that flatline 113MBytes/sec. The Chinese knock-off cards often work fine but sometimes don't, so you might pursue some testing to determine why this isn't working. However, since you may only have that one card available and then the built-in on the other mainboard, that's going to be a difficult proposition.

In general, copper is finicky and any loss, crosstalk, etc., will cause the signal to degrade. Since your signal may be subpar to begin with due to what I expect is a knockoff card with substandard parts, you may not be able to get this to work reliably. Check what link speed is reported on the network cards on both ends and see if you can get it to link at 10GBase-T.

Otherwise, eat the loss and instead get yourself some inexpensive fiber stuff following the guidance in the 10 Gig Networking Primer.

Alright, I'm pretty new to all this, I have a couple of questions:

1. I'm a bit lost with the 32GB of ram VS L2ARC - What does that mean exactly? 32GB of RAM is not enough for what I'm trying to do?
2. How do I check my "link speed" on both ends of the network exactly?
3. Is there a way to check exactly the max speed I can get from the cheap Ebay card I got? That way I can see if this is really the problem, if it is I will just get a refund and get a better card like you pointed

I got some updates on my side:

1. My first problem was that my server was linked on my internet port and not on the 10GB port, probably why I couldn't get higher than 113MB/s. Now that it is linked to the 10GB port I manage to get a little over 1GB but it starts to drop around 300mb/s after some time and it's still far from the 10GB I'm trying to get.

2. I did a speed test with OpenSpeedTest for server and got pretty much the speed I'm looking for. Only, in practice it's not even close to that though as you can see above

Curious to have some of your thoughts this

 

[jgreco]

I'm a bit lost with the 32GB of ram VS L2ARC - What does that mean exactly? 32GB of RAM is not enough for what I'm trying to do?

It means that you should have either 128GB (CORE) or 256GB (SCALE) RAM in order to support an L2ARC that large. That board looks limited to 128GB, so you could run that L2ARC on CORE and that would be okay-ish.

My first problem was that my server was linked on my internet port and not on the 10GB port, probably why I couldn't get higher than 113MB/s. Now that it is linked to the 10GB port I manage to get a little over 1GB

Okay, so, this is just you and Microsoft, then. You're both using crappy abbreviations and it is causing you to miscommunicate. Please see

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

You don't have a 10 gigabyte ("GB") port. That's a measurement of storage. You have a 10 gigabit-per-second port. ("Gbit/sec"). You are seeing the "GB/s" on the Microsoft display and misunderstanding what it means; it means "Gigabytes per second". Now, since there are 8 bits per byte, you have to multiply that number by 8 to convert it to "Gigabits per second"; so you are getting about 8.24 Gbit/sec on a copy when it says "1.03GB/s". Actually just a bit more than 8.24, since there is network overhead. That's super good news for you because it means that you managed to score a 10G card that seems to work. Getting refunds from China is a painful process.

Now that it is linked to the 10GB port I manage to get a little over 1GB but it starts to drop around 300mb/s after some time and it's still far from the 10GB I'm trying to get.

I would indeed be concerned if you were getting 300 millibits per second after some time (see, again, use of crappy abbreviations leading to ambiguity and miscommunication). However, if we assume you mean 300MBytes/sec, that could be entirely reasonable. If you had something like a single RAIDZ2 vdev with 4 hard drives, that's right in the range of what you might expect on throughput. An average hard drive tops out at about 100-150MBytes/sec, so two or three of them writing data will be about 300-450MBytes/sec, and the other two are busy writing parity data. Any fragmentation would slow that down. But you didn't indicate anything about your storage pool so I have to assume that you're limited to no more than the 6x SATA provided by the mainboard. This could still be fine for a 6 drive RAIDZ2.

 

[BRIO]

It means that you should have either 128GB (CORE) or 256GB (SCALE) RAM in order to support an L2ARC that large. That board looks limited to 128GB, so you could run that L2ARC on CORE and that would be okay-ish.



Okay, so, this is just you and Microsoft, then. You're both using crappy abbreviations and it is causing you to miscommunicate. Please see

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

You don't have a 10 gigabyte ("GB") port. That's a measurement of storage. You have a 10 gigabit-per-second port. ("Gbit/sec"). You are seeing the "GB/s" on the Microsoft display and misunderstanding what it means; it means "Gigabytes per second". Now, since there are 8 bits per byte, you have to multiply that number by 8 to convert it to "Gigabits per second"; so you are getting about 8.24 Gbit/sec on a copy when it says "1.03GB/s". Actually just a bit more than 8.24, since there is network overhead. That's super good news for you because it means that you managed to score a 10G card that seems to work. Getting refunds from China is a painful process.



I would indeed be concerned if you were getting 300 millibits per second after some time (see, again, use of crappy abbreviations leading to ambiguity and miscommunication). However, if we assume you mean 300MBytes/sec, that could be entirely reasonable. If you had something like a single RAIDZ2 vdev with 4 hard drives, that's right in the range of what you might expect on throughput. An average hard drive tops out at about 100-150MBytes/sec, so two or three of them writing data will be about 300-450MBytes/sec, and the other two are busy writing parity data. Any fragmentation would slow that down. But you didn't indicate anything about your storage pool so I have to assume that you're limited to no more than the 6x SATA provided by the mainboard. This could still be fine for a 6 drive RAIDZ2.

Alright! It seems to slowly comes together. Yes you are right, I mean 10GB/s and not 10GB only, sorry about that. My setup is currently:

-RAIDZ1
-3X 16TB WD ULTRASTAR SATA
-1TB M.2 Cache VDev

Though since I noticed I couldn't make my system work in RAIDZ2 with only 3 HD I just ordered a 4th 16TB drive. My stetup is going to be:

-RAIDZ2
-4X 16TB
-1TB M.2 Cache VDev


1. You mentioned that I'm getting 8.24 Gbit/sec on a copy when it says "1.03GB/s". I actually thought I would get around 10GB/s when I would copy files on my server and not around 1GB/s. Am I missing something or 1GB/s is normal?

2. Once I get 128GB of RAM, what will it helps for exactly? Will it be useful for the speed of my files transfer?

 

[jgreco]

8.24 Gbit/sec on a copy when it says "1.03GB/s". I actually thought I would get around 10GB/s when I would copy files on my server and not around 1GB/s. Am I missing something

Yes, you missed my sly dig at Microsoft when referring to the Terminology and Abbreviations Primer. Microsoft is guilty here too, because they're too lazy to spell out "1.03GBytes/sec" or "1.03 Gigabytes/sec", which you need to multiply by 8 to get Gigabits per second. Your network card works in Gigabits per second, so the 8.24 Gigabits per second is reasonable.

thought I would get around 10GB/s when I would copy files on my server

In order to get 10 GigaBYTES per second, you would need a network card that does more than 80 GigaBITS per second (or, in other words, a 100Gbit/sec ethernet card). Yours is only 10 Gigabits per second. Sorry to disappoint. Hopefully it is clear why it is so important to use unambiguous abbreviations or even to spell it out. I've been doing this for many years and I easily translate stuff in my head, at least when the units are reasonable. However, even with years of experience, it is jarring and dissonant when units are mixed or misused. I *know* it has got to be harder for newcomers than it is for me.

 

[BRIO]

Ok sounds great to me, thanks a lot for clearing everything up!

 

[jgreco]

Given that many of these threads wind up with people having been taken for a ride by crappy ethernet cards that cannot be made to work well, yours was a refreshing change of pace. Good luck and enjoy.

 

[BRIO]

Given that many of these threads wind up with people having been taken for a ride by crappy ethernet cards that cannot be made to work well, yours was a refreshing change of pace. Good luck and enjoy.

Hey I encountered an other problem actually. When my server is connected, I get a jerky internet speed on my computer (takes roughly 1 to 3 seconds to load some pages).

My guess is that there is a problem with the fact that everything go through the same router in addition to have connection to the server but I might be wrong. Here's my setup, let me know if you have an idea of what the problem might be!

 

[sretalla]

Probably worth noting that standard abbreviations are:

GB/s or GBps (GigaBytes per second)

Gb/s or Gbps (Gigabits per second) sometimes also Gbits/s to make it clearer.

Bytes with uppercase B and bits with lowercase b.

Microsoft is conforming to that.

So is OpenSpeedTest with Mbps.

Your diagram still shows GB/s when you mean gigabits.

 

[Whattteva]

Hey I encountered an other problem actually. When my server is connected, I get a jerky internet speed on my computer (takes roughly 1 to 3 seconds to load some pages).

It is (usually) always DNS.

 

[BRIO]

Probably worth noting that standard abbreviations are:

GB/s or GBps (GigaBytes per second)

Gb/s or Gbps (Gigabits per second) sometimes also Gbits/s to make it clearer.

Bytes with uppercase B and bits with lowercase b.

Microsoft is conforming to that.

So is OpenSpeedTest with Mbps.

Your diagram still shows GB/s when you mean gigabits.

Alright, sorry about that, will rectify next time

It is (usually) always DNS.

Ok and how can I fix that?

 

[Whattteva]

Ok and how can I fix that?

Usually by not using your ISP's DNS and also by running your own DNS cache. If I remember correctly, recent Ubuntu-based installations already do the latter by default.

 

[Davvo]

Hey I encountered an other problem actually. When my server is connected, I get a jerky internet speed on my computer (takes roughly 1 to 3 seconds to load some pages).

Are you sure the issue lies within your internet speed?
Is this verified by a speed test?

 

[BRIO]

Usually by not using your ISP's DNS and also by running your own DNS cache. If I remember correctly, recent Ubuntu-based installations already do the latter by default.

Ok thanks a lot, I will look into that!

Little question, how do I run another DNS than my ISP exactly? Do I need to change it manually in Windows? Or it is un TrueNas?

Are you sure the issue lies within your internet speed?
Is this verified by a speed test?

Yeah, actually internet speed doesn't seems to be "affected". When I run my test, it's roughly the same numbers as when the server is not connected. This seems more like a "slow down" like I pointed it takes around 1 to 3 seconds to load pages at time