32GB ram but speed slows down after a few seconds

[vimsi]

Hi @ all,
I know there are A LOT of those threads already but none did really help me figure out how to solve this issue (or they are already several years old).

I have a dedicated Server with TrueNas Scale running:

In there I have a Software-Raid 5(RAIDZ1) of 3x4TB HDD from Toshiba with 7200rpm and 128MB Cache.
My home network between the NAS and PC is 10Gbit - iperf speed test also confirms this speed in both directions.

If I copy a large file the speed plummets after a few seconds:
from:

to:

What can I do to maintain the speed the whole time?

Thank you for reading this far :)

 

[WI_Hedgehog]

The data has something like 5 seconds to sit in memory and then starts to ferment, so if the drive array cannot keep up the server will stop accepting new data until the current data is written. Tweaking the settings manually is dangerous.

You can, in my limited experience, set the array up so it's faster, add an NVME cache, or use Ubuntu and software RAID with ext4.

 

[Davvo]

In order to add L2ARC you need at least 64GB of RAM.
You can also change the block size of the dataset to match your files size.
Besides, RAIDZ# is not the most performing configuration: mirrors are.
Even then, you don't have enough drives to saturate your network.
Oh, and make sure you aren't using a Realtek NIC.

Suggested readings:

ZFS Storage Pool Layout

This resource was originally created by user: @Davvo on the TrueNAS Community Forums Archive. https://www.truenas.com/community/resources/zfs-storage-pool-layout.201/download [1] This amazing document, created by iXsystems in February 2022 as a “White Paper”, cleanly explains how to qualify...

www.truenas.com

Is my Realtek ethernet really that bad?

This resource was originally created by user: jgreco on the TrueNAS Community Forums Archive. Please DM this account or comment in this thread to claim it. One of the longest running issues faced here in the forums may be people who come to these forums, wanting to recycle an old PC into a NAS...

www.truenas.com

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

 

[Johnny Fartpants]

Can we see a screenshot of your dataset settings just to sanity check them?

 

[Johnny Fartpants]

So your RAID-Z1 of 3 drives will have a streaming write speed of 2 of your 3 drives. So let’s say the drives can do approx 185MB/s you would get two times that so 370MB/s or 2.96Gb/s. This is the absolute theoretical best you can expect from your current setup when hitting your data pool just to put things into perspective.

 

[vimsi]

Can we see a screenshot of your dataset settings just to sanity check them?

this details?

In order to add L2ARC you need at least 64GB of RAM.
You can also change the block size of the dataset to match your files size.
Besides, RAIDZ# is not the most performing configuration: mirrors are.
Even then, you don't have enough drives to saturate your network.
Oh, and make sure you aren't using a Realtek NIC.

so I have to double my ram at least?
i am not using realtek i think.
should i use more HDDs instead of larger ones?

 

[Constantin]

To saturate a 10GbE connection on write with HDDs likely requires at least 4 Zx VDEVS, if not more… and that’s only if you’re writing big, continuous files to a fairly empty pool. Small file saturation @ 10GbE to HDD would likely require multiples more, and then only if you have multiple users doing the saturating.

“Caches” aren’t used by TrueNAS like they are on other platforms, so you won’t find simply adding a SSD to suddenly speed things up. TrueNAS cares first and foremost about permanence - so writing something to a fast cache and potentially mangling it in there before transferring mangled content to the HDD pool is not something they’re interested in. Never mind losses, etc.

You can create fusion pools where SSDs help HDDs by picking up the small files and handling directory services. But your fusion pool will die if either the parity limits of your HDD VDEVs or the SSD VDEVs are exceeded. Plus, some planning up front is a really good idea since the GUI still doesn’t alert the admin re a full sVDEV.

I’m kind of wondering if what windows was showing you re progress was some kind of mirage. I’d confirm the kind of network transfer speed at the NAS GUI.

 

[WI_Hedgehog]

I'll mention as a friendly note many gamers see YouTube videos saying how great TrueNAS is, and it is, but usually not in the context of game data storage.

Gamers are looking for raw speed at the expense of reliability--if a few bits are lost here and there it's usually imperceptable. Gamers usually try to repurpose old gaming systems, which in their own right are very fast, however also not well-suited to industrial storage solutions like TrueNAS. For ripping data fast use Ubuntu, for reliability and speed use TrueNAS--but throw datacenter grade equipment at it; just like a gaming system is designed for a specific purpose a data storage system is also, and they're vastly different in requirements.

I mention this because many gamers are disappointed in TrueNAS performance without understanding the purpose of TrueNAS and without having configured a system that meets their desires. Whatever your goals we want you to achive them and are supportive of you having a great outcome within your budget.

 

[Constantin]

It’s a philosophy you have to consider important before going down its path. Data integrity supreme is not for everyone.

But if data integrity is important to you, then TrueNAS is a great vehicle to ride down the ZFS path to potential zero data corruption/loss.

 

[Johnny Fartpants]

this details?

Yep. That all looks fine.

 

[vimsi]

I'll mention as a friendly note many gamers see YouTube videos saying how great TrueNAS is, and it is, but usually not in the context of game data storage.

Gamers are looking for raw speed at the expense of reliability--if a few bits are lost here and there it's usually imperceptable. Gamers usually try to repurpose old gaming systems, which in their own right are very fast, however also not well-suited to industrial storage solutions like TrueNAS. For ripping data fast use Ubuntu, for reliability and speed use TrueNAS--but throw datacenter grade equipment at it; just like a gaming system is designed for a specific purpose a data storage system is also, and they're vastly different in requirements.

I mention this because many gamers are disappointed in TrueNAS performance without understanding the purpose of TrueNAS and without having configured a system that meets their desires. Whatever your goals we want you to achive them and are supportive of you having a great outcome within your budget.

well, i am for data security over everything else!
beside my raid i drive a daily backup onto anoethr drive - and i am also building a second raid 5 on another server to backup everything once again.. i dont want to lose any data anymore in my life
and as you swa i don't repurpose an old gaming rig - all my 3 servers running xeon CPUs with DDR ECC Memory at least

But since I also invested into 10Gbit ethernet I would like to have not only data integrity and security but also speed.

To saturate a 10GbE connection on write with HDDs likely requires at least 4 Zx VDEVS, if not more… and that’s only if you’re writing big, continuous files to a fairly empty pool. Small file saturation @ 10GbE to HDD would likely require multiples more, and then only if you have multiple users doing the saturating.

“Caches” aren’t used by TrueNAS like they are on other platforms, so you won’t find simply adding a SSD to suddenly speed things up. TrueNAS cares first and foremost about permanence - so writing something to a fast cache and potentially mangling it in there before transferring mangled content to the HDD pool is not something they’re interested in. Never mind losses, etc.

You can create fusion pools where SSDs help HDDs by picking up the small files and handling directory services. But your fusion pool will die if either the parity limits of your HDD VDEVs or the SSD VDEVs are exceeded. Plus, some planning up front is a really good idea since the GUI still doesn’t alert the admin re a full sVDEV.

I’m kind of wondering if what windows was showing you re progress was some kind of mirage. I’d confirm the kind of network transfer speed at the NAS GUI.

i rather keep it safe and simple and don't mess around with Fusion Pools.. I'd rather make a raid out of a couple more drives, maybe 5 instead of 3. eventually this gives me a couple more MB/s

Yep. That all looks fine.

at least one good news :D

 

[Davvo]

well, i am for data security over everything else!
beside my raid i drive a daily backup onto anoethr drive - and i am also building a second raid 5 on another server to backup everything once again.. i dont want to lose any data anymore in my life
and as you swa i don't repurpose an old gaming rig - all my 3 servers running xeon CPUs with DDR ECC Memory at least

But since I also invested into 10Gbit ethernet I would like to have not only data integrity and security but also speed.


i rather keep it safe and simple and don't mess around with Fusion Pools.. I'd rather make a raid out of a couple more drives, maybe 5 instead of 3. eventually this gives me a couple more MB/s

Please, stop using hardware RAID terminilogy.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com