Slow write speed, drop after a few seconds

[wonders]

I did find similar topics but none of them were relevant or were resolved!

System spec:

HP DL380p Gen8
CPU: 2 x Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
RAM: 64 GB
HBA: HP H220 - IT Mode v20
Disks: 8 x SAS 10K Seagate ST900MM0006 (900GB) RAIDZ2
NIC: Mellanox ConnectX2 10gbe

Please see below for the details of my pool and disks, I don't have any cache or logs, last disk is a 128Gb ssd connected via sata and it's used for boot!






Everything seems to be working fine, I can get stable network speed of 9.5 Gbps tested with iperf3 (send and received )
For testing purpose I have connected my PC directly to the NAS using same NIC (10gbe) and everything is okay and stable in terms of network speed!

However when I try to write to NAS, through SMB and Windows 11, using a 20GB file it starts good, with the speed of 1GB/s then it drops to about 200MB/s. If I try to copy large number of small files then it goes down to 1 MB/s extremely slow!



However when I read from the NAS and copy files back to my PC, it is perfect and it stays with 1GB/s which is exactly what I expected!

Just to add I use the NAS, as backup storage, running a couple of VMs and as a Plex media server!

Appreciate it if anyone can help me understand what could be the issue!

 

[morganL]

I did find similar topics but none of them were relevant or were resolved!

System spec:

HP DL380p Gen8
CPU: 2 x Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
RAM: 64 GB
HBA: HP H220 - IT Mode v20
Disks: 8 x SAS 10K Seagate ST900MM0006 (900GB) RAIDZ2
NIC: Mellanox ConnectX2 10gbe

Please see below for the details of my pool and disks, I don't have any cache or logs, last disk is a 128Gb ssd connected via sata and it's used for boot!

View attachment 57431

View attachment 57432
View attachment 57433

Everything seems to be working fine, I can get stable network speed of 9.5 Gbps tested with iperf3 (send and received )
For testing purpose I have connected my PC directly to the NAS using same NIC (10gbe) and everything is okay and stable in terms of network speed!

However when I try to write to NAS, through SMB and Windows 11, using a 20GB file it starts good, with the speed of 1GB/s then it drops to about 200MB/s. If I try to copy large number of small files then it goes down to 1 MB/s extremely slow!

View attachment 57434

However when I read from the NAS and copy files back to my PC, it is perfect and it stays with 1GB/s which is exactly what I expected!

Just to add I use the NAS, as backup storage, running a couple of VMs and as a Plex media server!

Appreciate it if anyone can help me understand what could be the issue!

When the file write 1st starts, its writing to cache (in this case RAM)
When the cache is full, writes have to go the the drive pool.
1 RAIDZ2 group = a few hundred IOPS.. 200 MByte =200 IOPS
More VDEVs = more IOPS


Copying files is often slow because the Copy process is
read 1 chunk....
write that chunk..
wait for it to complete
Repeat

it's like passing water through a bucket rather than a pipe.... Better to have multiple copy processes running in parallel

If the files are small, its like passing water through tiny buckets

 

[wonders]

When the file write 1st starts, its writing to cache (in this case RAM)
When the cache is full, writes have to go the the drive pool.
1 RAIDZ2 group = a few hundred IOPS.. 200 MByte =200 IOPS
More VDEVs = more IOPS


Copying files is often slow because the Copy process is
read 1 chunk....
write that chunk..
wait for it to complete
Repeat

it's like passing water through a bucket rather than a pipe.... Better to have multiple copy processes running in parallel

If the files are small, its like passing water through tiny buckets

Thank you, that make sense in terms of Ram, but how can I improve it, how come no one else has this issue!
In terms of small files, what's the solution? What do I need to do to improve the performance?

 

[c77dk]

How are the datasets setup regarding sync? Sync writes hurts performance

 

[wonders]

How are the datasets setup regarding sync? Sync writes hurts performance

I have disabled sync in all datasets and the pool! Same results!

I just tried a folder of about 2000 small files, total size of 300MB, and it was copying them at 40 KB/s, that can't be right! :(

 

[diogen]

RE: Copying large files between NAS and PC
The settled speed of around 240MB/s is the upper write limit to a single drive.
When copying to the PC you most likely copy to a SSD - don't expect such write speeds to the NAS.

RE: Copying many small files
Try a different 10Gbe card... Or simply try copying over a 1Gbe card...

Issues with Mellanox Connect-X 3 cards

Hey friends. Recently i have upgraded my home lab and installed Mellanox Connect-X 3 Dual 40Gbps QSFP cards in all of my systems. My TrueNAS system is running on a dedicated machine, and is connected to my virtualization server through 2x 40Gbps links with LACP enabled. All my virtual machines...

www.truenas.com

 

[wonders]

RE: Copying large files between NAS and PC
The settled speed of around 240MB/s is the upper write limit to a single drive.
When copying to the PC you most likely copy to a SSD - don't expect such write speeds to the NAS.

RE: Copying many small files
Try a different 10Gbe card... Or simply try over a 1Gbe card...

That's right, I'm copying them to my PC on a ssd drive, make sense!
However considering the initial boost when writing to Nas is caused by writing to ram, is there anyway I can increase this cache/ram to make sure at least for a 20GB file, it is fully done on the ram/cache then the server takes care of copying it to the disks?

Also you mentioned 240MB/s is speed on writing to a single disk, does that main raidz2 isn't making any difference in the write speed?

I tried with a 1gbe NIC but the same result!

 

[diogen]

...is there anyway I can increase this cache/ram to make sure at least for a 20GB file

I don't know the "logistics" of the copy process... And am not sure you can change it...

Also you mentioned 240MB/s is speed on writing to a single disk...

Where does the 20GB file (in the first test) come from? SSD or another HDD?

...does that main raidz2 isn't making any difference in the write speed?

I wouldn't expect the difference to be significant...

I tried with a 1gbe NIC but the same result!

1Gbe NIC on the NAS? Or on the PC? Or both?

10G networking on TrueNAS must be done right to take full advantage of it.
And having iperf3 report the "right" numbers isn't enough...

 

[wonders]

I don't know the "logistics" of the copy process... And am not sure you can change it...

Where does the 20GB file (in the first test) come from? SSD or another HDD?

I wouldn't expect the difference to be significant...

1Gbe NIC on the NAS? Or on the PC? Or both?

10G networking on TrueNAS must be done right to take full advantage of it.
And having iperf3 report the "right" numbers isn't enough...

I meant increasing cache in TureNAS

20GB test file is on my PC and stores on a SSD

1Gbe card on both PC and Nas!

I'm almost certain the NIC is configured correctly as when I try copying a 3GB file it gives me full and consistent speed!
The problem is only when the file is large!

 

[wonders]

I'm confused now!
The first reply by @morganL suggest that this is expected considering number of vdevs, and for the small files it's because there is one process trying to copy large number of files!

But others suggest there might be an issue that needs to be found!

If the first few gig is copied into the ram, is there anyway to increase RAM used by TrueNAS and improve write speed for a few gigs more!?

I tried adding an SSD as Cache and log but addition of neither made any difference!

 

[morganL]

I meant increasing cache in TureNAS

20GB test file is on my PC and stores on a SSD

1Gbe card on both PC and Nas!

I'm almost certain the NIC is configured correctly as when I try copying a 3GB file it gives me full and consistent speed!
The problem is only when the file is large!

It think its a pool issue.

How large are the small files?
What is the ZFS record size for the dataset? (too large or too small can have negative impacts)

You get more performance from more VDEVs. so arranging your drives as 4x mirrors would give you 4X write performance and maybe 8X read performance.

 

[wonders]

It think its a pool issue.

How large are the small files?
What is the ZFS record size for the dataset? (too large or too small can have negative impacts)

You get more performance from more VDEVs. so arranging your drives as 4x mirrors would give you 4X write performance and maybe 8X read performance.

They are very random between 1KB and 2MB

Record size is 128K on both dataset and pool

Can you please explain how I can do that, I have 8 x 900 GB drives at the moment and I need at least 3TB, how can I configure them for good performance and redundancy?

 

[diogen]

I meant increasing cache in TrueNAS

My understanding is, when left to its own devices, TrueNAS SCALE will grab half the unused RAM for cache.
And it did just that in your case as well... It will release RAM if needed (e.g. apps/VMs)...
I've never used dedicated cache.

EDIT
Here is a reasonable explanation to bandwidth difference when transferring one large file vs. many small ones

Why are network file transfers so slow with multiple small files?

When transferring large (Several GBs) of data over various forms of file transfer, such as: FTP, SFTP, NFS and Samba. They all suffer from the same issue of multiple small files hampering speeds do...

serverfault.com

Some web apps (e.g. Nextcloud) will automatically compress a folder of files into one ZIP file before downloading...

 

[morganL]

They are very random between 1KB and 2MB

Record size is 128K on both dataset and pool

Can you please explain how I can do that, I have 8 x 900 GB drives at the moment and I need at least 3TB, how can I configure them for good performance and redundancy?

First question is always what do you think "good performance" is?

What is your typical workload.... is this normally copying files or are you just setting up? In which case, I would just accept the slowness, it will be OK in normal operation.

For fast file copies, there are tools like Robocopy that provide parallel file copies.... that speeds copies up.

For a faster pool.... configure 4 mirror pairs for >3TB usable. However, if you don't have a spare drive or ability to get one, this may not be the most reliable.

Is the 3TB before or after compression?.... you may be able to use 3 pairs and have 2 spares. There's a useful whitepaper here: https://www.truenas.com/zfs/#TrueNA...nding-zfs-capacity-in-truenas-white-paper/11/

A ZFS primer here: https://www.truenas.com/docs/references/zfsprimer/

 

[wonders]

First question is always what do you think "good performance" is?

What is your typical workload.... is this normally copying files or are you just setting up? In which case, I would just accept the slowness, it will be OK in normal operation.

For fast file copies, there are tools like Robocopy that provide parallel file copies.... that speeds copies up.

For a faster pool.... configure 4 mirror pairs for >3TB usable. However, if you don't have a spare drive or ability to get one, this may not be the most reliable.

Is the 3TB before or after compression?.... you may be able to use 3 pairs and have 2 spares. There's a useful whitepaper here: https://www.truenas.com/zfs/#TrueNA...nding-zfs-capacity-in-truenas-white-paper/11/

A ZFS primer here: https://www.truenas.com/docs/references/zfsprimer/

Thanks very much for the detailed reply,
Will def read the articles!

Well, copying large files only happen if I'm copying to Media dataset for Plex, however I am using a lot of docker containers on daily basis, and a few VMs just to test stuff!
So I guess random IO is important!

Performance to me, I guess is filling the entire 10gbe bandwidth of my network! If even not possible at this stage, I would like to know what I can do to achieve it!

I actually have 12 more exact hard drives on the side, I just don't have caddies or SAS expander to use them!
But I'm going to sort them out and use them all!

With all 16 bays usable what is your suggestion and pattern to get the most performance and reliability out of it?

 

[morganL]

Thanks very much for the detailed reply,
Will def read the articles!

Well, copying large files only happen if I'm copying to Media dataset for Plex, however I am using a lot of docker containers on daily basis, and a few VMs just to test stuff!
So I guess random IO is important!

Performance to me, I guess is filling the entire 10gbe bandwidth of my network! If even not possible at this stage, I would like to know what I can do to achieve it!

I actually have 12 more exact hard drives on the side, I just don't have caddies or SAS expander to use them!
But I'm going to sort them out and use them all!

With all 16 bays usable what is your suggestion and pattern to get the most performance and reliability out of it?

For max performance and reliability... I would do 7 mirror pairs and 2 hot spares. You would see about 10X the performance... on small files.

However, you may need to work out how to get more parallelism going for file copies. A NAS is optimized to support multiple users, not just a single user doing one very single-threaded task.

 

[Kylev]

For max performance and reliability... I would do 7 mirror pairs and 2 hot spares. You would see about 10X the performance... on small files.

However, you may need to work out how to get more parallelism going for file copies. A NAS is optimized to support multiple users, not just a single user doing one very single-threaded task.

I experience the same, I have slow performance speed (10-15 mbps) when copying files from the main pc over to the NAS. I have a pool with 4 2 TB disks (8 TB, 3.4 raw capacity with RAID2Z). Do you think it would be possible to create two vdevs with two disks (one for mirror) to increase performance? Or are there any other best solutions?

 

[morganL]

I experience the same, I have slow performance speed (10-15 mbps) when copying files from the main pc over to the NAS. I have a pool with 4 2 TB disks (8 TB, 3.4 raw capacity with RAID2Z). Do you think it would be possible to create two vdevs with two disks (one for mirror) to increase performance? Or are there any other best solutions?

2 mirrors would be about 2x faster... more for reads.

There are numerous "copy software" options... most are more parallel and faster than the native ones.

Best File Copy Software To Enjoy Fastest Transfer Speeds [2024]

Looking for a fast and reliable way to transfer files on your Windows 11, 10, 8 PC? Check out our list of the best data copy software and find the one that suits your needs.

wethegeek.com

 

[wonders]

For max performance and reliability... I would do 7 mirror pairs and 2 hot spares. You would see about 10X the performance... on small files.

However, you may need to work out how to get more parallelism going for file copies. A NAS is optimized to support multiple users, not just a single user doing one very single-threaded task.

Thanks very much,
How would these look like from the pool and dataset point of view?
Can all of them be part of one pool and combine all the spaces?

 

[Kylev]

2 mirrors would be about 2x faster... more for reads.

This would be something like this:

One main pool, two vdevs inside the pool with two disks on each one, one for data and the other for mirror, correct?