Raidz1 4x4TB HDD zfs pool write speed is low (lower than 7MB/s)

[Proxy5]

Hi,

Raidz1 4x4TB HDD zfs pool write speed is low (lower than 7MB/s)

I am trying to copy files by accessing the truenas server via ssh.
the source SSD pool is 1 x RAIDZ2 | 4 wide | 931.51 GiB
target HDD pool 1 x RAIDZ1 | 4 wide | 3.64 TiB

the write speed of the hdd pool is about 7MB/s, which I consider quite low.
for the test I copied an .m4v file with a simple cp command, the file size is 3.4 GiB

the target hdd zfs pool is currently 78% full, this may be important.

the hardware environment:
Dell T620
CPU E5-2630 v2 @ 2.60GHz
RAM 192 GB ECC
HBA H310 IT mod

software environment:
TrueNAS-SCALE-22.12.3.3


during copying, the data does not go out to the network, so I think it has no significance.

Any idea where I should go?

 

[LarsR]

What brand and model are your Disks?
If they're seagate baracuda or WD Red drives the possibility that they're SMR drives are very high.
SMR drives are known to cause slow write speed and a bunch of other problems...

 

[Proxy5]

What brand and model are your Disks?
If they're seagate baracuda or WD Red drives the possibility that they're SMR drives are very high.
SMR drives are known to cause slow write speed and a bunch of other problems...

the goal is 4 hdd equally:
WDC_WD4000F9YZ-09N20L1

Is this a normal speed for these discs?

 

[ChrisRJ]

I am trying to copy files by accessing the truenas server via ssh.

Does that mean you are using an SSH shell and perform a local copy? Or do you have an additional machine, from which you copy via SFTP/SCP?

the source SSD pool is 1 x RAIDZ2 | 4 wide | 931.51 GiB
target HDD pool 1 x RAIDZ1 | 4 wide | 3.64 TiB

Linked to the question above: Are those pools on the same machine?

 

[Davvo]

I would argue that 7 MB/s is actually good speed for 4 spinning rust in RAIDZ1, especially if your network speed is 1 Gbps and your pool is almost 80% full. You can also look at the CPU stats in order to check whether you have a bottleneck there, as others have hinted.

 

[Proxy5]

Does that mean you are using an SSH shell and perform a local copy? Or do you have an additional machine, from which you copy via SFTP/SCP?

Linked to the question above: Are those pools on the same machine?

There is no other machine, the TrueNAS Scale server has both zfs pools.

Also, I tried that the HDD and SSD were controlled by the same HBA, and also that the SSDs were on a separate controller.
There is no difference.
The write speed of the SSDs was fine.

 

[Davvo]

Free space fragmentation could also be contributing.

 

[Proxy5]

I would argue that 7 MB/s is actually good speed for 4 spinning rust in RAIDZ1, especially if your network speed is 1 Gbps and your pool is almost 80% full. You can also look at the CPU stats in order to check whether you have a bottleneck there, as others have hinted.

Based on the page below:
https://calomel.org/zfs_raid_speed_capacity.html
I expected higher speed.


Although it is a fact that I currently have riadz1 and not raidz2.

No network data movement!

Copying is done between two zfspools within TrueNAS.

I don't know this bonnie++ command, I need to read up on whether it is possible to test it on a zfs pool full of data.

 

[Davvo]

Based on the page below:
https://calomel.org/zfs_raid_speed_capacity.html
I expected higher speed.

It's giving me a 410 - Access denied error.

Although it is a fact that I currently have riadz1 and not raidz2.
View attachment 70561

The image shows two mirror vdevs, not RAIDZ1 or RAIDZ2 vdevs.

Please read the following resource:

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

As I said, 7 MB/s is honest speed for 4 drives in RAIDZ1 in your conditions (near 80% space used, likely high fragmentation rate, copy from a pool to another done via cp); the dataset's recordsize is also unknown. Looking at the CPU statistics will likely show a bottleneck.

If you want to test the drives speed, use the following resource:

solnet-array-test

Back in the late '90's, I was managing a bunch of large whitebox storage servers. For the largest of these, I had the pleasure of building and deploying a massive storage server, 8 shelves of 9 drives each, Seagate ST173404LW 73GB drives, a...

www.truenas.com

Also, for anyone interested here is the drives datasheet.

https://products.wdc.com/library/SpecSheet/ENG/2879-771475.pdf

 

[Ericloewe]

As I said, 7 MB/s is honest speed for 4 drives in RAIDZ1

Maybe, but it's still pretty crappy. It's a large file, and unless the situation really is terrible, mid-double-digit megabytes per second should not be hard to achieve...

 

[Davvo]

Maybe, but it's still pretty crappy. It's a large file, and unless the situation really is terrible, mid-double-digit megabytes per second should not be hard to achieve...

It sure is, especially when the drives state a hundredfold that speed on their datasheet page.

 

[Proxy5]

It's giving me a 410 - Access denied error.


The image shows two mirror vdevs, not RAIDZ1 or RAIDZ2 vdevs.

Please read the following resource:

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

As I said, 7 MB/s is honest speed for 4 drives in RAIDZ1 in your conditions (near 80% space used, likely high fragmentation rate, copy from a pool to another done via cp); the dataset's recordsize is also unknown. Looking at the CPU statistics will likely show a bottleneck.

If you want to test the drives speed, use the following resource:

solnet-array-test

Back in the late '90's, I was managing a bunch of large whitebox storage servers. For the largest of these, I had the pleasure of building and deploying a massive storage server, 8 shelves of 9 drives each, Seagate ST173404LW 73GB drives, a...

www.truenas.com

Also, for anyone interested here is the drives datasheet.

https://products.wdc.com/library/SpecSheet/ENG/2879-771475.pdf

Sorry, the link was corrupted by copy-pasting for some reason, I fixed it and changed the image.

The processor probably shouldn't be a problem.
There are 2x E5-2630 v2 @ 2.60GHz in the machine.
The load is rarely above 5-10%

When copying HDD -> SSD, the numbers are good.

 

[Davvo]

Sorry, the link was corrupted by copy-pasting for some reason, I fixed it and changed the image.

Noticed; don't have a clue about the bonnie++ command... I recall it maybe was a user script used in ancient times?
Anyway, given the same conditions RAIDZ1 is always faster than RAIDZ2.

The processor probably shouldn't be a problem.
There are 2x E5-2630 v2 @ 2.60GHz in the machine.
The load is rarely above 5-10%

Depending on the comand used, the process might be single-threaded thus resulting in a low (overall) load.

When copying HDD -> SSD, the numbers are good.
View attachment 70565

This tells us that the drives cannot keep up with a sustained write speed, a strong indicator of high fragmentation and a confirmation of the performance drop experienced as the pool fills up. Running zpool list -v | grep frag if I recall the command correctly should vanish any doubt.

Agreed that the CPU has a marginal role on this.

 

[Proxy5]

Noticed; don't have a clue about the bonnie++ command... I recall it maybe was a user script used in ancient times?
Anyway, given the same drives number RAIDZ1 is always faster than RAIDZ2.


Depending on the comand used, the process might be single-threaded thus resulting in a low (overall) load.


This tells us that the drives cannot keep up with a sustained write speed, a strong indicator of high fragmentation and a confirmation of the performance drop experienced as the pool fills up. Running zpool list -v | grep frag if I recall the command correctly should vanish any doubt.

Agreed that the CPU has a marginal role on this.

the measurements shown in the attached pictures were not made with bonnie++, precisely because I am not familiar with it.

zpool iostat -v Backup 2
I used the command to check the speed during copying.

thanks, I'll check the fragmentation.

 

[Davvo]

Just noticed you used a different pool for the HDD -> SSD test; this pool contains much less data, likely allowing for much better streaming read speeds.
If anyone has ideas please chime in, I don't see other factors than: space, fragmentation, cpu, and recordsize.

 

[Proxy5]

Just noticed you used a different pool for the HDD -> SSD test; this pool contains much less data, likely allowing for much better streaming read speeds.
If anyone has ideas please chime in, I don't see other factors than: space, fragmentation, cpu, and recordsize.

There are several zfs pools in the TrueNAS machine, they can be seen in the picture.
It can be seen that the zfs pool named Apps, T620 contains minimal data.
These are SATA SSD devices, called Apps, the recommended mirror format for applications.
And the T620 is a 4x1TB SATA SSD zfs pool, where personal data goes.
The Backup zfs pool is the disks of an older TrueNAS Scale (Microserver G8) installation.
I exported this pool from there, and then imported it in the "new" system.
The data from this will be transferred to a future 4x8TB zfs pool.

I'm at the beginning of the process and I was surprised by the low write speed of the hdd pool, I'm not complaining or complaining (why would I), it's just technically interesting.

I would like to recognize if I have made a mistake so that I can eliminate it in the future.

I am grateful for all advice and suggestions.

 

[Davvo]

I am a bit confused. Everything but pool Backup is SSDs, correct? If so:

• post #13, last third is invalid since I based my analysis on your statement of HDD to SSD copy;
• post #15, first half is invalid since an SSD pool will always have better performance than a HDD one;

I will also await confirmation that the issue is about the Backup pool's slow write speed.

Please help me understand what's going on.

 

[Proxy5]

I am a bit confused. Everything but pool Backup is SSDs, correct? If so:

• post #13, last third is invalid since I based my analysis on your statement of HDD to SSD copy;
• post #15, first half is invalid since an SSD pool will always have better performance than a HDD one;

I will also await confirmation that the issue is about the Backup pool's slow write speed.

Please help me understand what's going on.

I'm sorry if I confused you.
English is not my native language, so there may be misunderstandings.

From the beginning.
A Dell T620 server with 2 CPUs and 192GB memory was given
8x 3.5" slot and 16x 2.5" slot.
There are currently 3 zfs pools (+ the boot pool):
- zfs pool called "Apps" 2x 1TB Samsung SATA SSD.
- zfs pool called "Backup" zfs pool 4x 4TB SATA HDD.
- zfs pool called "T620" 4x 1TB Samsung SATA SSD.

After logging in via SSH, I started copying a 3.4GB file to the server with the cp command
from the "T620" pool to the "Backup" pool, the copying speed was about 7MB6s

So from SSD to HDD!

I then tested other directions because I think this value is very slow.

In terms of SSD - SSD, there are excellent speeds in both writing and reading directions.

There is no writing problem in the HDD - SSD direction either

Writing in the SSD - HDD direction is slow!

We can rule out the network layer because no data left TrueNAS, the CPU because 2 E5-2630 v2 CPUs are available.

The HDD zfs pool, whose name is "Backup" is 78% full, this could be the cause of the problem, if so, then there is no problem.
If that's not the problem, I'd like to find out what it is.

 

[HoneyBadger]

Hey @Proxy5

Even with a pool nearing the famous "80%" mark, 7MB/s is still "too slow" in my opinion for a large (3.4G) file copy, unless you have massive fragmentation issues.

Can you show the output of zpool list for your machine? There is a FRAG column which indicates "average free-space fragmentation."

You can also run iostat -x 5 during the copy process - if one of your four HDDs has outsized values (such as %util) then it's possible that one is bottlenecking the others, whether from a fault on the disk itself or a cable.

The other question I have is regarding your H310 - did you cross-flash this with LSI firmware, or is it still running the Dell OEM firmware? Are all drives cascaded from it through an expander?

 

[Proxy5]

Hey @Proxy5

Even with a pool nearing the famous "80%" mark, 7MB/s is still "too slow" in my opinion for a large (3.4G) file copy, unless you have massive fragmentation issues.

Can you show the output of zpool list for your machine? There is a FRAG column which indicates "average free-space fragmentation."

You can also run iostat -x 5 during the copy process - if one of your four HDDs has outsized values (such as %util) then it's possible that one is bottlenecking the others, whether from a fault on the disk itself or a cable.

The other question I have is regarding your H310 - did you cross-flash this with LSI firmware, or is it still running the Dell OEM firmware? Are all drives cascaded from it through an expander?

Hey HoneyBadger

I will try to answer the questions asked accurately.

Can you show the output of zpool list for your machine? There is a FRAG column which indicates "average free-space fragmentation."

If I understand correctly, the fragmentation is 18%.

You can also run iostat -x 5 during the copy process - if one of your four HDDs has outsized values (such as %util) then it's possible that one is bottlenecking the others, whether from a fault on the disk itself or a cable.

The disks affected are:
sdb
sdc
sde
sdd

I am also attaching two files,
one is SSD to HDD iostat,
the other is an SSD to SSD iostat

The other question I have is regarding your H310 - did you cross-flash this with LSI firmware, or is it still running the Dell OEM firmware? Are all drives cascaded from it through an expander?

The firmware on the H310 controller has been replaced, it is in IT Mode.
The original cable and 4 HDD (zfs pool called Backup) and 4 SSD (zfs pool called T620) are connected to this with the 8xLFF backplane.

The other 2 SSDs (zfs pool called Apps) and the boot-pool are connected on the motherboard controller with an 8087 to 4xSATA cable.

Thank you for following this thread.