Will vdev rebalancing speed up pool performance ?

[Félix Abt]

Hi all,
My pool performance seems to be rather slow, the read speed can't go more than 250 Mo/s, but i think i should be able to have 350, maybe 400Mo/s out ouf my pool.
Here is my config :

Show : My config

Dual Xeon E5-2690 8C x 2,90ghz
256 GB RAM ECC
one MD1200 attached with a LSI SAS 9207-8E HBA
2 x raidz2 vdev (each vdev as 6 Disk ironwolf pro 7,2K 8TB)

Based on the fact that i can expect the speed of one drive out of one vdev, and that i've seen all my disks going at 190Mo/s during scrub, i should be able to go up to 350Mo/s at least.

I think i have the answer to my question, and i'm posting here hoping that someone will confirm.
Since i've added the second vdev after copiying all my data to the first vdev, going as far as 85% usage, i think most of the data are only on the first vdev.

About the new data i've added after the pool expansion, i've read that ZFS will balance the data based on the vdev usage, so i assume they will go mostly on the second vdev, and therefore, the read/write throughput will be capped by the speed of one vdev only.

I've tried to read a lot about ZFS ecosystem, but i'm still a noob about the subject, so i wanted to verify by beliefs.

 

[NugentS]

you are essentially correct AIUI.
If you have multiple datasets then you could try zfs send | zfs recv to a new dataset, the destroy the old dataset and rename the new one. This, done several times would gradually balance the pool.

There is a pool balance script - not sure if it does anythingh differently to what I say above - I haven't looked

 

[Félix Abt]

OK thanks a lot ! i currently use this one that i found on this forum :

GitHub - markusressel/zfs-inplace-rebalancing: Simple bash script to rebalance pool data between all mirrors when adding vdevs to a pool.

Simple bash script to rebalance pool data between all mirrors when adding vdevs to a pool. - markusressel/zfs-inplace-rebalancing

github.com

which is very simple and straight-foward to use.
So my pool is slowly rebalancing itself. I need to wait for this process to finish and i'll keep you posted about how the things are going.

 

[Félix Abt]

OK now my vdev are rebalanced , but i have the exact same problem than before. I think there is another bottleneck : i work in video post-production ,and this server primary task is to give acess to massive raw footage to the editors or colorists.

When i'm playing some raw footage that needs more than 200MB/S throughput (like R3D or DPX files ) the playback is choppy ,and i see the bandwith capped at 200MBs. (seen with iostat -v )

To verify that's not a client-side problem, i've copied these same R3D and DPX footage on the computer local nvme drive , and the playback is perfectly smooth @300MB/s.

The thing that made me through that there is another bottleneck than disk speed or vdev topology ,is that when i copy files from the pool to my local Nvme , the throughput is blazing fast , around 550 MB/s , and i can see the disks read bandwith go up to 200MBs per disk in iostat.
It's roughly the same bandwith for writes.
Also , i've done some test with primarycache=metadata to be sure to look at disk speed and not RAM .

So why my playback is so slow compared to the copy speed , is a mystery. I've searched everywhere for thread about similar issue on playback with no luck.

 

[sretalla]

why my playback is so slow compared to the copy speed , is a mystery

Not if the workload is IOPS intensive...

raidz2 vdev

With RAIDZ2 x 2 you have the IOPS of 2 HDDs, which is orders of magnitude less than an NVME drive (something like 200-500 compared to 50000+).

Also:
If you force ZFS to call up tiny chunks at a time (like 128K if you did nothing to change the recordsize on your dataset), your reads will be slower.

I guess your videos are large, so a recordsize of 1MB would be appropriate.

Changing a dataset property doesn't apply that to all existing files, so consider that in your testing.

Also:
Depending on how you're connecting (SMB from a Mac?) you may be requesting sync writes, which will be much slower and heavy on IOPS demand. (either consider other options or force sync=disabled on the dataset)

 

[Félix Abt]

I guess your videos are large, so a recordsize of 1MB would be appropriate.

Yes that's exactly the recordsize i choosed for my pool.

Changing a dataset property doesn't apply that to all existing files, so consider that in your testing.

yes i'm testing only on newly created media.

I just realized someting interesting :

When i'm playing these .R3D 8K camera files, which are using 283MB/s (2 min file = 34GB), zpool iostat -vy 5 1 show me that the pool send approx. 200MB/S. So the playback is not smooth it does stop every 2 sec, that seems normal given the numbers.
What seems really curious is when i try to play 2 other files simultaneously (one .DPX @ 200MB/s + one DNXHR @ 100MB/s) for a total of 300MB/s, so in theory, the playback should be even more choppy... But it's actually realtime, smooth and clean. zpool iostat -vy 5 1 show me even more than 400MB/s, when playing other files than R3D.
(i know that R3D files need a decoding process that is very GPU and CPU intensive, and can sometime impact playback, but i made shure this was not the problem by playing these same files on my local Nvme drive, on the same machine, and it was totally smooth)

Also:
Depending on how you're connecting (SMB from a Mac?) you may be requesting sync writes, which will be much slower and heavy on IOPS demand. (either consider other options or force sync=disabled on the dataset)

I'm using SMB from Windows 10 clients. is sync=disabled safe to use ? This storage will hold some critical data, so i want to avoid taking risks.

 

[NugentS]

Safe - not 100% - but its safe enough to test with - to see if thats the / an issue

 

[Félix Abt]

OK i'll try that asap. But if the playback is smooth with sync=disabled what conclusion should i draw ? the amount of iops is too high ? weird because i have enough iops to play heavier files...

 

[danb35]

But if the playback is smooth with sync=disabled what conclusion should i draw

None whatsoever, presuming (as I'd expect) playback is a read-only process. sync=disabled only affects writes.

 

[Félix Abt]

Ok i've tried Sync=disabled unfortunately i did not notice a difference in throughput in writes or reads, either playing those .R3D footage or just copying files back and foward. I've added a 3rd RaidZ2 (6x8TB) to my pool and i observe the same behaviour while playing R3D. I'm kindahopeless now. While playing other files like multiple DPX flux, i can achieve 350 / 400 Mo/s easily...

 

[JoeAtWork]

Hi all,
My pool performance seems to be rather slow, the read speed can't go more than 250 Mo/s, but i think i should be able to have 350, maybe 400Mo/s out ouf my pool.
Here is my config :

Show : My config

Dual Xeon E5-2690 8C x 2,90ghz
256 GB RAM ECC
one MD1200 attached with a LSI SAS 9207-8E HBA
2 x raidz2 vdev (each vdev as 6 Disk ironwolf pro 7,2K 8TB)

Based on the fact that i can expect the speed of one drive out of one vdev, and that i've seen all my disks going at 190Mo/s during scrub, i should be able to go up to 350Mo/s at least.

I think i have the answer to my question, and i'm posting here hoping that someone will confirm.
Since i've added the second vdev after copiying all my data to the first vdev, going as far as 85% usage, i think most of the data are only on the first vdev.

About the new data i've added after the pool expansion, i've read that ZFS will balance the data based on the vdev usage, so i assume they will go mostly on the second vdev, and therefore, the read/write throughput will be capped by the speed of one vdev only.

I've tried to read a lot about ZFS ecosystem, but i'm still a noob about the subject, so i wanted to verify by beliefs.

one vdev you should be safe, once you have more than one and you scrub you will see times the whole pool is reduced to the performance of one vdev(a mirror or raidz vdev). I have observed this where a pool was scrubbing at 3 or 5 gigabytes per second and then for 15 minutes the scrub slows down to 500 megabytes per second becauase it is reading blocks from just one vdev.

The only way to fix this is to re-balance the blocks on the vdev. I believe there is no way to re-balance at this time. I have tried to use identical settings and fill my pool to 79% and then scrub it. It might be the worst case is filling teh disk with DD random data and moving vmware vmdk files to a TrueNAS iSCSI/NFS/Fiber Channel store. :-(

Here is my sad results(15 minutes of IO only on the first vdev):

 

[Morris]

If your CPU is pegging during the file transfers, try raid Z1 x 2 as less parity calculation is needed

 

[Davvo]

If your CPU is pegging during the file transfers, try raid Z1 x 2 as less parity calculation is needed

That's a bad idea safety-wise.

 

[Morris]

That's a bad idea safety-wise.

This depends on your uptime requirements and ability to hit restore time object from a proper backup system and operation. Your oversimplification of busyness contingency planning can lead to a waste of recourses. Furthermore it might prevent the OP from solving his performance issue.