Another disks extension question: what is the right choice?

[DeleteMe]

Hi there,

I know this is a recurring question, but I'd like to make sure I don't miss an option if I ever have one other than to backup and redo everything.

I had until now a pool of 4 disks 12Tb in Raiz2.
I decided to add 4 more 12Tb disks in the server. I created another Zpol Raiz2 with the 4 disks.

So I have just 10Tb more available. Not very efficient.
Logical, ok...

If I want to optimize my free space. Do I have any other choice than to start from scratch and create a Raiz2(or3) directly with the 8 disks ?

Thank you!

 

[soleous]

Unfortunately yes, as you know, you need to match the existing vdev when adding multiple. IMO RAIDz2 is currently the standard (for storage) per vDev at a max of 8 wide. Just consider future expansion and cost, for example, is purchasing another 8 disks for expansion unreasonable. Would another pool be a compromise, or would 6 wide be better.

Whatever you decide, I think you need to rebuild that pool, RAID2Zx4, isn't very efficient and again IMO 6 wide is nice for a small deployment. However I am running 5 wide in RAIDz2, it was due to cost and space requirements, but I would have preferred 6 wide.

On the question of other options, you can wait, expansion of vdevs have been in development for quite a while, I wouldn't have gone 5 wide if it wasn't for this. Check it out: https://github.com/openzfs/zfs/pull/8853

 

[sretalla]

I decided to add 4 more 12Tb disks in the server. I created another Zpol Raiz2 with the 4 disks.

So I have just 10Tb more available. Not very efficient.

I don't think you're doing the addition correctly... you should have 20TB+ more storage available (OK, so you shouldn't use all of it, but clearly more than 10)

 

[DeleteMe]

Unfortunately yes, as you know, you need to match the existing vdev when adding multiple. IMO RAIDz2 is currently the standard (for storage) per vDev at a max of 8 wide. Just consider future expansion and cost, for example, is purchasing another 8 disks for expansion unreasonable. Would another pool be a compromise, or would 6 wide be better.

Whatever you decide, I think you need to rebuild that pool, RAID2Zx4, isn't very efficient and again IMO 6 wide is nice for a small deployment. However I am running 5 wide in RAIDz2, it was due to cost and space requirements, but I would have preferred 6 wide.

On the question of other options, you can wait, expansion of vdevs have been in development for quite a while, I wouldn't have gone 5 wide if it wasn't for this. Check it out: https://github.com/openzfs/zfs/pull/8853

Hi
That's clear. I will create à new pool from scratch.

Thank you for the Github dev. Really interesting dev

 

[DeleteMe]

I don't think you're doing the addition correctly... you should have 20TB+ more storage available (OK, so you shouldn't use all of it, but clearly more than 10)

Hi
You are right! My bad
I was "comparing" with another mini server in RAIZ1; Not the same!
So yes, indeed it represents an addition of +20Tb. But it's true that for 4 DD of 12Tb that's not a lot ;-)

 

[Evertb1]

Raidz2 with 4 disks is terrible if you look at the economics. With 6 x 8TB drives in Raidz2 you would have more usable space against lower costs. E.g.: Here in the Netherlands six 8TB Ironwolf disks would cost me around eur. 1289.00 and it would give me around 22.47 TiB Practical Usable storage. Four 12TB Ironwolf disks would cost me around eur. 1316.00 and it would give me around 16.34 TiB Practical usable storage capacaty. I have taken slop space allocation and 20% free space limit in to the calculation.

 

[Constantin]

That's why I went with a 8-drive Z3 - once I accounted for parity (3 disks worth) and leaving 20% open for ZFS productivity to stay high, I was left with 4 drives worth of usable capacity. The addition of a sVDEV added a smidge of capacity (1.6TB) but realistically, it doesn't add to the pool capacity in a meaningful manner - it's main purpose is super fast metadata and small file storage.

 

[soleous]

Regarding the allocation of space, OpenZFS now recommends above 10% free space. I'm not saying 80% is bad because it really depends on the management of storage but thought you might be interested. TrueNAS has a warning at 80%.

The following is an interesting read on the matter:

Workload Tuning — OpenZFS documentation

openzfs.github.io

 

[DeleteMe]

Raidz2 with 4 disks is terrible if you look at the economics. With 6 x 8TB drives in Raidz2 you would have more usable space against lower costs. E.g.: Here in the Netherlands six 8TB Ironwolf disks would cost me around eur. 1289.00 and it would give me around 22.47 TiB Practical Usable storage. Four 12TB Ironwolf disks would cost me around eur. 1316.00 and it would give me around 16.34 TiB Practical usable storage capacaty. I have taken slop space allocation and 20% free space limit in to the calculation.

Yes I know too.
In fact I prepared this machine a short time ago in Raidz2 in anticipation of adding new disks.
Stupidly I didn't even look if in 2021 I could add disks to an existing pool.
I was "sure" that was possible now...
I have been on Freenas for a long time now (on a Dell workstation T390 - in 2003)
And I was reading around for so long that a development was in progress that I thought it was over. Not yet ;)

Anyway, I have to redo everything (again! :) )

Concerning the price, on my side I take advantage of WD Duo price. My last purchase was 2 WD DUO containing each 2 disks of 12To (RED) --> 1000€ (in France)
Not so expensive... And the empty cases are sold in second hand 100€ each. Correct deal.

 

[DeleteMe]

That's why I went with a 8-drive Z3 - once I accounted for parity (3 disks worth) and leaving 20% open for ZFS productivity to stay high, I was left with 4 drives worth of usable capacity. The addition of a sVDEV added a smidge of capacity (1.6TB) but realistically, it doesn't add to the pool capacity in a meaningful manner - it's main purpose is super fast metadata and small file storage.

Ok I see
On my side I will go on 8 drives Z2 !