Need a Plan Of Attack

[Binary Buddha]

A combination of me using 8GB RAM for 32 TB of storage (replication and light use) and this: https://www.reddit.com/r/DataHoarder/comments/47qedb/how_much_ram_for_100tb_zfs_pool/

If the OP is just using this as a gigantic datastore for Plex supporting 1-2 users, RAM usage doesn't need to be that high (in my opinion).

The most useful post thus far. Yes, it's home use on GIGe. With pretty much just me uploading stuff, maybe two Plex streams at most, and Transmission running throttled to 400kb/s. So, performance in regard to speed is maxed at the GIGe bottleneck.


In a ZFS pool, would it be better to have more drives in a VDEV or more VDEVs? For example, 10 drives in one RAIDZ3 VDEV or two five drive RAIDZ3 VDEVs?

 

[depasseg]

The more vdevs, the better performance (IOPS-wise). The more disks in a RAIDZ(x) pool, the longer it takes to resilver in the case of a failure (which increases the likelihood of another drive failure. And generally, vdevs don't go larger than 11-ish drives.

For your environment, I'd suggest 10 drive RAID Z3. However, 2 vdevs of 5 drive RAIDZ2 (Z3 is overkill for 5 drives) is something to consider if you think you will ever need to upgrade your pool. You can increase storage space by replacing drives in a vdev one by one until they are all replaced. Obviously a 5 drive vdev will be easier to afford than a 10 drive vdev.

 

[Binary Buddha]

The more vdevs, the better performance (IOPS-wise). The more disks in a RAIDZ(x) pool, the longer it takes to resilver in the case of a failure (which increases the likelihood of another drive failure. And generally, vdevs don't go larger than 11-ish drives.

For your environment, I'd suggest 10 drive RAID Z3. However, 2 vdevs of 5 drive RAIDZ2 (Z3 is overkill for 5 drives) is something to consider if you think you will ever need to upgrade your pool. You can increase storage space by replacing drives in a vdev one by one until they are all replaced. Obviously a 5 drive vdev will be easier to afford than a 10 drive vdev.

I thought five drives were minimum for Z3. I would have said Z2 or Z1, but I was expecting the "blah blah is bad! Use Z3!" people to show up. I was originally thinking of three Z1 VDEVs with five drives each. Which I believe allows me two disk failures per VDEV, while still getting the most space out of the drives. Buying 10+ drives in one swoop is ouch. If I could add drives to a Z3 VDEV later then I'd probably shove everything in the 10 drive Z3. Since I'm sitting behind a 1GIGe port, ZFS speed is not really a concern. So, the next target is redundancy I guess. Two failures per VDEV is more than adequate for that.

At the moment, I have a lot of open slots and ports to create a second VDEV. So, I don't have to resilver/replace drives one at a time. (Hopefully, the data on the first VDEV will stripe over to the new VDEV when I make it.)

Sound about right?

 

[AVB]

The port, on a local basis, is a cheaper fix than buying new drives. If you plan to upgrade storage then you do need to replace drives one at a time. If you add another vdev with larger drives in the same pool you just can't make the smaller unneeded vdev go away without rebuilding the whole pool. So unless you are making a completely new pool with the larger drives and moving everything over, then you can destroy the old pool. Of course, that has the same effect as just upgrading drives one at a time. Or you can add a vdev of the same type i.e. 5 drive raidZ2 to your existing pool to increase storage too. You aren't clear if you want to use up more slots or increase drive size or both.

Zi will allow only one drive failure per vdev, Z2 2 drives and Z3 3 drives. For a 5 drive pool Z2 is probably the most preferred way to go.

 

[Binary Buddha]

The port, on a local basis, is a cheaper fix than buying new drives. If you plan to upgrade storage then you do need to replace drives one at a time. If you add another vdev with larger drives in the same pool you just can't make the smaller unneeded vdev go away without rebuilding the whole pool. So unless you are making a completely new pool with the larger drives and moving everything over, then you can destroy the old pool. Of course, that has the same effect as just upgrading drives one at a time. Or you can add a vdev of the same type i.e. 5 drive raidZ2 to your existing pool to increase storage too. You aren't clear if you want to use up more slots or increase drive size or both.

Zi will allow only one drive failure per vdev, Z2 2 drives and Z3 3 drives. For a 5 drive pool Z2 is probably the most preferred way to go.

Ironically, here in Hawaii it's harder to find PCI NICs than it is to find HDs.

I'm not seeing the correlation to the NIC and the drives as a problem. GIGe is sufficient for my home setup as far as speed is concerned. 10GIGe is Enterprise level overkill and they don't make consumer level 10GIGe switches and routers that would match the quality of my NETGEAR Nighthawk X6 R8000 .

As for the smaller VDEV, I can't just offline a drive at a time as I add larger VDEVs? I "can" make a new zpool with larger drives. Ideally not, because of the moving stuff over like you said. Doing the vdev of the same type is sort of what I was going for... except it's a six drive Z1 in my system right now. I can't just add a new 5 drive Z2 vdev to my existing zpool with autoexpand on? My intent is to do both using up more slots and increasing drive space. But, done knowing the limitations of 24GB RAM and 1GIGe NIC.

 

[AVB]

No, you can't offline a drive at a time as you add Vdevs. Autoexpand doesn't have anything to do with pool size directly it has to do with Vdev size. Auto expand is when you replace the current drives in a Vdev with a larger size. Once all the drives are replaced the Vdev will grow to the new size if you have autoexpand on. I guess if you only have one Vdev it will grow the pool size directly but that is not the scenario you have been talking about. When you add Vdevs to a current pool they should be the same as the current Vdev already in existence. A six drive RaidZ1 Vdev would be added to the current six drive RaidZ1 pool you already have. I'm not saying that is an absolute but is is the recommended way to do it. It seems that you need to figure out what Vdevs you want based on number of disks you currently have or will have shortly and redundancy you want and if necessary reload your data into the new structure. I had to do it when I went from a 10 drive RadZ2 Vdev to two 8 drive RaidZ2 vdevs but now I'm set for any future growth as I'll just replace drives in one of the Vdevs with a larger size ad infinitum or until the hardware won't recognize the larger drives.

The goal should not be to fill up slots but to have good quality drives that will fit your need for X amount of time - the X being up to you. If you find you need more space then you can add Vdevs if you have the slots to do so, or increase the drive size of the current Vdev drives or both. The upgrade drives approach is what I use because 1) it can be done over time, 2) the replaced drives can be use to make another vdev if even more space is needed or 3) the replaced drives can be added to the backup so you can keep all that data you are putting on the new drives.

I mentioned the Nic because you seem to make it a point to regularly mention the limitations of the Nic you currently have.

 

[Binary Buddha]

To conclude the discussion... I'm setup with a 5 drive Z2 pool at 8TiB a drive.

So, now I'm contemplating on whether or not I should add 5 of my old 2TiB drives as another vdev with only about 24GB of maxed out memory. And whether or not to start saving up for a full upgrade of the motherboard. https://amzn.com/w/3I4YTIDNHRB66

 

[AVB]

To conclude the discussion... I'm setup with a 5 drive Z2 pool at 8TiB a drive.

So, now I'm contemplating on whether or not I should add 5 of my old 2TiB drives as another vdev with only about 24GB of maxed out memory. And whether or not to start saving up for a full upgrade of the motherboard. https://amzn.com/w/3I4YTIDNHRB66

f you need the additional 6TB of space and can put up with a slower response until you upgrade the MB then go for it. The additional Vdev is basically free since you already have the drives and you are planning to replace the MB anyway at some point. You can always auto expand the 2TB vdev over time as you need more space.

 

[AVB]

I picked up a server board here