I don't know how many HDDs i should buy

[Kevin Horton]

Thx. and performance wise, is there a problem if i dont follow the 4,6,8 drive rule in the raidZ2 setup?

That rule might possibly make sense if the data is not compressed. But, modern CPUs have more than enough processing power to handle the small overhead from compressed data, and data compression is now the default. The sizing variability due to file compression throws out the assumptions that underpin that "rule", so it is no longer valid. Much more info here.

Bottom line - don't fret over hitting some magic number of disks for a given RAIDZ level. Just be aware that if the number of disks in a single vdev gets too high that performance suffers. Seven disks in RAIDZ2 is perfectly fine.

 

[Zwck]

No, not a problem ;)

I am confused by your smiley in many ways.

Are you saying it's not a problem and you are sexually interested in me, or it is a problem because your smiley is supposed to highlight the sarcastic meaning.

 

[Zwck]

That rule might possibly make sense if the data is not compressed. But, modern CPUs have more than enough processing power to handle the small overhead from compressed data, and data compression is now the default. The sizing variability due to file compression throws out the assumptions that underpin that "rule", so it is no longer valid. Much more info here.

Bottom line - don't fret over hitting some magic number of disks for a given RAIDZ level. Just be aware that if the number of disks in a single vdev gets too high that performance suffers. Seven disks in RAIDZ2 is perfectly fine.

Thank you!

 

[Bidule0hm]

I am confused by your smiley in many ways.

Are you saying it's not a problem and you are sexually interested in me, or it is a problem because your smiley is supposed to highlight the sarcastic meaning.

It's not a problem and the smiley was just to support that. I maybe use smileys too often however...

NB: if you want more info you can read this thread: https://forums.freenas.org/index.php?threads/misaligned-pools-and-lost-space.40288/ especially the "Why the 2^n + p rule is broken" part.

 

[Scharbag]

The suggestion of a 7 drive Z2 vDev to start is a good one. Leaves you the ability to double your capacity (or more with larger disks) without changing hardware. To start, I would hook up 7 drives to the SAS controller to start with, and in the future, connect more drives to the SATA ports and last port on the controller or look into a SAS expander (Intel makes a good one based on LSI). IIRC the 9211-8i based HBA can handle up to 256 SAS/SATA devices, so you have room to grow:)

IMHO, stay away from Z1 production pools with large drives or large drive counts. Also, just to beat the dead horse, any/all RAID levels do NOT replace backups.

Mirrors are a good way to go if you want to add storage in smaller increments but will always use 50% for parity.

Happy FreeNASing!

 

[Zwck]

Thanks all for the kind replies!

 

[joeschmuck]

My two cents: Purchase seven 4TB drives or six 5TB drives in a RAIDZ2, both give you 18TB of storage or ~15TB of true usable storage (20% unused for ZFS performance). There are advantages to both but the differences are minimal for those configurations, being how fast a resilver process takes, it will be a little bit longer using the six 5TB drives. Long resilvering times really do dictate the need for RAIDZ2.

When you hit a storage of 15TB then you have two options...
1) Replace each hard drive with a larger drive.
2) Add another vdev of drives.

Lets talk about RAIDZ1 and RAIDZ2 for a second. If you have data that is expendable then you could use a RAIDZ1 and if you have data that you would rather not loose then RAIDZ2 or RAIDZ3 if it's really important (plus backups). Lets say you want a lot of storage for HD video content, it's not really important and capacity is more important, but you also have some data which is critical to you. Well you could build two pools, one consisting of many drives in a RAIDZ1 and store the expendable data there and then either a mirror(s) or RAIDZ2 for your critical data.

So plan it out and try to purchase what you will need for the first 3 years. After 3 years typically a hard drive warranty is gone so you can plan to replace hard drives at that point in time. If you have seven 4TB drives now then you can upgrade to a larger capacity for each drive and replace them as they fail or if you need an increase in storage, and this can be done with less impact to your wallet. Of course the new capacity will not be realized until all the drives have been upgraded, only them will the capacity jump up.

In my situation I have six 2TB drives in a RAIDZ2 and they are all over 3 years now and running strong. I am just over half of the capacity so I was fortunate to have planned my capacity correctly and I have no intentions to increase the capacity unless 3TB hard drives are just that cheap. I'd like to jump into the SSD arena but I'm not a fool with my money. 2TB SSDs are 6X the cost of a 2TB WD Red drive, or six hard drives (my entire pool) for one SSD. Not smart yet.

In your first posting you asked if you should move from SSD to DOM or M.2 card for the boot device. I would not. If you have something that works, leave it alone. While the 120GB SSD is a bit large, think of it as just giving it a lot of space for failed blocks to be mapped to.

 

[Zwck]

My two cents.. Well you could build two pools, one consisting of many drives in a RAIDZ1 and store the expendable data there and then either a mirror(s) or RAIDZ2 for your critical data ...

Will i run into problems when the vdev1 with raidz1 fails that i might lose my zpool? Also if i run 2 vdevs one z1 and one z2 do i need to have the same amount of Hdds, i know the "circlejock ppt" advises against it.

Or could i just have 7 Hdds in z1 (space optimized) and 4 Hdds in mirror (redundancy optimized)

Best

 

[Kevin Horton]

If one vdev fails, then the whole pool that contains that vdev also fails. There is no hard requirement that the various vdevs have the same number of hard drives. It is considered good practice to have the same number, to optimize the performance, but the system will still work if you don't match them up.

If you have some data that is less critical, and other data that is more critical, it may make sense to have two separate zpools, as @joeschmuck suggested. That way a failure in the RAIDZ1 zpool would not affect the more critical data in the RAIDZ2 zpool.

 

[joeschmuck]

My suggestion above was made in order to meet your suggested needs. I wouldn't build anything other than a RAIDZ2 for my system but there are others out there who are fine running a RAIDZ1 system.

 

[Zwck]

I understand! Thank you again!

 

[joeschmuck]

;)

 

[Zwck]

;)

Joe, I have another question, and you seem quite knowledgeable in comparison to me who has literally no idea what he is doing.

So, the board x11ssh-f sports 6 grey-sata ports, 2 yellow-sata ports, plus the 8 sata ports from my LSI-SAS-Card.

My boot drive is an m.2. 128GB harddrive that boots wicked fast. Lets say I max out my the 14-SATA ports with 14 HDDs can i still connect lets say 2 SSDs in a mirror onto the yellow-Sata ports to host my virtual servers and docker containers or is this not possible since they are typically reserved for the OS

In the end it would leave me with
Boot-Partition-M.2.
vdev0-7 HDDs (sas controler)
vdev1-7HDDs (mainboard+ 1 from sas)
vdev2-2SDDs(mirror) (mainboard the os)

Thank you also for the time!

 

[Stux]

You have 8 SAS ports. And 8 Sata ports. Two of your Sata ports are powered (the yellow ones).

Yes. You can use them as you see fit. ZFS just needs access to a disk.

 

[joeschmuck]

@Stux answered the question, the yellow SATA connectors are able to provide power to a device which recognizes it such as a DOM. What I don't know (I'm tired and just quickly checking email) is if your m.2 slot is taking up one of your SATA ports, sometimes that does happen. If your m.2 card is a SATA card then it might. If it's a PCI-E card then you might be safe.

WARNING! When you add drives to your system you MUST be very aware of what you are doing. Ensure that if you do this that you create a new pool, do not add/extend your current pool, this would be a very bad thing to do.

 

[Zwck]

Thanks to Both Stux and Joe!

Currently, i only have 7 completely empty HDDs attached, and was waiting for your reply about the SSDs. Currently, i am playing around with different boots and getting used to the jails/VMs/docker environments, to get to know the system.
I still have to read the article about "burning in HDDs", and i am not looking forward doing it for 7 HDDs.

So in the end of the day this is what i am aiming for:

Boot-Partition-M.2.
vdev0- 7 HDDs (sas controler)
vdev1- 7 HDDs (mainboard+ 1 from sas)
vdev2- 2 SDDs(mirror) (mainboard the os)

 

[Ericloewe]

@Stux answered the question, the yellow SATA connectors are able to provide power to a device which recognizes it such as a DOM. What I don't know (I'm tired and just quickly checking email) is if your m.2 slot is taking up one of your SATA ports, sometimes that does happen. If your m.2 card is a SATA card then it might. If it's a PCI-E card then you might be safe.

WARNING! When you add drives to your system you MUST be very aware of what you are doing. Ensure that if you do this that you create a new pool, do not add/extend your current pool, this would be a very bad thing to do.

It's a PCIe-only slot.

 

[joeschmuck]

It's a PCIe-only slot.

That's a very good thing.

 

[Ericloewe]

Agh, I have to drop my PCI-e habit. It's supposed to be PCIe. I'd even save key presses by writing it properly...

 

[joeschmuck]

That's okay because PCI-E can also be PCIe or just PCI Express. Although technically PCIe is technically the correct way now to write it.