2 * 3 or 1 * 6 SSD Vdevs

[arcadeperfect]

Hi

6 * 2TB sata SSD's. 10Gbe. Platform is a Supermicro Xeon D-1521 with 64 GB ram (could expand).

Fully backed up nightly, used for post production video work, which is read heavy. I think RaidZ2 is a decent level of protection.

Question is, 2 * 3 drive vdevs, or 1 * 6?

2 * 3 would make it easier to expand obviously. But I think at a cost of read performance? Will I notice that performance cost with 10Gbe, or would 2 * 3 SATA ssd's saturate it anyway?

 

[artlessknave]

you can't make a raidz2 with 3 drives, so that doesnt make sense. if that is not what you meant, than the way you constructed your post is confusing to me.
2x 3-way raidz1 would be less safe than raidz2 but take up the same amount of parity disk space, and thus be irelevant here.

if you have a good backup, I would probably seriously consider raidz1 for all 6 disks. raidz2 will likely be overkill.
1. the recomendations against raidz1 are actually for over 2TB HDDs. you are below that number AND using SSDs.
2. the recomendations against raidz1 are typically due to the resilver speeds on HDDs. SSDs will almost certainly resilver orders of magnitude faster. think a few hours instead of days.
3. technically the best read speed will mirrors, but you will likely not notice speed performance differences b/w raidz or mirror vdevs types on SSDs. you *might* start to notice on 12-24+ drive pools, but if you were doing tha would probably be using nvme anyway, and that would definitely make it irelevant. well. unless they are GenericCheapBigWePromise SSDs from aliesxpress or something.

 

[arcadeperfect]

Whoops sorry I had a senior moment there.

My question was all about RaidZ in both cases. Not RaidZ2.

I'm now seeing that I get ~ 1TB more usable space with one 6 drive, raidZ1 vdev though. I will probably go with that.

 

[artlessknave]

RaidZ in both cases.

RaidZ represents the type of technology that allows all raidz1/2/3 levels. there is no "raidz" mode. you clearly mean raidz1.
it's like how you say "I built a RAID array" to indicate you used RAID technology, but you say
I used RAID0/1/5/10/69/666" when you mean a specific RAID level. "I used RAID mode" has no meaning.

raidZ1 vdev though.

make sure you verify both that your backups exist and that you can use them to restore. many people just set them, forget them, and only find out they are useless when they lost everything.
also, seriously consider having a cold spare on hand.

 

[NickF]

OKay let's circle back around here...
There are some key words I picked up on. Let me know if I am off base here.

used for post production video work,

I think RaidZ2 is a decent level of protection.

These two things should never meet. Video production work from more than one or two workstations would likely be impossible with a single vdev RAIDZ2. I would highlt recommend usng a zpool topology focused on IOPS rather than capacity or really even raw speed. Use mirrors. 2 way mirrors 3 wide will kick ass.

Mirror vdevs​

Mirror vdevs have many good points; the main drawback is that compared to other vdev types, you need more disks to hold the same amount of data. As mirrors keep identical copies of the data in several disks, they can provide the best IOPS, the number of read and/or write operations that can be performed per second. ZFS distributes the writes amongst the top level vdevs, so the more vdevs in the pool, the more IOPS that are available. Now it’s even possible to shrink a pool comprised of several mirrors by removing one of them, so the flexibility provided by mirrors is an important feature to consider when deciding which type of vdev to choose.

Choosing the Right ZFS Pool Layout - Klara Systems

Learn how to build and customize FreeBSD package sets with Poudriere. Optimize packages for specific devices or manage a FreeBSD cluster in this article.

klarasystems.com

 

[artlessknave]

Video production work from more than one or two workstations would likely be impossible with a single vdev RAIDZ2.

these are SSDs....pretty sure the 10GBE network will be the bottleneck, not the pool.

 

[NickF]

these are SSDs....pretty sure the 10GBE network will be the bottleneck, not the pool.

Not worried about the throughput here. You're thinking about the wrong thing. IOPS are what matters here, not sequential speed benchmarks with CrystalDiskMark :P

But yes, from the networking perspective more than ~2 clients probably won't be possible above 1080p...whereas the backing pool could probably do more like 4 or 5.

 

[artlessknave]

IOPS are what matters here

absolutely. the thing is SSD's IOPS are like thousands each, while HDD's are like dozens for the whole pool (exxageration for effect).
I'm reasonably sure that, unless the CPU is a potato, this would work fine.

 

[NickF]

absolutely. the thing is SSD's IOPS are like thousands each, while HDD's are like dozens for the whole pool (exxageration for effect).
I'm reasonably sure that, unless the CPU is a potato, this would work fine.

I don't disagree. But here's my counter argument. Disk workloads are dynamic in general. There's a mix of random I/O and sequential work. For the sake of visualization here, those pesky random I/Os cause the read head to move. Think of each track on a hard drive as if it were the face of a clock. Now imagine that the random I/O keeps stopping the second hand from moving. Sometimes, it's even making it go backwards. Other times, it's referencing another clock, 120 clock lengths away, and then returning.

Then we have a basic L2 network. Computer sends frame. Switch receives frame. Switch forwards frame. Other computer recieves frame.
Switches be swithin and frames be frames. It's much more predictable since there are many less variables than there are in SAN/NAS. So I'd rather the network stack be my bottleneck, so I can ensure steady-state predictable storage speeds that are less volatile when compared to the workload.

TL/DR Bottlebecks exist. You might as well plan for them at layer you can predict and control

 

[artlessknave]

track on a hard drive as if it were the face of a clock.

my counter-counter argument is that this is 100% about an SSD pool so I don't get how this is relevant at all. there are no tracks, no HDD iops limits, that cannot be the bottleneck.
I would never recomend raidz1 for HDDs. period. if we were talking about HDDs, then yes, that would make a difference. but we...aren't.

 

[NickF]

my counter-counter argument is that this is 100% about an SSD pool so I don't get how this is relevant at all. there are no tracks, no HDD iops limits, that cannot be the bottleneck.

While you are mostly correct, the same underlying logic to how SCSI works, SCSI and even SATA SSDs are "talking" as if the are hard drives. This is why NVME was created in the first place.

Additionally, the logic is relevent here. The physical distance and time it takes the SSD controller to "move it's read head" from one cell to another is still a factor. It's just an order of magnitude faster. The real issue here is that doesn't change that even SSDs suck at 4k random I/O when relatively compared to sequential. My point is that relative performance is what matters, not absolute..

 

[artlessknave]

The physical distance and time it takes the SSD controller to "move it's read head"

what? it's an SSD. there is nothing to move. there IS no physical distance, only access time, which, unless the SSDs are CheapAndWorkingWePrimise SSDs from aliexpress, are, again, orders of massive magnitude higher than HDDs. like picoseconds vs nanoseconds.
the seek time will be functionally 0. I doubt a few users would notice. yes, a hundred users would notice, but I'm pretty sure that is also a red herring and not relevant to this thread.

IIRC, SSD's suck at randomIO when compared to non random IO speeds. they are still usually pretty great, and often limited by the interface. these are SATA disks, they will likely already be bottlenecked by the interface itself.
THAT's why NVME is faster, because its NOT SATA, which was designed to abstract slow disks on slow south bridges with the other slow IO.

 

[NickF]

what? it's an SSD. there is nothing to move. there IS no physical distance, only access time,

Right. That's my point here and I am sorry if my analogy has muddied the waters. Regardless of whether there is a physical spinning platter and a head head or a chip accessing cells in NAND, time is not on our side.

If we look at this monster And we look at random 4k:

Solidigm D5-P5430 15.36TB SSD PCIe Gen4 NVMe SSD Review

In our Solidigm D5-P5430 review, we see how this 15.36TB PCIe Gen4 NVMe SSD performs and how it is optimized for hard drive displacement

www.servethehome.com

Even it gets destroyed by random I/O. Mirrors will help smooth these edges out in a pool when compared to RAIDZ. Again key phrase here is in a pool. You will hit a bottleneck on throughput eventually. The top numbers can only get so big before physics hits a wall. But the bottom three graphs will slowly equalize towards the center of relative performance the more vdevs you have.