Do i understand disk upgrade correct

[aadje93]

Hi guys,

my current setup is 24x 4TB in "raid 10" zfs (striped mirror vdevs)

Now i'm building a NVR setup so space is getting a bit on the premium as it wil be >10x 8MP camera's. The disk speed is amazing for VM work over 10G, but 50% of disks is parity now.

So i have 3 questions,

1. If i understand correctly, 1 vdev fully upgraded give larger pool size, so if i upgrade 1 vdev mirror, i get the increased disk space to the pool, but until all 24 are upgraded the performance is pretty worse when the 4TB's start to fill, while the big new disks are like 20% full (thinking about investing in 12-18TB straight away with upgrade).

2. Converting it to Z2 (4x 6disk Z2) will give some nice bonus, from ~48TB raw now (actually more like 38TB in windows) it should become 64TB raw. So 25% improvement for a slight data security drop, and pretty big performance hit.

3. Or go all overboard and upgrade the 24 disks straight in a couple days due to all the resilvering 24x, and then have some used spares for the backup server :P (yes, also a 24x 4TB copy with just 64G ramm and its configured in 4x 6 disk Z2 due to me thinking the change that those 2 machines f*ck up at the exact same moment is realy, realy, realy rare due to the slightest disruption on SMART -> replace disk. This would give me 24 disks laying around for possible alot of years, and changing the main server to 6TB disks will cost some pretty big $$$$



Now lastly, i was thinking, if i upgrade 1-2 vdevs with 18T purple drives instead of the current red. Then i get a major boost in storage space, but the NVR system will take alot of the space, i would like to keep most cams for 30 days at least, only some minor interior cams are like 14 days max as there is little to see. Will the performance hit be huge if those 4TB's get filled, or is ZFS smart enough to fill drives like unraid does where its "least full % first" kind of thing?

As i don't yet fully trust blue iris' motion detection with deep-stack, i haven't done conditional storage, for now its 24/7 record to disk. If deepstack gets 99,99% sure storage of moving persons with some false positives then i may reduce the disk storage space by alot, as at night not alot happens here.

 

[NugentS]

Do you really need that level of performance of 12 mirrored pairs - which seems unlikely
2. Looks good to me as you have a backup system. Convert to Z2 or Z3 with multiple vdevs. Gets you the IOPS of the number of vdevs - so a lot better than a single vdev (which would be silly). In your stated idea - thats 4 vdevs of rust which will do quite well.

Incidentally what are you doing that needs a SLOG?

 

[aadje93]

The current machine is not only NVR storage, its also a VM host, and off course NAS for multiple users ;)

Yeah, maybe i should ditch the SLOG, get some 1T SSD's for the VM's and go 4x Z2 or something.... But that only gives like 25% more storage compared to now, and loses A LOT of iops/throughput for the spinning rust.

What do you recommend for a 24 disk system that also hosts VM's (now running pretty good on the 12 vdev mirrors with slog + l2arc)

I'm reading alot about 11 disk Z3 with some hot spare, but i rather have a cold spare in a box next to the servers as they are in my home and 24 active disks. Hot-spare is not realy needed for time saving or so. Rather have the extra storage/parity.

In the past i had 4x 6 disk Z2, but with current high capacity drives that brings me a bit in the 4T z1 setup feeling of baddnes. In the future i possibly upgrade the disks to 14T models when they start to fail (first few have gotten a 4TB replacement). With the 12 mirrors i have now my plan was to upgrade the full mirror when 1 of the 2 gives errors, so slowly they upgrade and the pool grows with time. When i wipe and go to a different Z setup, then i get mixed drives in vdev, and it takes alot of time for the storage space increase to become active.

What is your suggestion. I would realy like to keep the truenas machine as VM host, just plain and simple working good with some nice stack of memory. A couple SSD's would be possible for the VM's, should i make a mirror of SSD for the VM hosting and go 2x 12 Z3? that gives 18 data disks, so roughly 64T effective space after parity + VM space