Help with Expansion Design

[csardoss]

Hi, I am currently running a TrueNAS Core system for CCTV and it is working perfect under the current load. We are currently recording 268 4K cameras to a 50 drive RAIDZ2 storage pool (50x 16TB enterprise drives [10 drives per zdev]). The system will be expanding to accommodate another 400+ cameras and will be adding additional drives as well. Since 99% of the traffic is being written to the drives, with the occasional read from pulling archived footage, I am interested to get feedback on how best to expand the storage.

The Write Transaction happens every 4-5 seconds writing an entire 1GB of data in a sec. I have tried to extend the write transactions to 30 seconds, but it seems the system applys a cap of 1GB RAM Cache before writing the transaction to the drives. I know extending the write transaction time could cause data loss in the event of a system failure, but I am willing on accepting this risk. I hear of another idea to create another layer of Cache by creating a drive using your RAM but I believe my current motherboard does not support this. The goal is to minimize write time to allow for better read speeds. If there is a better solution I am all ears.

Overall, I have 3 ideas on how to expand. The first being to add additional zdevs to the original 50 drive pool. This will allow us to achieve better read and write speeds, but 1 pool means 1 point of failure. Another Idea is to just create another pool with the additional drives and associate the new cameras with the new pool. This sounds the easiest, but will restrain read and write and create a little complexity with iSCSI extents. The last idea I have is to just add another TrueNAS server and send all the new cameras there. This will incur more cost and maintenance, but will alleviate all said issues.

Current Throughput:
1.5Gbps

Estimated Future Throughput
3.7Gbps

Communication Protocol = iSCSI

The current build is:
TrueNAS-12.0-U2.1
2x Intel(R) Xeon(R) Silver 4208
Drives: 50x Segate SAS Exos 16TB
64GB RAM (Will expand to 256GB)
Raid: HBA Controller
Network: Dual 10Gb SFP+ (LAGG)

 

[sretalla]

It sounds to me like you just want to set sync=disabled on that zvol, which should then always write at RAM speeds as long as that's possible.

It's the sync=standard/always that's probably holding you back right now (since iSCSI asks for it by default, I think) due to the limitations of ZIL and/or SLOG if you have it.

This post from @jgreco probably does a much more elegant job of breaking it down for you in terms of the problem you're seeing and what you might do about that:

Some insights into SLOG/ZIL with ZFS on FreeNAS

Yes, and there's good reason for each choice made. Look, VMware's stuff has to actually WORK. ESXi is not a "bad actor" for having made pragmatic choices about being paranoid with VM data. They've made the safest reasonable choices that can be generally implemented across a variety of hardware -...

www.truenas.com

 

[csardoss]

It sounds to me like you just want to set sync=disabled on that zvol, which should then always write at RAM speeds as long as that's possible.

It's the sync=standard/always that's probably holding you back right now (since iSCSI asks for it by default, I think) due to the limitations of ZIL and/or SLOG if you have it.

This post from @jgreco probably does a much more elegant job of breaking it down for you in terms of the problem you're seeing and what you might do about that:

Some insights into SLOG/ZIL with ZFS on FreeNAS

Yes, and there's good reason for each choice made. Look, VMware's stuff has to actually WORK. ESXi is not a "bad actor" for having made pragmatic choices about being paranoid with VM data. They've made the safest reasonable choices that can be generally implemented across a variety of hardware -...

www.truenas.com

Thanks for the info. I will look it over to see how that can help with Write Performance. Truthfully, I was interesting to get feedback on how best to expand for the additional cameras throughput and space. See below

Overall, I have 3 ideas on how to expand. The first being to add additional zdevs to the original 50 drive pool. This will allow us to achieve better read and write speeds, but 1 pool means 1 point of failure. Another Idea is to just create another pool with the additional drives and associate the new cameras with the new pool. This sounds the easiest, but will restrain read and write and create a little complexity with iSCSI extents. The last idea I have is to just add another TrueNAS server and send all the new cameras there. This will incur more cost and maintenance, but will alleviate all said issues.

 

[sretalla]

The first being to add additional zdevs to the original 50 drive pool

Actually you may reduce your IOPS by doing that (unless you clear the pool and repopulate from backup, which will evenly distribute the data across all VDEVs) since the new VDEV will get most of the writes as it's the most empty VDEV.

1 pool means 1 point of failure

In the sense of pool metadata going bad, you're right. Although there are a lot of protections to avoid that happening and if it really concerns you, maybe a metadata special VDEV of 4 mirrored SSDs would be an option to be super-safe... and fast for metadata... (and clear/restore the pool to put your metadata on it).

Another Idea is to just create another pool with the additional drives and associate the new cameras with the new pool

That would work fine, with the limitation you mention.

add another TrueNAS server and send all the new cameras there

You also correctly identified the downsides of this option, but it will work fine as you suggested.

 

[csardoss]

Actually you may reduce your IOPS by doing that (unless you clear the pool and repopulate from backup, which will evenly distribute the data across all VDEVs) since the new VDEV will get most of the writes as it's the most empty VDEV.



In the sense of pool metadata going bad, you're right. Although there are a lot of protections to avoid that happening and if it really concerns you, maybe a metadata special VDEV of 4 mirrored SSDs would be an option to be super-safe... and fast for metadata... (and clear/restore the pool to put your metadata on it).


That would work fine, with the limitation you mention.


You also correctly identified the downsides of this option, but it will work fine as you suggested.

Thank you for the Response! I like the idea of creating another pool. As for the SSD's, I believe they will get destroyed from the constant writing.

 

[sretalla]

As for the SSD's, I believe they will get destroyed from the constant writing.

In the case of metadata VDEV, the amount written would be very low compared to the actual data written to the HDDs in the pool. I wouldn't be worried about that aspect. Much more concerning would be if you used SSDs for L2ARC (which would be a very bad choice for your use case as I see it).