Hardware Verification - New TrueNAS Build for Photography Work

[john.o]

As a TrueNAS user for a few months, I have been very satisfied with its performance. The current configuration of my server is as follows:

- TrueNAS Core
- ASUS X99 DELUXE II
- Intel 6950X CPU 10c/20t
- 128GB DDR4 RAM
- Intel X550 10G NIC
- 8x 16TB Ironwolf Pro Drives (Data VDEV 3x MIRROR-6Wide+1 Hotspare +1 coldspare)
- 2x 250GB Ironwolf 125 SATA SSD (mirrored metadata vdev)
- 1x Intel 750 U.2 400GB NVME (L2ARC)
- 1x Intel 600P 512GB NVME (boot drive)
- Fractal Designs Meshify 2 Case

I utilize this server for my photography work and currently have a 32TB iSCSI share mapped to my workstation that contains all my RAW files, video files, documents, and photos. However, as the hardware is more geared towards consumer/gaming use, I am planning to upgrade to server-grade components for added peace of mind.

My initial build list is as follows, and I already have a 3-2-1 backup of all my data:

- AsRock Rack SPC621D8-2T ATX Server Motherboard LGA 4189
- Intel Xeon Silver 4310 Ice Lake 2.1 GHz CPU 12C/24T
- NEMIX RAM 128GB 8x16GB DDR4-2666 PC4-21300 2Rx8 ECC Registered Memory
- Noctua NH-U12S DX-4189 Heatsink/FAN

My plan is to migrate the pool to the new hardware with a fresh install of TrueNAS CORE while using the same chassis. I also plan on utilizing the SATA ports on the new motherboard for the hard drives. However, I will remove the L2ARC drive from the pool prior to export as the new motherboard does not have a U.2 port.

While this setup may be considered overkill for my use case, I welcome any suggestions or feedback.

 

[Etorix]

I've not understood if you pool is 2 * (3-way mirror) or 3 * (2-way mirror). Either should be well suited to iSCSI anyway.
Your build list looks all fine—though you could probably save some money by going for refurbished parts of the previous LGA3647 generation.
You can use the U.2 drive in a PCIe slot.

But overkill indeed. May I ask why you went for iSCSI, and mirrors, rather than for a regular SMB (or NFS) share, which could have done well with space-efficient raidz2 and lesser hardware?

 

[john.o]

I've not understood if you pool is 2 * (3-way mirror) or 3 * (2-way mirror). Either should be well suited to iSCSI anyway.
Your build list looks all fine—though you could probably save some money by going for refurbished parts of the previous LGA3647 generation.
You can use the U.2 drive in a PCIe slot.

But overkill indeed. May I ask why you went for iSCSI, and mirrors, rather than for a regular SMB (or NFS) share, which could have done well with space-efficient raidz2 and lesser hardware?

Etorix - Thank you for your response! I would like to clarify my data VDEV setup, following my previous post. Specifically, I am currently using 3 * (2-way mirrors) configuration, which I opted for to ensure that I only need to purchase two drives at a time, in case I plan to expand in the future. Additionally, I have read that mirrored drives can resilver quicker, making maintenance easier.

I am aware that this configuration results in a loss of half of the storage space. However, given my current data storage needs (approximately 10TB), it suffices. As for my preference towards iSCSI, I like the idea of having a drive attached to my workstation, which would contain all my files. Nevertheless, I am open to exploring other alternatives, as I have experienced issues with some software, specifically Lightroom, not functioning well with SMB shares.

Currently, I am using my old, repurposed workstation/gaming computer, which I acknowledge is not optimal. Thus, I am looking to shift to server-grade components, and eventually experiment with VMs/docker once SCALE matures.

Regarding the use of SATA ports on the new motherboard, I am considering this option. However, I am uncertain if it would be the most efficient choice or if an HBA would be preferable. I would appreciate any feedback or suggestions on this matter.

 

[Etorix]

Specifically, I am currently using 3 * (2-way mirrors) configuration, which I opted for to ensure that I only need to purchase two drives at a time, in case I plan to expand in the future. Additionally, I have read that mirrored drives can resilver quicker, making maintenance easier.

All correct. But with 16 TB drives the resilver operation could still take a rather long time, during which the pool would be at risk.
For 10 TB to store, and knowing that block storage pools should aim for 50% occupancy or less, 2 * (3-way mirrors), with a matching 3-way mirror for metadata, could be considered for maximal security.
Of course, you have backups.

Regarding the use of SATA ports on the new motherboard, I am considering this option. However, I am uncertain if it would be the most efficient choice or if an HBA would be preferable. I would appreciate any feedback or suggestions on this matter.

SATA ports from chipsets are perfectly fine for ZFS. If you have enough of them there's no need for a HBA.

 

[john.o]

All correct. But with 16 TB drives the resilver operation could still take a rather long time, during which the pool would be at risk.
For 10 TB to store, and knowing that block storage pools should aim for 50% occupancy or less, 2 * (3-way mirrors), with a matching 3-way mirror for metadata, could be considered for maximal security.
Of course, you have backups.

Good point and didn't take the rebuild times into consideration although I do wonder how long a resilver would take from a RAIDZ2 vs a mirror vs a 3-way mirror. A 3-way mirror might be a bit too much redundancy/storage penalty if I have multiple backups.

SATA ports from chipsets are perfectly fine for ZFS. If you have enough of them there's no need for a HBA.

That's great and will make sure that the ports are not in any sort of "RAID" mode to prevent any issues.

As far as CPU speed is concerned, I know that it is a bit low with 3.3Ghz boost. Would that negatively impact performance on iSCSI shares?

 

[Etorix]

For the same amount of data, raidz2 would take longer to resilver than a mirror, but contrary to a 2-way mirror it would still have redundancy after losing a disk.
The suggestion to go 3-way mirrors, i.e. to dedicate six whole drives to have the raw space of two and the usable space of just one (16 TB) at 50% occupancy, highlights the amount of resources used by this pool. Plus a Xeon Scalable system and 128 GB RAM, which probably holds the whole working set in ARC. That's a lot of hardware for serving 10 TB. This amount of data could fit in just one hard drive inside the client computer, or two drives in RAID0 for some redundancy (but without ZFS checksumming for integrity), or a handful of SSDs, backed up 3-2-1.

As far as CPU speed is concerned, I know that it is a bit low with 3.3Ghz boost. Would that negatively impact performance on iSCSI shares?

That would impact SMB, which is notoriously single-threaded. I hope that iSCSI is smarter than that, but I have no experience with this protocol.

 

[john.o]

That's a lot of hardware for serving 10 TB. This amount of data could fit in just one hard drive inside the client computer, or two drives in RAID0 for some redundancy (but without ZFS checksumming for integrity), or a handful of SSDs, backed up 3-2-1.

I completely agree. Initially, my server had only four 16TB disks, while the other four 16TB disks were set up in a RAID 10 configuration via Storage Spaces on my workstation, which worked well since TrueNAS only served as a backup to my workstation through an SMB share. However, after reading a few books on ZFS and learning about its performance & self-healing nature, I became curious and decided to move all my workstation's drives to TrueNAS and work off my NAS to see how it would perform.

So far, using a single iSCSI share over a 10Gb network has been working well, but I started thinking that if I commit to working off the NAS, I should invest in ECC memory and server-grade hardware. I considered getting a TrueNAS Mini XL+, which fits my hardware needs better, but it costs $1,600 USD for a diskless setup. I realized that for just $200 more, I could have a significantly more robust system with the capability of running VMs in the future, so going with a Xeon Scalable setup seems like a no-brainer to me given the small price difference.

As always, thanks again for chiming in and offering your opinions/feedback.