Pool setup suggestions

[csdesigns]

Hello everyone,

Admittedly, I'm a newb here when it comes to TrueNAS, so looking for any and all suggestions regarding optimal setup of ZFS pools... I've long been in the QNAP ecosystem with many of their different NAS products, but recent events and performance has got me looking over the fence to see how green the neighbor's yard truly is.

I recently installed TrueNAS Scale onto a TS-1679U-RP with the following specs (no apparent hw issues or incompatibilities):

• Intel(R) Xeon(R) CPU E3-1275 V2 @ 3.50GHz
• 32GB 1866MHz RAM
• 16x 4TB 7200rpm SATA HDD
• 2x M.2 SATA 512GB SSD
• QNAP QM2-2S-220A Dual M.2 SATA SSD Expansion PCIe Card
• Emulex OneConnect 10Gbps PCIe NIC
• 1x 120GB SATA SSD

TrueNAS OS was installed onto the 120GB SATA SSD (there is an available port for a second SSD that I can install if OS redundancy is really that heavily recommended). Compared to QTS (booted from the DOM), boot up performance has increased 10-fold, so I'm extremely pleased on that front thus far. I currently have 16x 7200rpm HDDs in the 16 drive bays, but plan to replace these within the next few months with 10TB 7200rpm drives.

The main purpose of this storage will be for archiving large video files (i.e. actual lossless video projects, not BD-rips or torrent downloads) and Cloned Images / TimeMachine backups. I have the 2x M.2 SATA drives available for 1TB caching, but it is unclear based on my research how beneficial this will be to me with ZFS, or even how to best set this up with TrueNAS.

I'm hoping someone here can take a quick look at the above and recommend an optimal storage pool setup. Whether I should create multiple storage pools, the layout type, etc. I currently run with RAID 6 or 10 on most of my storage units, if that is of any help.

Thanks.

 

[Ericloewe]

How are all those disks actually connected to the host?

The main purpose of this storage will be for archiving large video files (i.e. actual lossless video projects, not BD-rips or torrent downloads) and Cloned Images / TimeMachine backups.

Seems like a clear-cut case for RAIDZ2 or RAIDZ3, depending on your desired reliability, with two vdevs of 8 disks each.

 

[csdesigns]

Thank you for taking the time to respond.

How are all those disks actually connected to the host?

The 16x 7200rpm HDDs are connected via the TS-1679U-RP's internal backplane. The 2x 512GB M.2 SSDs are connected via a QNAP QM2-2S-220A Dual M.2 SATA SSD PCIe card. And the 1x 120GB SSD boot drive is connected via an available 6Gb/s SATA interface on the motherboard. This system is connected to a 10GbE network.

Seems like a clear-cut case for RAIDZ2 or RAIDZ3, depending on your desired reliability, with two vdevs of 8 disks each.

I'm think I'm in agreement with this assessment. I will likely go with two vdevs of 8x HDDs in a single pool. I'll use the 2x M.2 SSDs as a pool for running VMs and plugins. Will also likely upgrade from 32GB of RAM to 64GB.

Much appreciated for your support!

 

[Ericloewe]

The 16x 7200rpm HDDs are connected via the TS-1679U-RP's internal backplane

Sure, but the real question is "what controller(s) does the backplane have?"

 

[csdesigns]

Sure, but the real question is "what controller(s) does the backplane have?"

Good question. Given the hw is a QNAP product, this information isn't always the easiest to decipher. AFAICT, it uses a ASMedia ASM1062 for the backplane (I know - not exactly the most performant), and a Marvell 88SE9125 on the motherboard (being used for the SSD boot drive only).

 

[PhilD13]

There are people who have setup Truenas on QNAP machines with good results. I have not tried it personally though.

QNAP uses Linux as a base and their version of middleware (firmware) on top. Their firmware is installed on a internal DOM and contents (boot) copied to a hidden partition or area on the raid array I think for daily use. The arrays a user creates are mdadm based arrays. I believe mdadm creates software raid arrays, so the hardware while maybe not as performant as it could be, would be compatible and work with Truenas and I would think the backplane and controller connected to the backplane would also be compatible.. Some of the QNAP machines I think may be limited to 16GB of ram and only have 4 ram slots but it tends to be hard to figure out the true specs. sometimes.

 

[csdesigns]

Their firmware is installed on a internal DOM and contents (boot) copied to a hidden partition or area on the raid array I think for daily use. The arrays a user creates are mdadm based arrays. I believe mdadm creates software raid arrays, so the hardware while maybe not as performant as it could be, would be compatible and work with Truenas and I would think the backplane and controller connected to the backplane would also be compatible.. Some of the QNAP machines I think may be limited to 16GB of ram and only have 4 ram slots but it tends to be hard to figure out the true specs

Indeed, most QNAP devices use some form of eUSB DOM for boot. I have removed the DOM on this unit however, and instead am booting off a 2.5" SSD connected to an unused SATA port on the motherboard (not advertised in the QNAP spec page). I've installed TrueNAS Scale onto this unit and everything appears to be running optimally in terms of hardware recognition and compatibility, so I think I'm covered on the front.

As I am new to TrueNAS, my main question was surrounding which RAID level to deploy, as 16 primary storage disks seems to fall between the line in terms of documented recommendation here. But after Ericloewe's suggestion above and further research here on these forums and elsewhere, I've decided that for my purposes, two vdevs of 8 disks into a single pool will suit me best.

The next question then becomes what to do with the dual m.2 SSDs I also have installed in this unit, as their original use as a caching solution with QTS is not as relevant with ZFS. I can remove the PCIe card housing these and install it into a different QNAP system (1270U), which will continue running QTS for the time being. Or I can create another vdev and pool solely to run VMs + containers. Most likely, it'll be the latter.

Eventually, if all goes well with this TrueNAS experience, my goal is to migrate all of my other QNAP devices to TrueNAS. I have 2x 1679 units, a 1270, 670, and 659 units all stuck on very old QTS releases (I understand that not all of those units are necessarily going to be compatible with TrueNAS). I also have a couple of 853 units; these run the latest QTS, but OS performance on one of the units is an absolute drag, so moving off QTS is likely the end goal there as well.

 

[PhilD13]

For 16 4Tb disks I would agree and do a Raidz2 dividing up the disks into 2 vdevs of 8 disks. I would not in general put more than 10 disks in any one vdev and if the disks were large capacity like 18-20+TB, I might drop that down to a lesser number in each vdev. Raid levels are a triangle of capacity, cost, reliability. Each point of the triangle is a tradeoff of the other two points so you generally want to hover around the center sweet spot then bias towards what you feel comfortable with. Vdevs of 6 -10 drives in Raidz2 for me feels safe enough for my purposes, and has been proven itself reliable.
The QNAP systems were similar in setup with one being 2 arrays of 8 disks in Raid6 for the pool and the other being 2 arrays of 6 disks (larger capacity drives) in Raid6 for the pool on it. Both of these ran 24/7 for 7 years for data storage, backup, and file serving without data loss. I think there were a total of one redundandt power supply replaced, and 4 drives replaced using cold spares between the two systems in those 7 years.

With the m.2 drives you could make a fast pool and use them for VM's or other hot data that could benifit from fast access.

You won't see any value in cache until physical memory gets above 64GB and likely won't see benifit until you get to 128 or 256GB of physical (ram) memory. Below 64GB you are wasting time shifting data around between the cache and real (ram) memory which will slow things down.