I am in the process of building a new TrueNAS machine with an all-SSD pool. I have read through a number of forum posts here and elsewhere in an effort to try and compile what would be the "best practices" for building an SSD pool. My intent with this post is to archive what I have found as well as get some community review as to whether there is something I could be doing better (or is just plain wrong).
The NAS being built will have two use cases: SMB share for general (infrequent) file storage and S3 for object storage, each on its own dataset.
Hardware
Given ZFS's copy-on-write nature, TRIM support is critical, which requires DRAT (deterministic read after trim) and RZAT (read zeros after trim) when using an LSI HBA in IT most (reference). I wanted to stick with ~4TB drives to give a sufficient balance of price and reliability. My initial options considered were Samsung 870 EVO (TLC), Samsung 870 QVO (QLC), and Crucial MX500. TLC has better longevity over QLC, however questionable reliability of the Samsung drives (reference, reference) and a lack of proper TRIM on the MX500 led me to look elsewhere (plus they are consumer-grade drives). Eventually I stumbled upon the Micron 5200/5300 drives which can be had for an acceptable price on eBay. These are enterprise drives with SED support (if I go that route) but also have great durability and full TRIM support, which is expected for enterprise drives.
Performance sudo-testing
While waiting for the X11 and Xeon to arrive, I ran some early tests using some adhoc hardware (an X10 motherboard w/ i3 CPU). I was able to consistently get 5Gbps read/write over the X520 via SMB with a pool configured for Raid-Z1. The CPU was regularly in the 70%-80% with both compression and deduplication on. I need to re-test, but I believe that during on both only showed limited reduction in CPU usage. Thus, my current assumption is that the i3 just isn't strong enough to support more throughput. Once I have the X11 parts in hand I will be able to re-test.
The one case I did test was with encryption on/off for the pool. Having encryption only seems to contribute 10%-15% CPU usage, although I need a more thorough test on the actual hardware to see if this is true.
Tuning
While there seems to be a log of options for tuning a setup like this, most of the defaults seem to already be correct.
The NAS being built will have two use cases: SMB share for general (infrequent) file storage and S3 for object storage, each on its own dataset.
Hardware
- Supermicro X11SCL-F motherboard
- 32GB ECC RAM
- Xeon 2174G 3.8 GHz
- LSI 9003-8i HBA in IT mode
- Intel X520-DA1 NIC for 10Gb networking
- 8 x Micron 5200/5300 3.84TB SATA SSDs
Given ZFS's copy-on-write nature, TRIM support is critical, which requires DRAT (deterministic read after trim) and RZAT (read zeros after trim) when using an LSI HBA in IT most (reference). I wanted to stick with ~4TB drives to give a sufficient balance of price and reliability. My initial options considered were Samsung 870 EVO (TLC), Samsung 870 QVO (QLC), and Crucial MX500. TLC has better longevity over QLC, however questionable reliability of the Samsung drives (reference, reference) and a lack of proper TRIM on the MX500 led me to look elsewhere (plus they are consumer-grade drives). Eventually I stumbled upon the Micron 5200/5300 drives which can be had for an acceptable price on eBay. These are enterprise drives with SED support (if I go that route) but also have great durability and full TRIM support, which is expected for enterprise drives.
Performance sudo-testing
While waiting for the X11 and Xeon to arrive, I ran some early tests using some adhoc hardware (an X10 motherboard w/ i3 CPU). I was able to consistently get 5Gbps read/write over the X520 via SMB with a pool configured for Raid-Z1. The CPU was regularly in the 70%-80% with both compression and deduplication on. I need to re-test, but I believe that during on both only showed limited reduction in CPU usage. Thus, my current assumption is that the i3 just isn't strong enough to support more throughput. Once I have the X11 parts in hand I will be able to re-test.
The one case I did test was with encryption on/off for the pool. Having encryption only seems to contribute 10%-15% CPU usage, although I need a more thorough test on the actual hardware to see if this is true.
Tuning
While there seems to be a log of options for tuning a setup like this, most of the defaults seem to already be correct.
- Raid-Z1 vs Raid-Z2: The data will be backed up nightly to a secondary NAS, so the loss of a single drive is not immediately catastrophic. However, Raid-Z2 does show a ~7% performance improvement over Z1 (reference), and a slightly less usable space in the pool is acceptable. Plus the general guidance for 8 drive is Raid-Z2.
- Pool Auto-Trim: Best to leave it off, even though it seems counter-intuitive and instead use daily trimming during off-peak hours (reference)
- ZFS primarycache set to metadata: Although an LTT video shows them configuring this for a TrueNAS setup (reference), there appear to be implications when reading data which results in a hit to performance (reference). Thus, the guidance would seem to be 'just leave it set to all'.
- More RAM: The mantra of "more RAM is better" holds true, but money is a finite resource and 32GB is where I landed for now.
- SMB multichannel: This is one performance adjustment that I have not yet had a chance to investigate but may be a contributing factor to why I could not get above 5Gbps over SMB.
