First TrueNAS build - advice on drive setup and boot media

[BitByteBit]

Hi TrueNAS community,

This will be my first build (and first post). I've done a fair bit of reading and research and have settled on some hardware components, but unsure how to best setup and optimize my storage and boot media (and if I'll benefit from SLOG at all).

Decided hardware:

Motherboard: Supermicro A2SDi-H-TF (https://www.supermicro.com/en/products/motherboard/A2SDi-H-TF), which supports 12 SATA3 ports - 4 SATA3 ports; 2 MiniSAS HD ports. It also has 1x m2 with pcie 3.0 x2 on board. I plan to add another pcie3.0 x4 to m2 adapter in the pcie slot on the board, which gives me a total 2x m2 drives.
Also has 1x internal USB 3.0 port which I could potentially use for a boot drive.
CPU: (Embedded Intel Atom C3758)
RAM: 32 GB (2x 16GB ECC RDIMM DDR4)
NIC: 2x Intel RJ45 10GBase-T ports


Use case for NAS:

Storage shares via both NFS and SMB for less than 5 users.
1 or 2 jails for things like Git server, Syncthing etc.

Initially connected over 1G ethernet (probably 2x LACP LAG), but plan to move to 10G BASE-T in near future.

Storage Options (seeking advice):

I'm wanting to optimize (and future proof) the build as much as possible.
I'll likely have 2 pools - 1 larger pool for fileshare / archive, and 1 smaller & faster SSD pool for jails and minimal file share usage

I was originally planning on using 4x 12TB WD HDDs for the larger pool in RAIDz2. Thinking RAIDz2 to be able to sustain *any* 2 drive failures, instead of 2x mirrored VDEVs, even though lower performance.
This will use 4 out of my 8 hot-swap drive bays, leaving room to add 4 more drives (another 4x disk RAIDz2 vdev?) in future.

Then plan to use 2x SATA SSD in mirror vdev for the second smaller pool, leaving 2x SATA ports free for further expansion.

For boot, am thinking either a USB boot drive (flash drive or USB-connected SSD), as I don't want to waste the 2x m2 NVMe slots on boot. Is this correct? Otherwise I could use 2x mirrored SATA SSD for boot on the remaining 2x SATA ports?

I could then use the 2x m2 slots with mirrored Intel Optane 16GB m10 memory modules for SLOG, although I'm not convinced my NFS use will really justify this? Also I realise these Optane memory dont have best endurance, but they're really cheap. Not really sure what best to use the m2 slots for, except maybe putting the the 2x mirror SSD VDEV here instead of on the 2x SATA ports as mentioned above?


Alternatively...

I could skip the 4x HDDs entirely, and instead use 4x Samsung QVO SSDs, which will give less storage but hopefully quieter, faster and more energy efficient.

Would love to hear the thoughts and suggestions from the community.
Thank you.

 

[BitByteBit]

Bump... anyone?

 

[Tigersharke]

I would strongly recommend against using a USB flash drive and believe that trying to use the USB port/subsystem for an SSD could also be problematic at least from any limitations of USB itself, compared to SATA or other connections.

Generally it is best not to put any sort of automation or control between ZFS and the media (drive), in other words give ZFS direct easy access to the media so it can do its thing and know everything about what is happening. This is why it is a bad idea to use a hardware raid controller (which still works as its own raid) for ZFS, or caching apart from what ZFS would do, since ZFS could recognize a problem and either not be able to correct it, be out of sync, or be fighting with some other mechanism to do the same thing -- much like having two anti-virus programs installed is a very bad idea.

Anything further, as well as advice counter to my own, I leave to the more seasoned actual FreeNAS/TrueNAS experts.

 

[BitByteBit]

Thanks Tigersharke.
After further research and following the advice to avoid USB for boot, I'm leaning towards the following solution, which I'd appreciate any advice on:

Boot: Crucial MX500 250GB or WD Blue 250GB SATA SSD (possibly both as a mirror)
"Fast pool": 2x Samsung 980 1TB NVMe M2 drives as a mirror vdev for a pool used for Jails and file sharing requiring lowest latency
"Archive / fileshare pool": 4x Samsung 870 EVO 2TB SATA SSDs in 2 mirrored VDEVs as my main pool for archiving and fileshare. (Given up on the QVO idea, and probably go with 2 mirrors as opposed to 1 raidz2)
This leaves 4x hot swap bays free for future expansion.
No SLOG.

But I am open to any other suggestions?

 

[HarambeLives]

Agreed on the boot SSD's instead of flash drives. I think a mirror of SSD's would be much better. If you buy new, consider getting two completely different drives.

On more than one occasion in my life I've had brand new SSD's suffer crib death at the exact same time (Usually large deployments of desktop PC's)

Personally I would just buy used Intel SATA SSD's, under $30 you can get a quality SSD

Intel SSD for sale | eBay

Get the best deals for Intel SSD at eBay.com. We have a great online selection at the lowest prices with Fast & Free shipping on many items!

www.ebay.com

 

[HarambeLives]

I think you are on the right track there, that sounds like a solid plan. I was in the same boat as you looking at the Samsung 980, but it is DRAMless. In my research, I found that the general consensus is that the 970 EVO/EVO Plus will be more reliable.

I'm sure the 980 is fine, but I went with the 970

 

[BitByteBit]

Thanks @HarambeLives for the good tip about the Samsung 980 being DRAMless. I'll take a look at the 970 Evo instead (the 970 Evo Plus has higher speeds but I don't need to pay the premium for them).

Regarding your comment about SSDs failing at the same time, how do you monitor and preempt this in a mirrored pair?
Also if I only have a single boot SSD, but I have a full config backup, and it fails, can I simply just install a new SSD then and restore from the backup (with the asociated downtime hit)?

 

[HarambeLives]

You can monitor SMART, but in my experiences SSD's tend to often die catastrophically, where they just completely fail entirely, or become read only with very little if any warning. It doesn't happen very often though, I haven't seen many good SSDs fails (Lots of cheap ones though!)

I have identical SSD's in my system, but they are both used and from different sources. So the chance of them failing at the same time is very slim. But if you order 2 SSD's brand new, and get very close batches, and then have a mirror with identical workloads and the SSD's have some kind of manufacturing problem or defect, you could end up with 2 failing at the same time which wouldn't be great

Yes, you can restore the backup and import the pool, but for the small price of SSD's, I feel like that's not really worth it. Especially since it looks like you will have pretty nice hardware

 

[BitByteBit]

Yes good points, & that's what I'd read as well (about sudden failure) which worries me, as I will be monitoring SMART.
For the boot SSDs, I'm more concerned about using up another sata port, than the minimal cost of a 2nd drive, but I may reconsider this ;)

I guess my biggest worry will be the 2x Samsung NVMe drives in a mirror and the mirrored Samsung 870 EVOs, all of which will be brand new and ordered at the same time... I may have to try spread my order over different suppliers or times potentially to avoid drives from same batch.

 

[serfnoob]

Hi TrueNAS community,

This will be my first build (and first post). I've done a fair bit of reading and research and have settled on some hardware components, but unsure how to best setup and optimize my storage and boot media (and if I'll benefit from SLOG at all).

Decided hardware:

Motherboard: Supermicro A2SDi-H-TF (https://www.supermicro.com/en/products/motherboard/A2SDi-H-TF), which supports 12 SATA3 ports - 4 SATA3 ports; 2 MiniSAS HD ports. It also has 1x m2 with pcie 3.0 x2 on board. I plan to add another pcie3.0 x4 to m2 adapter in the pcie slot on the board, which gives me a total 2x m2 drives.
Also has 1x internal USB 3.0 port which I could potentially use for a boot drive.
CPU: (Embedded Intel Atom C3758)
RAM: 32 GB (2x 16GB ECC RDIMM DDR4)
NIC: 2x Intel RJ45 10GBase-T ports


Use case for NAS:

Storage shares via both NFS and SMB for less than 5 users.
1 or 2 jails for things like Git server, Syncthing etc.

Initially connected over 1G ethernet (probably 2x LACP LAG), but plan to move to 10G BASE-T in near future.

Storage Options (seeking advice):

I'm wanting to optimize (and future proof) the build as much as possible.
I'll likely have 2 pools - 1 larger pool for fileshare / archive, and 1 smaller & faster SSD pool for jails and minimal file share usage

I was originally planning on using 4x 12TB WD HDDs for the larger pool in RAIDz2. Thinking RAIDz2 to be able to sustain *any* 2 drive failures, instead of 2x mirrored VDEVs, even though lower performance.
This will use 4 out of my 8 hot-swap drive bays, leaving room to add 4 more drives (another 4x disk RAIDz2 vdev?) in future.

Then plan to use 2x SATA SSD in mirror vdev for the second smaller pool, leaving 2x SATA ports free for further expansion.

For boot, am thinking either a USB boot drive (flash drive or USB-connected SSD), as I don't want to waste the 2x m2 NVMe slots on boot. Is this correct? Otherwise I could use 2x mirrored SATA SSD for boot on the remaining 2x SATA ports?

I could then use the 2x m2 slots with mirrored Intel Optane 16GB m10 memory modules for SLOG, although I'm not convinced my NFS use will really justify this? Also I realise these Optane memory dont have best endurance, but they're really cheap. Not really sure what best to use the m2 slots for, except maybe putting the the 2x mirror SSD VDEV here instead of on the 2x SATA ports as mentioned above?


Alternatively...

I could skip the 4x HDDs entirely, and instead use 4x Samsung QVO SSDs, which will give less storage but hopefully quieter, faster and more energy efficient.

Would love to hear the thoughts and suggestions from the community.
Thank you.

Sorry for the bump just wondering if you built this system and how it going for you? I wanted to do the same.