Boot Devices for TrueNAS Scale Bluefin

[Michael Schefczyk]

Dear All,

The hardware recommendations contain the quite open statement "modern TrueNAS versions perform increased drive writes to the boot pool. For this reason, all prebuilt TrueNAS Systems ship with either M.2 drives or SATA DOMs."

I wonder what this means. Please imagine a hyperscaled system with one all flash pool for KVM virtualization and kubernetes, one spinning disk pool for classical NAS and two mirrored boot devices. The questions are:

1) Are quality SATA DOMs still good enough in terms of (a) durability and (b) not slowing down when used instead of M.2 drives? For example, one may use two 32 GB Supermicro SATA DOM SSD-DM032-SMCMVN1 https://www.supermicro.com/products/nfo/SATADOM.cfm with 50 MB/s write speed and 1 DWPD endurance.

2) Alternatively, one could use two 256 GB SAMSUNG 870 EVO https://www.samsung.com/de/memory-storage/sata-ssd/870-evo-250gb-sata-3-2-5-ssd-mz-77e250b-eu/ with 530 MB/s write speed 150 TBW. The endurance is obviously better, but would one also notice the increased (write) speed in terms of virtualization and kubernetes performance.

3) Alternatively, one could use two M.2 drives like GigaByte Aorus NVMe 500 GB with 2.500 MB/s write seed and 850 TBW. They would consume two M.2-slots which are not available in sufficient quantities in most situations. Obviously, their endurance is much better, but would one also notice the increased speed at all (I do not care about the boot time, but about the virtualization, kuernetes and NAS performance)?

Regards,

Michael Schefczyk

​

 

[joeschmuck]

I'm a bit confused, or maybe you are? The boot device only boots TrueNAS (Core or Scale) and any extra storage on the boot device cannot be used for anything else (by normal supported setup), well but then again I have not looked at Bluefin so if that includes the ability to have the boot device support multiple items other than just bootstrapping, I'm just not aware of it.

 

[Michael Schefczyk]

Yes, I am confused. I have no intention whatsoever to do anything extra with/on the boot drive. I just do not know what the cryptic "modern TrueNAS versions perform increased drive writes to the boot pool. For this reason, all prebuilt TrueNAS Systems ship with either M.2 drives or SATA DOMs." means. My goal is to avoid any impediments against KVM, Kubernetes and NAS performance. If this means that a boot drive with slower (write) speed may cause a bottleneck, I would pick a faster one. If more writes mean that a small boot drive with 1 DWPD endurance may become a hurdle, I would pick a more durable one (albeit 32 GB of writes per day for five years is probably more than a normal TrueNAS Scale system will ever write to its boot drive. If nothing in this direction would have any noticeable impact, I would stay with high quality SataDOMs.

 

[joeschmuck]

The boot drive as I recall basically now has more write operations whereas a few versions ago the write data was very limited because we use to use USB Flash Drives which had a write endurance problem. Now TrueNAS uses better quality boot devices of either a DOM or M.2 SSD, both of which are SSD's. I don't recall what data is written to the boot device but you can place the System Dataset on the boot device as I recall which is where the SSD will shine. I keep my System Dataset on my pool, the default location.

The move to a SSD is for endurance.

If nothing in this direction would have any noticeable impact, I would stay with high quality SataDOMs.

This sounds like your path forward.

 

[Davvo]

It's basically a fancy "no USB are used here" sign.

 

[JmarcSyd]

It may be slightly off-topic but my recent post about the PH58-JMB582 card may be a good solution for mirrored boot M.2 SATA drives (I haven't tested in mirror mode though).

 

[joeschmuck]

It may be slightly off-topic but my recent post about the PH58-JMB582 card may be a good solution for mirrored boot M.2 SATA drives (I haven't tested in mirror mode though).

Yes, a bit off topic. So for TrueNAS (any version) mirrored boot drives does not mean automatic failover. There is only one way to have automatic failover and that is to use a RAID card in RAID mode that supports this and it would present the two drives to TrueNAS as a single drive. Too many people think that automatic failover is built into TrueNAS, it's not. The only thing you would have is a duplicate hopefully working copy of the boot drive. You would need to tell the BIOS the new boot drive or physically remove the failed drive and insert the good drive into it's slot (the BIOS may still need to be accessed on some motherboards).

 

[JmarcSyd]

My apologies then. Thanks for the info about the lack of failover. I won't bother mirroring the boot drive and just stick to regular configuration backups.

 

[WI_Hedgehog]

Documentation Hub/TrueNAS SCALE/Getting Started with SCALE/SCALE Hardware Guide:

You do not need an SSD boot device, but we discourage using a spinner or a USB stick for obvious reasons. We do not recommend installing TrueNAS on a single disk or striped pool unless you have a good reason to do so. You can install and run TrueNAS without any data device, but we strongly discourage it.

TrueNAS does not require two cores, as most halfway-modern 64-bit CPUs likely already have at least two.

From the posts in this thread I have come to understand:

• USB sticks are unreliable, don't use them if you can avoid doing so.
• Hard Disk Drives (spinners) are not as reliable as Solid-State Devices, use SSD where possible.
• SSD has no real advantage over spinners other than reliability, excepting Install/Config Changes/Updates will be faster.
• 16GB (or maybe 20GB) boot device is the minimum, 32GB is recommended and is quite sufficient.

1.) Is my understanding correct so far?

• The preferred configuration is 1 SSD boot drive, and if it fails shut down the server, replace it, install TruNAS from USB stick, and import the Configuration from backup.

----

2.) If/when the boot drive fails, does it "fail safe" so the dataset is not corrupted?

Coming from a non-ZFS/non-TrueNAS background I would otherwise 3-way mirror (triple mirror) the boot drive via hardware RAID 1E so if one drive failed the other 2 would remain online, replacing the failed drive would cause the array to rebuild and become fully resilient. I think that's not preferred with TrueNAS SCALE, but don't understand why not given small SSDs are inexpensive.

3.) What is the proper way to configure the boot "device" for TrueNAS?

 

[Davvo]

@WI_Hedgehog

1) SSD has better performance and lower noise compared to HDD. But they can fail unexpectedly, which is something that doesn't happen much with HDD if you monitor the smart data.
Boot device of such sizes are beconing harder to find but basically yes.
Preferred config is right.
2) if the boot drive dies everything in there do so too. In regards to the redundancy question please read the following resource.

Highly Available Boot Pool Strategy

I've pounded a few versions of this out over the years, but I hate explaining over and over. TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw. Due to...

www.truenas.com

3) you don't configure it, you install the OS on it. And nothing else. ZFS does not like multiple partitions on the same drive.

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com