Boot device: USB or M.2 2280

[Monkey_Demon]

Hello,

I'm building a NAS drive with a Supermicro X11SSH-F motherboard and trying to decide what to use as the boot device, and I'm asking you for advice.

The board can boot off of any one of several possible devices, but the two most likely options are a USB flash drive or a M.2 PCI-E 3.0 x2 drive. Here are the prime candidates, along with comments:

• USB: a 16GB SLC flash drive. The MX-Technology 16GB USB 3.0 MX-ES Series SLC Flash Drive - MXUB3SES-16G (currently $52.61 on Amazon) is a good example. Less expensive, less obtrusive than the M.2. SLC technology should make it last longer than the more common MLC technology, but I'm still nervous about using a flash drive.
• M.2: These do not seem to be available in small sizes. Also, reviews on some of the less expensive ones report high failure rates, excess heat, and other problems. Right now the top candidate is a Corsair Force Series MP500 120GB M.2 NVMe PCIe Gen. 3 x4 SSD selling on Amazon for $65.99. I have more confidence in an SSD than a flash drive. This particular drive is overkill: the MB is x2, but the drive is x4; and I don't need 128 GB for a boot drive. Still, for the small price increment over the USB, it may be worthwhile.

Another thread, M.2 SSD and X11SSH-F motherboard, was very helpful to learn about this stuff, but the M.2 world is filled with black arts straight out of Ashai. I'm hoping someone reading this understands all that stuff and can advise.

Also, if I were to get the SSD, is there any productive way to use the extra storage (e.g. partition the device and use most of it for Docker containers)? What would you recommend?

Thanks!

 

[Inxsible]

Also, if I were to get the SSD, is there any productive way to use the extra storage (e.g. partition the device and use most of it for Docker containers)?

You can use it to for your jails and system configuration on it. Not particularly sure about Docker.

What would you recommend?

SSD > USB > HDD in terms of performance. But that is never the only criteria to go with. Price comes in to play big-time especially with home users. I feel both your choices are extreme overkill.

I use a Sandisk Cruzer 8GB USB for 2 reasons:

1. My board had 6 SATA ports and my case supported 6 HDDs and I wanted to use all 6 for my storage HDDs
2. I had it lying around

I have been using the same USB key without failure since the first day I put my FreeNAS box together. YMMV. But the Sandisk Cruzer Fit USB costs about $10 online. If you are going to go with USB, I would not spend $52.61 on a USB. You would be able to get a 120GB SSD easily or even a 256GB SSD for less than that if you look hard enough.

Now M.2 will definitely be much faster than the USB or even the SATA SSDs... but then again how often are you going to reboot your NAS?

If you have an additional SATA port available, I would use a decent SATA 2.5" SSD. This should cost you about $35-$55
If you don't have an extra SATA port available, you can simply use a USB if you want to save money or you can choose to use the NVMe drive since your board supports it. Comparing it to a $52 USB is one thing but comparing it to a $10 USB, are the advantages of NVMe still relevant to you?

Edit :With USBs you can choose to put them in a mirror or even keep a cold spare in case the USB craps out.

 

[MrToddsFriends]

If you have an additional SATA port available, I would use a decent SATA 2.5" SSD. This should cost you about $35-$55

This is most often recommended here, if a free SATA port is available. A 60 ... 64 GB SATA SSD has plenty of space for boot environments as well as unused space (read: is available for SSD-internal wear leveling mechanisms), typically is more reliable and faster than any USB flash drive and doesn't cost a leg and an arm.

 

[Inxsible]

This is most often recommended here, if a free SATA port is available. A 60 ... 64 GB SATA SSD has plenty of space for boot environments as well as unused space (read: is available for SSD-internal wear leveling mechanisms), typically is more reliable and faster than any USB flash drive and doesn't cost a leg and an arm.

It's difficult to find a decent 60/64GB SATA SSD for cheaper than the 120GB at least in the USA. I have seen some Chinese brands like KingDian 32 GB and even 64GB for around $18-$25, but I would rather spend $35 on a Crucial MX or AData SU800 120GB

 

[Stux]

One of the more interesting boot drive options is an intel 16/32GB Optane m2.

 

[Monkey_Demon]

Now M.2 will definitely be much faster than the USB or even the SATA SSDs... but then again how often are you going to reboot your NAS?

I'm very new to FreeNAS, so I'm not familiar with its system architecture. But in my experience, "boot drive" is a bit of a misnomer: "system drive" would be better. This is because booting is only one of its functions. The system drive is usually used for swap space and system modules are loaded from the drive on an as-needed basis. Speed is a real asset for these functions, especially swapping.

But this is a small nit to pick. Your answer is very helpful. Thank you.

 

[Monkey_Demon]

It's difficult to find a decent 60/64GB SATA SSD for cheaper than the 120GB at least in the USA. I have seen some Chinese brands like KingDian 32 GB and even 64GB for around $18-$25, but I would rather spend $35 on a Crucial MX or AData SU800 120GB

Newegg is having a flash sale with a Mushkin 120GB SATA III drive for $30. I may take your advice. Thanks.

 

[Monkey_Demon]

One of the more interesting boot drive options is an intel 16/32GB Optane m2.

I thought the Optane line is used only as cache for slower mass-storage devices. It might be quite effective if one could, for example, partition a 32 GB Optane to have a 8 GB boot partition and use the remaining 24 GB as cache for all the hard drives on the system.

Has anyone been successful doing something like this?

 

[Ericloewe]

SLC technology should make it last longer than the more common MLC technology

Nope. That is a gross oversimplification that kinda applies to similar SSDs of the same era. I guarantee you that, on average, over a large population, any consumer SSD from Samsung, Toshiba/SanDisk/WD, Crucial or Intel will outlive any USB flash drive that isn't an SSD in disguise - firmware bugs and such excluded, as they may affect any model of anything.

The system drive is usually used for swap space

It is not, in FreeNAS. And if you're swapping at all (more than a few MBs because the allocator took a bit longer), you have problems that need solving - and swap isn't the answer.

system modules are loaded from the drive on an as-needed basis. Speed is a real asset for these functions

Not a factor. Even a crappy little USB flash drive is good enough during normal operation. You will notice the difference when manipulating boot environments, though.

SSD > USB > HDD in terms of performance.

I strongly disagree. A typical USB flash drive is significantly slower than a modern HDD. USB flash drives are optimized for cost above all, in most cases.


The bottom line is as follows:

• USB flash drives are "good enough". The reliability is often dubious and speeds doubly so.
• Typical consumer SATA SSDs are good choices, but the extra SATA can be rather expensive in some cases. This applies to M.2 SATA SSDs as well, except that they're a bit more expensive than 2.5" form factor drives. Think whatever is cheapest from Toshiba/SanDisk/WD, Intel, Samsung and Crucial.
  
• Low-end enterprise/high-end consumer SATA SSDs are perfect for businesses. You get the warm, fuzzy feeling of the word "enterprise" and you probably have an LSI HBA plus an expander for your main pool, with the PCH SATA ports being free. Think Samsung 860 Pro or one of the SATA Intels.
  
• NVMe/PCIe is crazy overkill, but may make sense in some situations. Say you have a free PCIe slot (or M.2 slot wired for PCIe) but no SATA ports - the SSD plus an adapter (if needed) might be cheaper than an HBA.
• As for capacity: Just ignore it! Who cares if the 120GB SSD is never going to even see 20GB full? Why would you spend time tracking down something older, possibly even used, just because it's smaller? I've heard of people buying old SLC SSDs "because they're better" for more than a new, low-end SSD. That's pure folly! The "better" part is very dubious, especially where used SSDs are concerned and it doesn't get you anything. (Note that buying an older used SSD might make sense while SSD prices are still high, but it's not something I particularly recommend)

 

[Ericloewe]

I thought the Optane line is used only as cache for slower mass-storage devices.

It's a standard NVMe drive. Everything else is Intel marketing and system firmware.

It might be quite effective if one could, for example, partition a 32 GB Optane to have a 8 GB boot partition and use the remaining 24 GB as cache for all the hard drives on the system.

This is unlikely to help you, as they're not particularly fast and a 24GB L2ARC is probably not very useful. An 8GB boot disk is also on the rather small side.

 

[Inxsible]

I'm very new to FreeNAS, so I'm not familiar with its system architecture. But in my experience, "boot drive" is a bit of a misnomer: "system drive" would be better. This is because booting is only one of its functions. The system drive is usually used for swap space and system modules are loaded from the drive on an as-needed basis. Speed is a real asset for these functions, especially swapping.

But this is a small nit to pick. Your answer is very helpful. Thank you.

Not quite. The "boot" drive is used to pretty much write logs, configuration and such during normal operation. Log files tend to be smaller in size unless you are debugging and logging everything and in that case -- your USBs might crap out earlier than expected. But if you are debugging, you should have taken precautions to back up configs and have extra USBs on hand.

 

[Ericloewe]

The "boot" drive is used to pretty much write logs, configuration and such during normal operation

No, logs go to the system dataset. You can place the system dataset on the boot pool, but I don't recommend it unless it's mirrored. If the system dataset is unavailable, the system will crash.

 

[Monkey_Demon]

Nope. That is a gross oversimplification that kinda applies to similar SSDs of the same era. I guarantee you that, on average, over a large population, any consumer SSD from Samsung, Toshiba/SanDisk/WD, Crucial or Intel will outlive any USB flash drive that isn't an SSD in disguise - firmware bugs and such excluded, as they may affect any model of anything.

I was only talking about USB flash drives. All other things being equal, a SLC USB drive will last longer than a MLC USB drive. Totally agree about SSD being more reliable than USB flash drive.

But otherwise, thanks so much for your very informative post.