USB not recommended? SSD are for boot/operating system?

FlangeMonkey

Contributor
Joined
Dec 6, 2012
Messages
111
Hi Guys,

A buddy of mine pointed this out in the release notes for 11.2 U4:

[NAS-101056] – Clarify that USB is not recommended as the operating system device in Guide

First off, when did FreeNAS stop loading the OS into memory and started using Operating System devices? I must have missed this, apparently, it's been years and I haven't been in a position, in all that time, to spot this change. Or I've been blind to it!

Secondly, why are SSD's now recommended? As far as I know, the OS device isn't persistent or doesn't require a lot of reads/writes, especially quick reads/writes and no scratch is needed on this device. I can understand reliability, but that can and often is mitigated.

Thanks,
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
when did FreeNAS stop loading the OS into memory
About five years ago (edit: more precisely, on 8 Dec 14), when 9.3 was released and the boot device was moved to a live ZFS pool.
why are SSD's now recommended?
Because they're cheap now, and USB sticks are generally crap. And with the boot device now being a live filesystem, we've been seeing lots more USB boot device failures--again, for the last five years. This is nothing new.
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
FreeNAS used to be based on NanoBSD, which was a nice platform for an appliance-class device. However, engineering this kind of embedded system is an ongoing challenge (spoken as the guy who created the predecessor to PicoBSD). It became very limiting when FreeNAS decided to start introducing more complex functionality such as modern GUI's and jails/VM's.

FreeNAS switched to a ZFS-based boot system which brought with it some significant benefits, including the ability to roll back releases easily, and to store the system on a checksum-protected filesystem to help reduce boot device corruption. Because iXsystems is using FreeNAS as a testing platform for TrueNAS, and because TrueNAS ships with SATADOM's or better, this was fairly simple calculus for them - there are no downsides for TrueNAS to go this way. FreeNAS end users aren't as lucky, because many of them are tight on SATA ports.

USB devices are just dandy as a boot device, conditionally ... the conditions being that you have a boot device that can tolerate relatively heavy workload. The problem is, almost no USB devices that users want to buy are suitable. They do not contain wear leveling or other data protection features, so running a live filesystem off the USB device tends to be destructive to the underlying flash. If you can find non-bargain-basement USB thumb drives, i.e. military or high endurance ones, they are expected to work fine, but the cost will be several times of what a small conventional SATA SSD is. As a result, the suggestion is now to go for an inexpensive SSD, or, better yet, pair of them.
 

FlangeMonkey

Contributor
Joined
Dec 6, 2012
Messages
111
I personally don't think the quality of cheap/crap USB devices is a reason for the recommendation, it the technology, not part. Regarding cost its irrelevant IMO, a quality USB stick/SD Card is still cheaper than SSD and the cheap SSD's are unreliable. This is, however, getting into requirements, which this type of recommendation is generalised.

Specifically saying "not recommended" normally implies a last resort, development only, not supported, unrealistic, underperforming or unreliability. Just saying its an option and we recommend SSD is more appropriate IMO, which is what the documentation does but the release notes don't, hence questioning it.

Having said that, please correct me if I'm wrong but there is no heavy workload to the OS device for IO read/writes. It is persistent and SWAP is on the storage.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
I personally don't think
I don't recall that anyone asked. The bottom line is that, since the release of 9.3, USB boot devices fail at a high rate. SSDs (even cheap ones) don't. The consequences of boot device failure are minimal, as long as you have a recent backup of your configuration (and encryption keys, if your pool is encrypted), but if they can be avoided at minimal cost (and they can), why not?

Your question was based on an incorrect premise--indeed, a premise that's been incorrect for a relative eternity (specifically, the last four major releases of the product). The reason for the recommendation has been explained.
 

FlangeMonkey

Contributor
Joined
Dec 6, 2012
Messages
111
I don't recall that anyone asked.
Anyone asked what?

Your question was based on an incorrect premise--indeed, a premise that's been incorrect for a relative eternity (specifically, the last four major releases of the product). The reason for the recommendation has been explained.
It has not been explained because "please correct me if I'm wrong but there is no heavy workload to the OS device for IO read/writes. It is persistent and SWAP is on the storage." which would be the main reason IMO.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
It has not been explained
I've explained it twice and @jgreco has explained it once--IOW, every post in this thread except yours has explained the answer: since the release of 9.3, which uses a live ZFS pool for the boot device, USB sticks have been failing at a much higher rate than previously. SSDs don't fail at nearly such a high rate. You're right that there isn't a heavy workload for the boot device, but there's considerably more of one than there was prior to 9.3.
 

garm

Wizard
Joined
Aug 19, 2017
Messages
1,555
due to the lack of sata power outlets in the Dell T20 I’m running dual Sandisk Ultra Fit, fairly reliable brand but small form factor (don’t want to knock them out it the ports) is bad for heat dissipation, as boot devices.

My highly unscientific observation is that USB sticks fail at about a rate of 9-15 months and the SSDs in my flash pool have racked up many times more then that in much heavier load (Nextcloud, mariaDB, zvol’s for VMs and other services) and show no signs of age as of yet.

When it’s time to retire the Dell I will definitely have a SATA device for boot.
 

FlangeMonkey

Contributor
Joined
Dec 6, 2012
Messages
111
I've explained it twice and @jgreco has explained it once--IOW, every post in this thread except yours has explained the answer: since the release of 9.3, which uses a live ZFS pool for the boot device, USB sticks have been failing at a much higher rate than previously. SSDs don't fail at nearly such a high rate. You're right that there isn't a heavy workload for the boot device, but there's considerably more of one than there was prior to 9.3.

Dude, seriously, tone down the passive-aggression. I was questioning the workload of the boot disk, not the 9.3, ZFS boot Pool, etc. Thanks for your added opinion on the workload.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I personally don't think the quality of cheap/crap USB devices is a reason for the recommendation, it the technology, not part.

Please restate. That simply doesn't parse for me.

Regarding cost its irrelevant IMO, a quality USB stick/SD Card is still cheaper than SSD and the cheap SSD's are unreliable. This is, however, getting into requirements, which this type of recommendation is generalised.

Certain forum posters have put extensive effort into identifying "more reliable" USB boot devices, even during the era where NanoBSD was in use, and there was still an uncomfortably high level of boot device failures. As a pragmatic observation, SSD's are at least an order of magnitude more reliable than USB thumb drives, even the cheap SSD's. USB thumb drives have numerous negatives, including that most of the "reliable" ones seem to be older now-impossible-to-find USB2 ones that cause other performance issues.

Specifically saying "not recommended" normally implies a last resort, development only, not supported, unrealistic, underperforming or unreliability. Just saying its an option and we recommend SSD is more appropriate IMO, which is what the documentation does but the release notes don't, hence questioning it.

Submit a bug report for the documentation if you wish, but I would not hold my breath for any correction. You can probably boot FreeNAS off a bunch of Jaz drives in RAID but I wouldn't try that either. ;-)

Having said that, please correct me if I'm wrong but there is no heavy workload to the OS device for IO read/writes.

Consider this correction, then: In the grand scheme of things, a FreeNAS host does not produce a "heavy workload" (100's of MBytes/sec), but unfortunately, USB thumb drives are not designed for even modest levels of ongoing writes. USB thumb drives are intended for ephemeral, not 24/7 continuous duty cycle, workloads. You plug 'em in, you write some pictures or a few documents, and unplug. They mostly work for that. The slow, continuous writes of FreeNAS to the boot device are observed to degrade USB thumb drives to unusability, often quite rapidly.
 

FlangeMonkey

Contributor
Joined
Dec 6, 2012
Messages
111
due to the lack of sata power outlets in the Dell T20 I’m running dual Sandisk Ultra Fit, fairly reliable brand but small form factor (don’t want to knock them out it the ports) is bad for heat dissipation, as boot devices.

My highly unscientific observation is that USB sticks fail at about a rate of 9-15 months and the SSDs in my flash pool have racked up many times more then that in much heavier load (Nextcloud, mariaDB, zvol’s for VMs and other services) and show no signs of age as of yet.

When it’s time to retire the Dell I will definitely have a SATA device for boot.

I have had loads of SSD's failures in my time with a heavy workload, but that was more the earlier days in an enterprise environment. I've only had one USB headless disk fail on a couple of different platforms and that's probably over 10 years (but I don't run my personal FreeNAS off USB). I've also had good experience with many enterprise servers using dual SD cards, just the one failure. At the moment I don't run FreeNAS off USB because I virtualize it. I'm just curious as to the changes for such configurations.
 

blanchet

Guru
Joined
Apr 17, 2018
Messages
515
With a USB-to-SATA adapter and a small SATA SSD, you can boot in USB but with a reliable boot device.
I use a Sabrent USB3-to-SATA and a small Kingston SSD as boot device in my HPE Microserver Gen8 for years and until now I have never encountered any failure. The adapter costs $9 and the SSD costs $30 on amazon.
 
Last edited:

FlangeMonkey

Contributor
Joined
Dec 6, 2012
Messages
111
Please restate. That simply doesn't parse for me.
What I mean is, when I look at recommendations, the quality and features of a part (be it a USB Stick, HBA, etc.) is a different subject. Therefore I don't think its fair if you recommend a USB stick to me and I use a free USB from a conference, then complain when it fails. I don't feel a recommendation should need qualifiers, of buts and maybes. I think it's implied that people should do their research into exact parts for the task, therefore the recommendation is usually based around the specific technology, feature or architectural layer.


Certain forum posters have put extensive effort into identifying "more reliable" USB boot devices, even during the era where NanoBSD was in use, and there was still an uncomfortably high level of boot device failures. As a pragmatic observation, SSD's are at least an order of magnitude more reliable than USB thumb drives, even the cheap SSD's. USB thumb drives have numerous negatives, including that most of the "reliable" ones seem to be older now-impossible-to-find USB2 ones that cause other performance issues.
I agree with you, especially when you talk about needs for a deployment. I'm thinking about the lowest denominator, like small SANs using stuff like T20's or MicroServers. I wouldn't be putting in a dedicated SSD. Thinking about it, I see the documentation is right recommended SSD, it's the "not recommending" on the release notes that made me say WTF :) hence my rant regarding what "not recommended" mean to me.


Consider this correction, then: In the grand scheme of things, a FreeNAS host does not produce a "heavy workload" (100's of MBytes/sec), but unfortunately, USB thumb drives are not designed for even modest levels of ongoing writes. USB thumb drives are intended for ephemeral, not 24/7 continuous duty cycle, workloads. You plug 'em in, you write some pictures or a few documents, and unplug. They mostly work for that. The slow, continuous writes of FreeNAS to the boot device are observed to degrade USB thumb drives to unusability, often quite rapidly.
Isn't the "slow continuous writes of FreeNAS to the boot device" logs and stats or is there other "stuff"? I recall at one point this data could be relocated to a specific pool.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
What I mean is, when I look at recommendations, the quality and features of a part (be it a USB Stick, HBA, etc.) is a different subject. Therefore I don't think its fair if you recommend a USB stick to me and I use a free USB from a conference, then complain when it fails. I don't feel a recommendation should need qualifiers, of buts and maybes. I think it's implied that people should do their research into exact parts for the task, therefore the recommendation is usually based around the specific technology, feature or architectural layer.

You can think what you like, but I can tell you that people tend to follow recommendations. The earliest generation of users here was full of AMD APU's with 4GB or less of RAM and there was a plethora of issues. I started pushing Supermicro (which we sell here, but not in competition with iXsystems) and people started discovering that all their platform-related problems ... went away. People generally do not do extensive research on their own on the exact parts for a task, so the experience here for those of us who have been providing advice and help for many years is that people want the answers that are most relevant. USB sucks. USB2 is too slow to be useful and causes strange performance problems. USB3 chipset support was dodgy for a long time, and still is on some controllers. USB thumb drives blow through endurance in a short period of time.

Isn't the "slow continuous writes of FreeNAS to the boot device" logs and stats or is there other "stuff"? I recall at one point this data could be relocated to a specific pool.

It used to be. Now it's anything that the base system writes. Because TrueOS is a general OS that does server and desktop stuff, it has substantial similarities to FreeBSD and seems to do a lot of the typical FreeBSD things. Some of us (raises hand) have been creating custom OS images for a long time. I originally did a floppy-based FreeBSD that was the basis for PicoBSD, many many years ago. I've moved on and now have a highly customized appliance version of FreeBSD that is used on hypervisor platforms that does quite a bit to minimize both unnecessary writes and also reduces a lot of superfluous reads. I've done a lot of work to reduce unnecessary I/O over the years - including the first "noatime" implementation for FreeBSD - so it's not like this stuff is unfamiliar to me. Observationally, FreeNAS does a fair bit of writing to its boot pool. Certainly enough to kill many USB thumb devices without wear leveling in somewhere between months and a year or two.
 

FlangeMonkey

Contributor
Joined
Dec 6, 2012
Messages
111
You can think what you like, but I can tell you that people tend to follow recommendations. The earliest generation of users here was full of AMD APU's with 4GB or less of RAM and there was a plethora of issues. I started pushing Supermicro (which we sell here, but not in competition with iXsystems) and people started discovering that all their platform-related problems ... went away. People generally do not do extensive research on their own on the exact parts for a task, so the experience here for those of us who have been providing advice and help for many years is that people want the answers that are most relevant. USB sucks. USB2 is too slow to be useful and causes strange performance problems. USB3 chipset support was dodgy for a long time, and still is on some controllers. USB thumb drives blow through endurance in a short period of time.
.

I think we are mostly on the same page for this. People do follow recommendations and that's my point, but the recommendations I was referring to earlier are vendor specific generalized recommendations (or more accurately "not recommended"), not your personal ones.

Thanks for your input on the IO side of things.
 
Last edited:

dsouleles

Dabbler
Joined
Nov 4, 2012
Messages
19
I know this is an old thread - but just an observation -. Here is a testament to the reliability and longevity of FreeNAS/TrueNAS...

I have been running FreeNAS and now TrueNAS off the same Sony USB stick for many years and never had a problem. Perhaps I am lucky or just an outlier. I only came across this now because I was reconfiguring my server and didn't want the big (2" long) thumb drive sticking out of the front of my server - so I picked up some SanDisk minis - which are 16GB so I was looking to install on those. It has been so long since I installed that I no longer remembered how to install. Imagine my surprise to learn that I had been running in a not recommended mode for at least 7 years. I'll probably pop an SSD in now that I have read the thread, but perhaps the warning is unnecessarily dramatic.

Cheers and thanks for the reliable storage - just replace my main photography work-horse machine and restored all my file to the local disk without missing a beat!

Dean
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
where was your system pool - I understand that its the system dataset that thrashes the boot disk and not the boot process.?
 

dsouleles

Dabbler
Joined
Nov 4, 2012
Messages
19
The System Dataset Pool is on the main ZFS storage volume - not the boot volume. I was surprised at the statement that a USB stick would die between months and a year or two when I have run with out a glitch for 7 years on the same Sony USB stick. Not questioning the advice or recommendation. Just curious. I've been running FreeNas and now TrueNas for 10 years on the same server - and always have booted from a USB stick. I did have to reinstall after one notoriously bad FreenNAS upgrade, but other than that it has been a pretty straight shot. I've upgraded the main storage pool disks twice. I've had one hard drive failure that was easily recovered from - but never had a problem with the USB stick.

When I started on FreeNAS if I recall the advice was to not "waste" a perfectly good SSD on a boot drive because it never got referenced after boot up. I understand the architecture is different now and that is no longer the case. But I am perhaps the exception the makes the rule - have had no problem running on a USB stick for years.

Like I said - just curious. All I really wanted to do yesterday was figure out if it was possible to clone and 8GB USB stick to 16GB and take advantage of that sweet, sweet extra 8GB. I learned that the answer is generally "no" but it is easy enough to re-install on a 16GB stick - except the recommendation now is not to do that.

Cheers,
Dean




Dean
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
I understand that its the System DataSet that kills the USB sticks. Not actually where you boot from
System DataSet on USB = rapidly dead USB
System DataSet on main Pool = no major issue boot from USB all year long (well 7 years)

I recall some checkpoint firewall appliances from some years ago - cheap for checkpoint (from memory) but they lasted about a year after which the constant logging (CP loves its logging) to the solid state disk in the box killed them
 
Top