Why you should avoid USB attached drives for data pool disks

[WI_Hedgehog]

If someone is on a technical forum and they are unwilling to read, they're an idiot.

That's most of the new members.

(I could say, "the majority of" if you want me to be wordy)

 

[NugentS]

If someone is on a technical forum and they are unwilling to read, they're an idiot.

I think you just described Joe Public

 

[calum74]

I think of people "unwilling to read..." as "experts in a different field," or "intelligent but very busy people." That kind of attitude belongs in the 1990s.

Anyhow, I definitely agree on the word "avoid" but just to point out that almost all of these drawbacks can be worked around and don't apply to every setup. USB would be terrible for enterprise, but we should support home users a bit better.

 

[Ericloewe]

Anyhow, I definitely agree on the word "avoid" but just to point out that almost all of these drawbacks can be worked around and don't apply to every setup. USB would be terrible for enterprise, but we should support home users a bit better.

You, along with everybody, are of course more than welcome to research and document the issue in depth. However, I suspect it would be a colossal waste of time due to the generally careless processes USB device manufacturers employ to track the finer details of what they're selling, with any results being outdated by the time they are published.

 

[Constantin]

I use the Oyen Digital external RAID enclosures (Mobius 5) w/o issues to create Mac-native backups of various file systems / pools. These enclosures tick a lot of the "gotcha" boxes, including the use of a JMicron port multiplier, consumer-grade eSATA/Firewire/USB interfaces, and hardware RAID 5. While hard drives have physically failed (which the box announces with a very loud beep), I have yet to experience silent or noticeable corruption. Yet I do not use these boxes for any primary storage. They are solely used for backups.

However, this is a very different use case from storing / manipulating a lot of data, having same accessed by multiple users, scrubbing, etc. What seems to work reliably on an occasional-use basis in a single-user scenario should not be confused as suitable for multi-user, constant high I/O enterprise applications. I'd limit these sorts of enclosures to backup use with a single user and perform periodic scrubs to ensure the data on them is still good (in my case, Carbon Copy Cloner does this via its "health check" feature).

I'd much rather use ZFS to transfer files, scrub, etc. But that would necessitate a connection to the server and OpenZFS to work on my Mac. Though it is nice to have the Oyen Mobius electrically isolated (i.e. 100% disconnected) all the time except when I am running backups and physically removed from the server location (making it less likely to be impacted by the same physical issue such as flooding).

 

[jgreco]

I think of people "unwilling to read..." as "experts in a different field," or "intelligent but very busy people." That kind of attitude belongs in the 1990s.

I definitely agree that your attitude belongs in the 1990's. The '90's were a terrible era for online knowledge, with difficult-to-search bulletin boards, Usenet, e-mail mailing lists, and pre-search-engine web pages. Information was hard to find, only exceeded by all the decades preceding it where you might have had to use a library. But this is the 2020's, and it is relatively easy to join user communities and find easy answers to common problems. Search engines work really well. There's a wealth of resources, forums, blogs, Reddit, even YouTube, offering insights and ideas so it is reasonable to expect people to avail themselves of some of these resources now and then.

USB would be terrible for enterprise, but we should support home users a bit better.

Well, then, by all means, feel free to take that upon your shoulders, put your money where your mouth is, or whatever your preferred way of saying it is. "We" (as in those of us who have more than a month of membership and three posts to our names) have been supporting home users just fine, the problem here isn't a lack of desire but rather the abysmal state of PC gear. The PC market is a race to the bottom of the barrel, and it simply isn't feasible to support a lot of the stuff that passes for "acceptable" in the Windows home PC consumer market. USB is particularly difficult due to its terrible architecture. Perhaps the best thing you can do with a USB drive is to shuck it and connect it directly to a host, and oh look, we have a guide for that....

EasyStore Drive Shucking like a pro ... by a pro

It has been quite common, for many years, for there to be awesome deals on large capacity hard drives by major vendors on Black Friday. The only downside is that they come in a plastic shell and are "external drives." This week, Post-Covidian...

www.truenas.com

Obviously it's frustrating to everyone involved that USB sucks, but that's just inherent in the consumer market. It's like wondering why they can't make a Ford Focus with the same level of quality as a Lexus or BMW. It's all about the target price point. It isn't some sort of conspiracy by the members of this forum to "support home users" poorly. Most of the membership here are home users.

 

[Constantin]

It's like wondering why they can't make a Ford Focus with the same level of quality as a Lexus or BMW. It's all about the target price point.

Great point. Many of the same sub-system suppliers are shared among the large brands - Bosch, Velus, among many others, yet the cumulative failure rate of some brands with simple platforms (Ford Focus) is significantly higher than much more complicated ones (i.e. Lexus). This is particularly apparent when expensive brands build complicated, low-quality cars (see Range Rover).

It's no different in the computer industry. You may be able to get away with a Lelon capacitor in your PSU but you'd much rather have a Nichicon 105*C, 5000 hr cap in there if given a choice.

 

[Arwen]

Hmm. To be clear, @Louis2, this was about data pool drives using USB as others have said. This forum discussion thread is for the Resource about that subject, and includes this exclusion right as the 2nd paragraph;

Please note that this is about USB attached storage for ZFS data pools. On occasion, people use USB drives, (flash, USB->SATA, USB->NVMe, etc...), for booting. In general, these do work acceptably, (when not using the cheapest USB flash drives). Though, it is always recommended to backup your configuration in case of boot drive failure.

It even suggests USB->SATA or USB->NVMe as more reliable than stock, (and likely cheaply made), USB flash drives.

And later, towards the bottom, the Resource has this;

All that said, USB attached storage does have a place with TrueNAS:

• Occasionally USB has to be used as the boot device connection method, (or better yet, boot Mirror).

All the comments about a live ZFS pool on a cheap USB flash drive seem to be right, in my opinion. Cheap USB flash drives just don't seem to have good controllers that handle both wear and error recovery as well as SATA SSD or NVMe do. Other than cost, their is no reason why a USB flash drive can't be "good". But, as someone else said here in the TrueNAS forums, USB flash drives were a race to the bottom in cost, sacrificing quality along the way.

In an another comment about USB enclosures, I seemed to have found a good one. It has both USB & eSATA for 3.5" disks, but it is easily 10 times the cost that most people are willing to pay;

RTX110-3Q - CRU

RTX110-3QOVERVIEWPRODUCT INFORMATIONRELATED PRODUCTSBUY PRODUCTS & ACCESSORIESStarting at $249.00BUY NOWTrayFree™ bays for fast drive swapsMultiple connection types including USB 3.0Rugged, all-metal constructionThe RTX100-3Q is a rugged and reliable desktop storage enclosure for data security...

www.cru-inc.com

I could afford it, and wanted something reliable for years.


One last comment, TrueNAS Core or SCALE is not the end all free NAS solution.

With it's exclusive use of ZFS for data storage, this can be a problem for some people wanting file system compatibility with MS-Windows or simple Linux FSes, (like EXT4). Other details like using a dedicated boot device or not being able to share a SSD / NVMe with multiple pools is a problem for some people.

 

[asap2go]

One last comment, TrueNAS Core or SCALE is not the end all free NAS solution.

With it's exclusive use of ZFS for data storage, this can be a problem for some people wanting file system compatibility with MS-Windows or simple Linux FSes, (like EXT4). Other details like using a dedicated boot device or not being able to share a SSD / NVMe with multiple pools is a problem for some people.

Plus it doesn't support (auto-)tiered storage and there is no write cache for short burst writing use cases.

 

[jgreco]

there is no write cache for short burst writing use cases.

Well, that's not true, obviously. ZFS's write cache is your system's main memory and it manages short burst write situations very well, at least if you have sufficient main memory. If you are looking for heavy sustained write caching, such as staging to flash before eventually writing to a slow HDD pool, then, no, ZFS doesn't have any special support for that. I expect that you meant the latter, especially since the unquoted part of your sentence is talking about auto storage tiering.

 

[Constantin]

… and nor should ZFS ever feature “fast” write-caches whose data are then automatically, gradually transferred to a slower pool the way some competing SOHO NAS’ do - see QNAP, ReadyNAS, or Synology, to name a few.

That approach can serve a function in a bursty, single-user environment just as some entry-level SSDs are known to use fast and slow flash to fool the user into thinking that they’re faster than the actually are. Gradually, the contents of the fast cache is transferred to the slower storage, just as SMR HDDs use a CMR sector to cache stuff until it’s time to write another SMR band.

The problem with such caches in the types of enterprise environments that iXsystems is targeting, is that the workload is continuous. Hence, the cache is going to be useless as once it’s full, it no longer can serve its function. Hence, those cheap SSDs I mentioned above and SMR HDD “feature” cratering performances under continuous loads.

Instead, iXsystems and ZFS in general focuses on using approaches that allow continuous performance improvement, ie ARC, L2ARC, SLOG (under the right use case), and sVDEVs. ARC and L2ARC benefit from a lot of memory. A sVDEV can dramatically speed up metadata and small file transactions in an otherwise HDD array. But what approach makes the most sense is quite dependent on the use case. That’s where this community can be really helpful thanks to its experience and willingness to share it.

 

[Ericloewe]

… and nor should ZFS ever feature “fast” write-caches whose data are then automatically, gradually transferred to a slower pool the way some competing SOHO NAS’ do - see QNAP, ReadyNAS, or Synology, to name a few.

I have to disagree with this one: There are valid use cases, in serious professional environments. That's not to say iX is targeting those or should target those, but not everything runs at full tilt 100% of the time.

 

[winnielinnie]

*looks around the room awkwardly*

So I do use USB drives as data pools, but only for offline, offsite backups. (It's a cheap, home-user alternative solution to something like tape drives or multiple NAS systems.)

They only ever get plugged in to replicate an (incremental) backup snapshot to them, then exported and stored somewhere safe.

I believe external USB drives do have their place in ZFS, as long as they remain narrow in scope. (This "offsite" backup solution has been working for me these past four years, and requires no expertise to handle.)

 

[asap2go]

Well, that's not true, obviously. ZFS's write cache is your system's main memory and it manages short burst write situations very well, at least if you have sufficient main memory. If you are looking for heavy sustained write caching, such as staging to flash before eventually writing to a slow HDD pool, then, no, ZFS doesn't have any special support for that. I expect that you meant the latter, especially since the unquoted part of your sentence is talking about auto storage tiering.

Yes. I meant the latter. Thanks for clarifying my poorly written comment.

 

[Constantin]

*looks around the room awkwardly*

So I do use USB drives as data pools, but only for offline, offsite backups. (It's a cheap, home-user alternative solution to something like tape drives or multiple NAS systems.)

Nothing awkward about that, I do the same thing with Mobius 5, for similar reasons.

I have to disagree with this one: There are valid use cases, in serious professional environments. That's not to say iX is targeting those or should target those, but not everything runs at full tilt 100% of the time.

No doubt, there are valid use cases, just as some folk can justify all-flash arrays while my NAS mostly plods along with HDDs. The big issue with cached systems is sudden, dramatic slow downs as the cache fills up. See the impact of SMR in TrueNAS systems while undergoing a resilver, for example.

Consider the development around sVDEV as a cautionary tale. Yes, iXsystems developed a solid setup implementation for sVDEV via the GUI, but many necessary planning and monitoring tools remain CLI-only and only really accessible to folk who hunt down the necessary information in the forums. iXsystems has yet to develop a GUI performance / fill dashboard box for sVDEVs even though the performance benefit of sVDEV is in part dependent on the admin ensuring enough buffer is left over for sVDEVs to do their magic re: small files and meta-data.

This should be a pretty simple task but so far, iXsystems seems to expect sysadmins to make use of the CLI to plan for sVDEV deployment and subsequently monitor if they got the sVDEV small file size threshold and metadata space allocation right. This is disappointing but I understand resources are limited. Tiered-speed storage would have an even bigger impact than sVDEV re: NAS performance, especially if the cache is filled. I wouldn’t want to deal with a tiered fast/slow storage solution without a complete set of GUI resources, starting with sVDEV as training wheels.

 

[jgreco]

The problem with such caches in the types of enterprise environments that iXsystems is targeting, is that the workload is continuous.

This isn't just enterprise environments. As a common example, NVR's encounter this type of workload as well, and there are many others too. In these cases, it's much better to design a pool that can suck down the workload at whatever rate is peak, rather than trying to bodge around it with fake write caching.

 

[jim99]

My experimentation with USB. I have used FreeNAS then SCALE for over a decade. This is a home environment not a corporate environment. For a mirrored boot pool I use small M.2 SATA drives plugged into an adapter and a USB external 2.5" case with a USB connection to the servers ports. This has given me zero problems. I did experiment with a 4 bay USB 3.0 external drive case plugged into a USB 3.0 server port. The 1 TB file copy went well then I was experimenting a bit and rebooting about every 10 minutes. On the 4th reboot I lost my data pool. On a different TrueNAS forum I read about the downside of using USB with a data pool and I experienced exactly what was discussed.