Why you should avoid USB attached drives for data pool disks

[Arwen]

Arwen submitted a new resource:

Why not use a USB attached drive for pool disks? - USB storage
Why you should avoid USB attached drives for data pool disks - USB storage

This subject has been coming up quite often in the last year or 2. Perhaps because of TrueNAS SCALE has brought forth some attention to TrueNAS in general and ZFS in specific.

Here is my take on why USB attached storage would make a poor selection for ZFS pool device:

• The connector is not latched, which may lead to inadvertent dis-connects.
• Multi-drive USB enclosures funnel all traffic through a single USB port, which may limit drive speed as they share bandwidth.
• USB...

Read more about this resource...

 

[Arwen]

Per suggestion from @HoneyBadger, I've updated the title to;

Why you should avoid USB attached drives for pool disks​

I knew the old title was not working, but I had a brain fart on why... (or perhaps it was sleep deprivation, which I am participating in at the moment).

 

[Heracles]

Hi,

I would also talk about power. The risk of too many power supplies, the excessive cabling, connection, heat and space required for that. The risk you mentioned for the USB connector will apply to the power connector as well. A USB port can only provide power to a certain level. No power redundancy. I see a lot of risks and limitations related to power.

Good text,

 

[Arwen]

@Heracles - Good points, I will update the list.

 

[ChrisRJ]

Just to avoid misunderstandings: What about adding a point that booting from USB (although not a cheap USB stick) is fine, whereas the resource talks about data drives.

 

[Arwen]

Just to avoid misunderstandings: What about adding a point that booting from USB (although not a cheap USB stick) is fine, whereas the resource talks about data drives.

I do have a point at the very bottom giving qualifications on when USB storage does work:

Occasionally USB has to be used as the boot device connection method, (or better yet, boot Mirror).

But, it does make sense to have something earlier on, so I added this as the second paragraph:

Please note that this is about USB attached storage for ZFS data pools. On occasion, people use USB drives, (flash, USB->SATA, USB->NVMe, etc...), for booting. In general, these do work acceptably, (when not using the cheapest USB flash drives). Though, it is always recommended to backup your configuration in case of boot drive failure.

 

[sretalla]

You could probably also link to this:

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

As those sort of cheap SATA controllers/port multipliers also tend to be involved with the multi-drive enclosures.

 

[Arwen]

You could probably also link to this:

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

As those sort of cheap SATA controllers/port multipliers also tend to be involved with the multi-drive enclosures.

I've add that reference at the bottom of the resource, as a further reading item.

 

[rvassar]

Backup disks attached via USB for temporary connection can be used successfully.

Probably needs a reason why this is acceptable, ala... "A failed backup can often be repeated without data loss. Coupled with planning and best practices, ZFS snapshots can preserve the previous backup, limiting the loss if it can't be repeated."

The other missing item is USB power behavior. You hit 500ma on a standard 2.0 type A plug and it simply turns off. So if you use a hub, and mix powered and un-powered devices, it's possible for the hub to exceed 500ma and your disks get disconnected despite having their own power source.

 

[Arwen]

Both items added, thank you @rvassar.

 

[AlexGG]

another point you may want to add:

If the drive has a bad block, SATA will report the bad block and, as a rule, carry on. The USB-to-SATA converter will often lock up (when the SATA drive hits a bad block) and require a hardware reset (unplug-replug or power cycle) to recover. If USB hubs are used, sometimes the entire hub will lock up, thus escalating a single bad block to a complete failure of multiple drives.

To paraphrase, SATA/SAS problems tend to stay limited and isolated to a faulty drive, but if the same drive is connected via USB-to-SATA adapter, these problems tend to increase in severity and ripple all over the bus.

 

[WI_Hedgehog]

I think the writeup is excellent.

I noticed it doesn't address: USB flash memory generally has a very limited number of write cycles. When used as intermittent, temporary storage for transporting files it's not typically an issue as that's what the device was designed for, however it was most likely not designed for use as a full-time continual-access file system. The on-chip heat generated by using a fast USB port & drive combination is pretty hard on the flash memory.

USB thumb drives typically have limited (if any) set-aside (reserved) memory for replacing worn-out or bad blocks, whereas SSD has at least "some," and in Enterprise drives "quite a bit."

SSD drives tend to be larger than USB drives which helps with wear-leveling, though this is a generic statement.

In some cases, such as a hand-held media server, flash drives make sense as files are usually WORM (Journey - Don't Stop Believin' probably doesn't get updated often, and if it does there's an issue....) and the form factor is extremely tiny.

The article might also list some alternatives and why to use them. For instance, the micro media server I mentioned as of recent has an alternative now that NVMe slots are available.

Great and very thorough writeup, I enjoyed reading it.

 

[rvassar]

I noticed it doesn't address: USB flash memory generally has a very limited number of write cycles. When used as intermittent, temporary storage for transporting files it's not typically an issue as that's what the device was designed for, however it was most likely not designed for use as a full-time continual-access file system. The on-chip heat generated by using a fast USB port & drive combination is pretty hard on the flash memory.

USB thumb drives typically have limited (if any) set-aside (reserved) memory for replacing worn-out or bad blocks, whereas SSD has at least "some," and in Enterprise drives "quite a bit."

SSD drives tend to be larger than USB drives which helps with wear-leveling, though this is a generic statement.

In some cases, such as a hand-held media server, flash drives make sense as files are usually WORM (Journey - Don't Stop Believin' probably doesn't get updated often, and if it does there's an issue....) and the form factor is extremely tiny.

This is highly variable these days, and it's not just USB flash. There is a huge spread between retail/consumer and enterprise. Even within the enterprise segment, we now have "read optimized" drives that have lower write performance, and higher performance read. The write "wear" comes down to this: The storage is built from linked lists of memory cells. These cells get "erased" and written into service with pointers set to determine what LBA they represent. When that LBA gets re-written, a newly recycled cell gets the new data written to it, and the pointer for the LBA assigned to it. The old cell gets unlinked and sent to the garbage collection heap, where it gets cleared by writing all 1's to it, and then dropped back in the free pool, unassigned. If it helps, think of it as "virtualized storage". The problem is the writing wears down the cells, and some times they don't accept the new data or the post use erase. When this happens they get tossed in the "bad" pool, and the free pool shrinks. The key concept here is the free pool starts out much larger than the advertised capacity of the drive. Some of the devices I'm familiar with are as much as 50% bigger than advertised capacity to accommodate years of service in high write-duty service. This fraction of capacity that's hidden is often referred to as "over-subscription", and it's size is determined to create a lifespan that is a somewhat linear function between cell erase failure, and consumption/burn-down of cell over-subscription. When you run out of spare oversubscribed cells, the device is toast.

This leaves a lot of engineering compromise between the $9 USB thumb drive you picked up in the grocery store checkout line, and the $7000 Enterprise NVMe drive Intel/Kioxia/Samsung certifies and re-sells thru the major system vendors.

 

[jgreco]

@Arwen , as you know I do really like your articles. That said, I have a minor request for a recommended improvement. You say

Multi-drive USB enclosures funnel all traffic through a single USB port, which may limit drive speed as they share bandwidth.

Which is absolutely correct, but I really think that this either needs to be explained more fully or perhaps a second bullet point.

When discussing slow storage topologies such as FC with people, I have often pointed out that while 2Gbps or 4Gbps FC for an ESXi cluster to store VM's is just fine because there are usually not crushing levels of I/O towards the array. When used as a ZFS disk shelf, however, FC (or iSCSI or USB) are problematic because I/O intensive processes such as scrubs or resilvers will present a crushing amount of I/O over that thin, low bandwidth pipe. A single modern SATA HDD is capable of almost 3Gbps and a single modern SATA SSD can peg 5.5Gbps easily; trying to run a 24-bay shelf over 16Gbps FC results in a massive oversubscription, and most people try "cheaper" options anyways. However, this also comes into play with external USB enclosures such as the one in this post.

advTHANKSance for considering my suggestion,

 

[Arwen]

@jgreco - Yes, some of the bullet points did not hit the target quite completely. How about this;

Multi-drive USB enclosures funnel all traffic through a single USB port, which may limit drive speed as they share bandwidth. This becomes especially problematic during scrubs or disk replacements. Worse, if there is inadequate cooling.

I've also added this initial paragraph;

Some of the points also apply to Thunderbolt enclosures. But, Thunderbolt might even be worse because the drivers would be even less tested than USB on TrueNAS Core or SCALE.

 

[jgreco]

You're not as wordy as I am.

 

[Arwen]

That is because I am a Professional Writer! Concise wording is the goal.

 

[jgreco]

Concise wording is the goal.

It might not have been a compliment. I usually worry that people won't focus on the thing unless you kinda hit them over the head with it a bit.

 

[Arwen]

I, on the other hand, have seen people stop if the paragraph is too long. Or the point is not made soon enough.

 

[jgreco]

I, on the other hand, have seen people stop if the paragraph is too long. Or the point is not made soon enough.

If someone is on a technical forum and they are unwilling to read, they're an idiot.