USB-3.1/USB-C multi-drive enclosure...

[rcfa]

I’ve read at various places that ZFS doesn’t play well with USB, as it’s designed for permanently attached drives, etc.
The details as to why were however always rather thin, from what I can find.

What is the issue: the removable nature of USB (which would equally apply to Thunderbolt), or fundamental flaws in USB itself?

Thing is, I know what kind of system I’d like to build eventually, but right now that’s not an option.

So the idea would be to attach something like this 8-bay USB-C drive enclosure permanently to the computer, i.e. not as removable drive, format the drives with ZFS, until I have the cash to build the kind of FreeNAS system I want, and then move the drives to that. (Even if I simply buy a FreeNAS XL+, the shipping to Europe costs more than this enclosure with shipping, i.e. just getting this and waiting until I’m in the US the next time, and bringing the FreeNAS XL+ back home myself would already pay for it)

So point being, not talking about attempting to use removable drives, but about using an enclosure like that, permanently attached to a computer.

Can someone enlighten me if in that particular use case USB is an issue and if so, how/why? I like answers, but more importantly I want to understand the issues that lead to an answer.

 

[sretalla]

the removable nature of USB (which would equally apply to Thunderbolt), or fundamental flaws in USB itself?

Both to an extent, although I would point more squarely at the quality of the controllers available on the marketplace (you could say that's a fundamental flaw of USB... being ubiquitous).

Although that is a neat looking unit and USB-C can certainly handle a lot more than 5Gbits/s supported by that unit (interesting question about how 8 disks at 6Gbits/s will work over that link in terms of command queueing), my experience with USB matches that of the forum, which is that although it may work for a (relatively short) while, things will end in disaster for your controller at some point far sooner than you wanted.

I have seen some posts (I think by jgreco) pointing to the way that ZFS batches instructions to the controllers which puts a much higher load on them than they are designed to sustain for the long-run.

 

[Constantin]

USB connections likely should be relegated to attaching keyboard and mouse and perhaps external storage for backup purposes. If something goes wrong, oh well, restart the backup. I don’t use any USB in any of my server disk I/O connections, because I’ve read too many posts lamenting this or that going wrong.

It’s the same reason that I’ve abandoned a SATA Port multiplier approach to hosting multiple drives for my laptop. Something invariably gets hosed and then the bus hangs. Instead, I hang them off individual USB ports, which is much easier to reset than SATA.

As for Thunderbolt, that is a different creature altogether. Depending on the implementation, the bus may be giving you USB C or PCI 3.0 x 4 speeds. I have found thunderbolt to be more reliable than USB but there are two issues that bother me. For one, the cables are very expensive AND can be very confusing (usb-c only, thunderbolt with different wattage allowances, active vs. passive termination, etc). Secondly, the connectors do not fill me with great confidence re long term use with the wattages that they are supposed to carry.

QNAP makes an interesting “DAS NAS” that offers thunderbolt connections in addition to Ethernet. The only downside is that the thunderbolt connection is thunderbolt over IP and the bus is dedicated. Thus, you can only have the NAS on that bus as the NAS will happily commandeer any attached devices (monitors, storage, etc) for its own use. So, if you want to go the QNAP route and have a thunderbolt monitor or other device, be sure to have a computer with multiple thunderbolt ports!!!

I am getting ready to put my XL rig on eBay... and I’d look at the oyen digital Mobius 5 instead. The mobius 5 likely is less expensive, provides better ventilation. However it’s only SATA and USB 3. I use those enclosures for Offsite backup storage. Granted, the RAID 5 jmMicro storage chip will likely trigger the shakes in @jgreco and others who have had to deal with it, but I consider it ok for backups. No issues with 10TB drives, solid, easy to use.

 

[Arwen]

USB's disk protocol was a bit primitive. Newer UASP is better, (USB Attached SCSI Protocol), as it then supports actual disk functions instead of minimal block storage. But, both sides have to support UASP, (host and disk).

Next, having a USB storage device tends to prevent SMART data access. So you are at the mercy of the enclosure & disks. Some USB chipsets do allow SMART access, but it's very chipset specific. Meaning you have to do your research to find out if that chipset supports SMART, and how to actually get the SMART data.

Last, bundling up 8 disks over a single link can be problematic. SAS links, (even a single Serial Attached SCSI link), are designed for this purpose. But, USB was never designed for continuous data transfers, (like what occurs during a Scrub, Re-Silver or ZFS Send / Receive). So using $50 heavy duty, shielded USB 3.1 Type C cable is NOT out of the question. The problem is FINDING a reliable source of such cables. Especially in the shortest length needed for the use.

So all in all, not a good idea.

Part of the problem is that when, (not if), things go badly, people with non-standard hardware tend to blame FreeNAS or ZFS. When it is really the non-server level of hardware being used as a NAS server.

All that said, it might work, and work for years.

 

[Yorick]

Consider that you can:
- Get a cheap HBA
- Get a cheap SAS extender
- stick the HBA into the FreeNAS box and the sas extender into the drive enclosure
- run breakout sas to SATA cables from your sas extender to your drives

There are some details such as “what kind of cabling do I need to power the extender”, and, that’s solvable.

Upside is you get something that is rock solid and oodles of FreeNAS users have in production - and it’s inexpensive to boot.