D720201 based USB3.0 works under FreeNAS 9.2.1.5?

[DaPlumber]

Before I muck around with installing one of these little PCIe beasties, can anyone confirm that this will, in fact add usable USB 3.0 speed ports under FreeNAS 9.2.1.5? I.e. there is solid driver support?

 

[cyberjock]

I have no clue if it is supported or not, but whatever you have planned for USB3 is likely to not work out well. USB shouldn't be used for drives that will be used to store data that is important(aka your pool(s)) and your USB key, even if USB3, will not appreciably affect bootup times. This is why it was considered okay to disable it in FreeNAS.

 

[DaPlumber]

I beg to differ. I have used USB connected disks mostly for backup and transport under Solaris, OpenSolaris and OpenIndiana starting with some of the earliest versions of ZFS. Within the limitations of USB bandwidth, latency, and OS hot-plug support I've never had a problem. My usage case here is a pair of drives with a mirrored zpool for ZFS send/receive backup/transfer. It's been working just fine, but I know the drives can go a bit faster than the current ~20MiB/s (mostly) limited by USB 2.0. The drives are obviously in USB 3.0 capable enclosures, tested elsewhere. While snapshot send/receive is not everyone's choice for backup and has issues (especially with a lack of restart), it's by far the easiest way to backup ZFS metadata painlessly. I'm doing this at the CLI because of familiarity, I suspect it would be a PITA in the GUI, if even possible. This is also about as sequential streaming scenario to the disks as you can get with ZFS so random access performance doesn't enter into it. eSATA is not an option here.

I'm using FreeNAS 2.1.5 booting off a SATA SSD stick so I could care less about USB booting.

That out of the way, a little FreeBSD digging seems to indicate anecdotal support for this chipset. However I'm a FreeNAS noob, so I'm a little concerned about why you said USB 3.0 support is turned off? Did you mean only for boot or kernel runtime as well? Care to elaborate or point to some documentation/threads? My Google-Fu seems to be failing me here. ;-)

 

[DaPlumber]

Murphy ya b'stard... Of course I find the 9.2.0 release notes after posting the above:

"USB 3.0 support is disabled by default as it currently is not compatible
with some hardware, including Haswell (Lynx point) chipsets. To enable
USB 3.0 support, create a Tunable named xhci_load, set its value to YES,
and reboot the system."

 

[cyberjock]

I beg to differ. I have used USB connected disks mostly for backup and transport under Solaris, OpenSolaris and OpenIndiana starting with some of the earliest versions of ZFS. Within the limitations of USB bandwidth, latency, and OS hot-plug support I've never had a problem. My usage case here is a pair of drives with a mirrored zpool for ZFS send/receive backup/transfer. It's been working just fine, but I know the drives can go a bit faster than the current ~20MiB/s (mostly) limited by USB 2.0. The drives are obviously in USB 3.0 capable enclosures, tested elsewhere. While snapshot send/receive is not everyone's choice for backup and has issues (especially with a lack of restart), it's by far the easiest way to backup ZFS metadata painlessly. I'm doing this at the CLI because of familiarity, I suspect it would be a PITA in the GUI, if even possible. This is also about as sequential streaming scenario to the disks as you can get with ZFS so random access performance doesn't enter into it. eSATA is not an option here.

Ok, still doesn't change the fact that:

1. The underlying technology just isn't designed for things that most file server afficionado's would classify as 'reliable'.
2. That USB has cause so much data loss that your personal experience is trumped by the dozens of lost pools that are directly attributable to USB.

Also, do not do from the CLI what can be done from the WebGUI. If the WebGUI supports it, you'd better do it in the WebGUI. FreeNAS is an appliance. If you are not happy with these limitations FreeNAS may not be for you.

 

[DaPlumber]

Ok, still doesn't change the fact that:

1. The underlying technology just isn't designed for things that most file server afficionado's would classify as 'reliable'.
2. That USB has cause so much data loss that your personal experience is trumped by the dozens of lost pools that are directly attributable to USB.

Also, do not do from the CLI what can be done from the WebGUI. If the WebGUI supports it, you'd better do it in the WebGUI. FreeNAS is an appliance. If you are not happy with these limitations FreeNAS may not be for you.

Having read some of your other posts here I suspect you and I have a lot in common, so please don't take this the wrong way :p : I have seen exactly the same accusation of reliability leveled at eSATA, SATA, SAS-II, SAS, FWD SCSI, Wide SCSI, and original SE "narrow" SCSI way back in the day vs various proprietary channel architectures. I've seen enough USB errors and the inherent hot-plug nature of USB would preclude me from considering it a "production" connectivity solution. The FreeBSD USB 3.0 support seems to still have that new code smell as well... :) That's the great thing about ZFS, it was designed with unreliable hardware in mind. I've had pools with disks that kept going on and offline like Yo-yos, intermittent bursts of errors, and the usual laundry lists of failures. I've never lost data with correctly configured ZFS pools ranging from small personal pools to large "enterprise production" pools. The key words there were "correctly configured". Your and other's advice in the FAQ and Wiki are pitch perfect no matter how the drives are connected. Anecdotes are not a trend, but I had a system where a (beer budget, non-enterprise class) hardware problem was yielding errors on two SATA ports. Until the card could be replaced I moved the two drives to USB enclosures. ZFS didn't care, although the unbalanced I/O profile worked about as well as you'd expect. As a short term measure it was great and something that would have given any other volume manager a fit! It allowed the pool to stay online and accessible albeit with degraded performance.

I do have a bit of a pet peeve with GELI as I feel it masks the dumb hardware from ZFS. I'll wait on encryption until it can be done the same way as compression and de-dup like in the non-OSS Oracle ZFS code. I'll live with encrypting at the file level until then.

tl;dr USB used with caution and aware of the "Big Boy" label works great in some circumstances.

You're preaching to the choir about letting an appliance do it's job. If I gave the wrong impression I apologize. Pool creation and export, filesystem creation, snapshot initiation and management were all done with the GUI. The only part that I did CLI was the actual "zfs send /pool/fs@snapshot | zfs receive /pool2/fs". You could probably kludge that in the GUI using 127.0.0.1 as the destination address? Since that's not really what the replication part of the GUI is designed for as far as I can see, that seemed like an acceptable tradeoff. Is local copying between pools already been put in an enhancement request?

When I get the chance to try this card out and enable USB 3.0 I'll post here, I was just hoping someone else who had already done so could share. "A wise man learns from others mistakes, a fool learns from his own." :D

 

[cyberjock]

Yeah, the evidence against USB is not anecdotal at all. We've literally seen many pools fail because:

1. SMART isn't always supported, which is a major indicator of a failing disks that you can never substitute.
2. USB masks disk errors.
3. On a disk error where the disk goes into "recovery mode" will cause the disk to disconnect from the system. This literally means that on a single sector error you could lose the disk. Not a good thing if you are actually looking for uptime and reliability.

USB 3.0 kind of sucks in FreeBSD, but that's a different problem as that is expected to be fixed someday via drivers.

I am that wise man and I have learned from watching and reading. USB is just a fail waiting to happen. Even as backups it has proven to have it's own problems. I mean, come on, who wants a backup that you can't validate with SMART, masks its errors, and drops the disk on a single disk error!? ;)

So when I say that the underlying technology isn't equipped for this tasks, that's 100% true and there's tons of evidence to support it. Yes, ZFS was designed to handle unreliable media. It was not designed to handle unreliable media that will drop out of your system on a single disk error, be incapable of identifying read or write errors, and has the potential for out of sequence writing to your ZFS metadata because USB tries to optimize itself.

Edit: I do have an encrypted pool. I chose to be a guinea pig and test it for myself. It's come with some problems, we've had some major bugs in the past with it, we've had a few people lose pools because they did silly things. I do agree that the geli layer masks some problems. However I haven't yet seen anyone have problems that came up with ZFS that weren't also in SMART, so I tend to think that SMART is a safe "backup" for the abstraction of the geli layer.

 

[DaPlumber]

Yeah, the evidence against USB is not anecdotal at all. We've literally seen many pools fail because:

1. SMART isn't always supported, which is a major indicator of a failing disks that you can never substitute.
2. USB masks disk errors.
3. On a disk error where the disk goes into "recovery mode" will cause the disk to disconnect from the system. This literally means that on a single sector error you could lose the disk. Not a good thing if you are actually looking for uptime and reliability.

USB 3.0 kind of sucks in FreeBSD, but that's a different problem as that is expected to be fixed someday via drivers.

I am that wise man and I have learned from watching and reading. USB is just a fail waiting to happen. Even as backups it has proven to have it's own problems. I mean, come on, who wants a backup that you can't validate with SMART, masks its errors, and drops the disk on a single disk error!? ;)

Hmmm, I seem to have higher standards than most then. I'm very picky about my USB enclosures, and S.M.A.R.T. support is a "must" for me as is USB controllers that don't try and be too clever about re-arranging my I/O. With the exception of S.M.A.R.T. and unlike most every other technology used in NAS, ZFS likes dumb devices, the dumber, the better. :D I'm also primarily a Mac user on my lap and desktop at home, so all those clever "vendor-optimized" drivers tend to not be an available temptation.

So when I say that the underlying technology isn't equipped for this tasks, that's 100% true and there's tons of evidence to support it. Yes, ZFS was designed to handle unreliable media. It was not designed to handle unreliable media that will drop out of your system on a single disk error, be incapable of identifying read or write errors, and has the potential for out of sequence writing to your ZFS metadata because USB tries to optimize itself.

Yea verily, "Big Boy" Label as mentioned before. :p It's the eternal Cost/Likelyhood/Complexity tradeoff. In this case because I'm not serving from the USB pool and because I'm careful in my choice of device that doesn't suffer from either of those issues I'm good with it. Any failures will be an annoying stop, troubleshoot, replace if needed, and restart cycle rather than the hairball that results when a NAS share goes down, or worse, gets potentially corrupted.

Edit: I do have an encrypted pool. I chose to be a guinea pig and test it for myself. It's come with some problems, we've had some major bugs in the past with it, we've had a few people lose pools because they did silly things. I do agree that the geli layer masks some problems. However I haven't yet seen anyone have problems that came up with ZFS that weren't also in SMART, so I tend to think that SMART is a safe "backup" for the abstraction of the geli layer.

The irony here is that I got into this because I'm in the process of standing up one of iX Systems great FreeNAS mini's (that you recently reviewed most excellently) for Home and Home Office use. It's my first FreeNAS system and I changed my mind about GELI encryption after seeing how it handled a few errors. I copied the ZFS filesystems to the USB pool so I could redo the primary zpool without having to load all the data back in I had already consolidated. Just for grins and giggles I re-installed via the IPMI Java craplet and it's CDRom .iso redirection. Indeed, as you pointed out, not a feature I'm used to having on "consumer" type gear! If I ever have any unresolvable issues with the USB backup pool my fallback will be to attach some SATA to eSATA headers onto some of the spare SATA ports and see if I can find some reasonably priced eSATA enclosures. Sigh. AFP/Netatalk has worked great for me so far, bar missing PERL to run the "macusers" command. (Logged as Bug #5077)

 

[cyberjock]

One thing to keep in mind. If your USB devices work with smartctl -a /dev/daX then it will work with the SMART that's built into FreeNAS' WebGUI. If you have to add parameters(I think I had to do -sat auto or something for one of mine) it is not able to be monitored by FreeNAS. Consider it a limitation of using the appliance. :(

 

[DaPlumber]

One thing to keep in mind. If your USB devices work with smartctl -a /dev/daX then it will work with the SMART that's built into FreeNAS' WebGUI. If you have to add parameters(I think I had to do -sat auto or something for one of mine) it is not able to be monitored by FreeNAS. Consider it a limitation of using the appliance. :(

Bugger. That explains why they're not showing up on the GUI. Yes you have to specify the USB chipset to smartctl more often than not.

 

[DaPlumber]

tl;dr Don't try this at home kids. The card works and is showing up as USB 3.0 with two devices I happened to have handy after setting the tunable. As per the README DO NOT do this with a Haswell chip! I am now getting 40-60MiB/s on the disks instead of a flat 20MiB/s, so I think it was worth it.

Note: This worked for me with my hardware. Per the discussion with cyberjock above USB pools are not a recommended practice for all the reasons outlined. I am a Big Boy and understand the risks I'm accepting and will not come whining when something blows up. I know we're running USB 3.0 because usbconfig on the CLI says so, but there is no way to check for this other that performance from the GUI. All ZFS operations were performed from the GUI except for the "zfs send | zfs receive" part. That said, if you're familiar with creating ZFS pools on USB devices in other environments: Come on in, The FreeNAS water's fine! This has had NOTHING to do with USB boot devices, don't go there.

 

[DaPlumber]

One thing to keep in mind. If your USB devices work with smartctl -a /dev/daX then it will work with the SMART that's built into FreeNAS' WebGUI. If you have to add parameters(I think I had to do -sat auto or something for one of mine) it is not able to be monitored by FreeNAS. Consider it a limitation of using the appliance. :(

Hmmm, and yet the "View Disks" tab has a column for "S.M.A.R.T. extra options"? Enquiring minds want to know...

 

[cyberjock]

Yeah, won't let you do what you want unfortunately because it adds those commands after the "smartctl -a /dev/daXX". That's why I had to submit code to make Areca controllers work.

 

[DaPlumber]

"Sneaky little hobbitses. Wicked, tricksy, false!" ;)