Solving "UFS Degraded" issue on reboot with External USB Hard Drives

[kriswithak]

This seems to be a pretty common problem, and not something you can find a very distinct write-up on. So I created one. If the drive is healthy you should not lose any data with this method, but I can't guarantee that.

In my case, I have all my SATA and IDE ports used up in my FreeNAS, so I am limited to using a USB external for an additional redundant backup. This volume is a Seagate Desktop Backup+ 4tb, it is of the SCSI-6 interface type.

After any shutdown or reboot, this backup drive becomes inoperable and "DEGRADED".
Curiously, a SMART test in shell reveals a OK health status.
Really, it is a matter of FreeNAS not liking USB drives, the database probably gets out of sync.

Important Considerations:

• You shouldn't use a USB external unless the data you are storing there is not essential. [In my case, they are just redundant Rsync backups of our MyDocuments folders]
• It is always a good idea to perform a SMART test, even if you are sure the drive is healthy. Never hurts, takes seconds. Just figure out what BASH string command works for your device. In my case, I need to use: 'smartctl -a -d scsi /dev/da1'. the -d scsi was needed, usb would not work (seagate desktop+ is SCSI-6). Look up your Device info
• Utilizing a ZFS volume for USB, if possible, is probably a better way of handling this. Auto-Import works a bit smoother, and zpool export/import is nice.

Solution: Detach/Reattach Volume (without losing data)
A more in-depth write-up is located here.
Follow these steps:

unplug the USB from your external Hard Drive, wait a few seconds for device removal on FreeNAS
if you have any shares configured for the drive, you'll have to remove them. Any relevant RSync tasks or other tasks can be left alone.
go to 'Storage', and choose "Detach Volume" for the drive affected. Be sure to leave existing data.
Replug the USB to the drive, Refresh after succesful detachment, go back to 'Storage' and click on "Import Volume"
Select the volume you wish to import, select the type [for my case, UFS], Name it, and click OK
Reconfigure share points, if any
And success!!! The External USB drive is imported without Data loss.

Unfortunately this method will probably have to be ran after every shutdown. I might even create a cron job to do this sometime, if I take the time to figure out all the BASH commands for Shell.

Cheers! Let's hope this saves some headache for another helpless soul.

 

[cyberjock]

I can't say I recommend this guide. Its documented in the manual to never unplug a hard drive without informing the system that you intend to detach the drive using the "detach" button of the GUI. In your case, you aren't doing that, and can suffer permanent data loss. Even aside from that, unplugging a zpool that is mounted and then detaching it could render your pool unmountable the next time you try to use it(aka you lose all data on the pool).

Also, I'm not sure what kind of SMART test you are running that is "taking seconds", but a long test(the most thorough of the SMART tests) takes upwards of 400 minutes on some hard drives.

For the record, its probably not about FreeNAS(or FreeBSD) not liking USB as it is that USB is really not that stable for long term high speed data transfers. It's an issue that affects every operating system(I have as well as friends have had problems with Windows and Linux) and its an inherent weakness of using USB for storing data. I refuse to buy, use, or recommend external USB drive that aren't flash based because spinning rust drives become worthless when they start having any kind of problems. I don't like recommending some Major Name Brand of external hard drive for backups that starts having problems and disconnecting and reconnecting itself during backups rendering the backups invalid. Not my idea of "trust me.. I know what I'm doing". I've seen where a single bad sector rendered the USB drive unreliable for storing data.

 

[kriswithak]

I can't say I recommend this guide.

Hello Cyberjock! :D
This is not a surprise to me, I came across a few of your responses on this very subject, and you seemed hard pressed to keep anyone from using external USB's.

Your reasoning is sound, however in some cases (like mine) the integrity of the drive is not important. It is merely used as a redundant Rsync backup of work files, similar to a shadowcopy configuration. It is not a primary backup. We have numerous full-system backups elsewhere, and this particular USB-drive just syncs work files to two locations at different intervals. This allows users to quickly access a mapped network share of their work files from an hour ago, and from the previous day. It functions supremely for us as a quick "oops" backup, accessible (mapped drive in windows explorer) and quick (over gigabit and with a decent nas box, our performance and memory usage reports are always low). Integrity isn't important because we have primary mirrored-backups each day. If the drive were to ever fail, there really is no harm done, the real backups are all there (SATA drives), and there is just a little downtime until a full sync on a new drive.

I feel it's important to realize that some people may actually want to use a similar setup, whether the cause is a lack of available SATA ports or what-have-you. I saw a need for a write up based on the number of instances I saw others asking the same questions. That being said, your input on the integrity of the drives is still appreciated, though I had hoped I had covered some of that in my OP.

Its documented in the manual to never unplug a hard drive without informing the system that you intend to detach the drive using the "detach" button of the GUI. In your case, you aren't doing that, and can suffer permanent data loss. Even aside from that, unplugging a zpool that is mounted and then detaching it could render your pool unmountable the next time you try to use it(aka you lose all data on the pool).

You are correct. However: I needed to do this as a work-around. I mentioned it more in the linked article, but with the volume degraded, the import feature does not show the disk in the dropdown list. This may be a bug, but I wondered if I needed to unmount the volume, so I attempted to do this via SHELL. The bash commands failed due to the location not being found... curious when I can see it in the GUI. The only way I was able to get it to work smoothly was to detach. And I found the only way which was possible (without another "can not be found" error during GUI detachment) was to unplug the usb first, then detach.

It might be good to note that my drive has an external PSU too, so while operations on the disk will cease, it still retains power.

As for the possibility of data loss, refer again to my statements about integrity of this particular drive not being a concern of mine. Even if it was, this isn't an operation that should need to be done at any regularity after the system is 100% configured to your liking, save for power failure. We utilize a UPS, so shutdown is only initiated manually, and rarely. But drive failure would not be a catastrophe; barely a nuisance.

Also, I'm not sure what kind of SMART test you are running that is "taking seconds", but a long test(the most thorough of the SMART tests) takes upwards of 400 minutes on some hard drives.

Using the bash command I mentioned, it takes less than a minute for me. My drive has SMART built in, so I use -a, which is a self test.

For the record, its probably not about FreeNAS(or FreeBSD) not liking USB as it is that USB is really not that stable for long term high speed data transfers. It's an issue that affects every operating system(I have as well as friends have had problems with Windows and Linux) and its an inherent weakness of using USB for storing data. I refuse to buy, use, or recommend external USB drive that aren't flash based because spinning rust drives become worthless when they start having any kind of problems. I don't like recommending some Major Name Brand of external hard drive for backups that starts having problems and disconnecting and reconnecting itself during backups rendering the backups invalid. Not my idea of "trust me.. I know what I'm doing". I've seen where a single bad sector rendered the USB drive unreliable for storing data.

All good points. I have used some form of external USB as a shared drive on Windows for years - and I've seen my fair share of problems with it. However, I've also had good experiences with good set-ups, and again, I'm not using the external USB for our primary backups. This is just an auxiliary backup feature, and we survive without it. Someone else out there might be in a situation where they don't care for the integrity either. If you use an USB external as a primary backup, you are asking for problems. Luckily for me that isn't the case.

tl;dr: Your points are valid, but since when does the human race do everything "by-the-book"? Your points are appreciated and true, yet lets not stop human ingenuity from working outside of the rules a bit. As long as one is aware of these concerns, and deem them of no consequence, I say have at it!

 

[cyberjock]

Hello Cyberjock! :D
This is not a surprise to me, I came across a few of your responses on this very subject, and you seemed hard pressed to keep anyone from using external USB's.

Your reasoning is sound, however in some cases (like mine) the integrity of the drive is not important. It is merely used as a redundant Rsync backup of work files, similar to a shadowcopy configuration. It is not a primary backup. We have numerous full-system backups elsewhere, and this particular USB-drive just syncs work files to two locations at different intervals. This allows users to quickly access a mapped network share of their work files from an hour ago, and from the previous day. It functions supremely for us as a quick "oops" backup, accessible (mapped drive in windows explorer) and quick (over gigabit and with a decent nas box, our performance and memory usage reports are always low). Integrity isn't important because we have primary mirrored-backups each day. If the drive were to ever fail, there really is no harm done, the real backups are all there (SATA drives), and there is just a little downtime until a full sync on a new drive.

Then why even bother with ZFS? Just use UFS, or don't do the backup at all. I view it as do it right or not at all. And I'd much rather do hot swap SATA bays than USB any day. There's really no easy, effective, and safe way to use USB with FreeNAS. It just wasn't an "expected" need. Doing zpool scrubs at an amazingly fast 30MB/sec over USB is a major drag when you have 2TB+ of data. Too many people have thought they had good backups on USB drives with ZFS, only to find they couldn't recover their data later. So what was the point? That's all I'm saying. :P

You are correct. However: I needed to do this as a work-around. I mentioned it more in the linked article, but with the volume degraded, the import feature does not show the disk in the dropdown list. This may be a bug, but I wondered if I needed to unmount the volume, so I attempted to do this via SHELL. The bash commands failed due to the location not being found... curious when I can see it in the GUI. The only way I was able to get it to work smoothly was to detach. And I found the only way which was possible (without another "can not be found" error during GUI detachment) was to unplug the usb first, then detach.

Yeah, I'm not sure about that. I had no problem auto-importing a degraded zpool before. It may be related to USB, or something else. It might be worth putting in a ticket at support.freenas.org and provide step-by-step instructions on how to reproduce your problem.

It might be good to note that my drive has an external PSU too, so while operations on the disk will cease, it still retains power.

That really doesn't matter much. The biggest problem with writing to USB is if the drive drops out of the USB root hub its attached to. Usually USB commands are cached to increase performance. But if a USB drive encounters all but the most minor of problems, the drive might disconnect and reconnect. The second it disconnects all the data in cache is discarded because you have no guarantee if that drive will reattach, if it will be the same device, etc. So you just lost data and you may not get a warning at all. If you get any warning it will probably be on the local console only. I don't know about you, but I never look at it.

Using the bash command I mentioned, it takes less than a minute for me. My drive has SMART built in, so I use -a, which is a self test.

-a doesn't run any tests at all. -a "Prints all SMART information about the disk...". Nothing more. If you want a short test(hint: these are worthless) you use -t short. If you want a long test you'd add -t long.

Your points are valid, but since when does the human race do everything "by-the-book"? Your points are appreciated and true, yet lets not stop human ingenuity from working outside of the rules a bit.[/B] As long as one is aware of these concerns, and deem them of no consequence, I say have at it!

Well, the problem (especially so in this forum) is that people don't do thing "by-the-book", but have no understanding of what is in "the-book" and then cry when they lose their data. It happens so frequently if I got $100 for each person that's lost everything I'd be a millionaire. Everyone in IT thinks they are the exception and not the rule. I'd bet I can find 5 threads just this month of people trying to use USB and having issues with storing or recovering their data.

Also, so few people build, maintain, and administer file servers its almost a specialty. But people think its just a desktop with Samba, and its not. It requires careful planning and thought to make it all work. FreeBSD has an extremely steep learning curve. It's not for a noob to understand in a week. FreeNAS tries to fill that gap between needing to be on the better side of that months-long(or years-long), but there are consequences for that. You gotta do what the manual says if you want your data to be safely stored.

 

[kriswithak]

Then why even bother with ZFS? Just use UFS, or don't do the backup at all. I view it as do it right or not at all. And I'd much rather do hot swap SATA bays than USB any day.

Ok... are you reading my posts?

I'm not using ZFS... I am using UFS. I suggested that ZFS might work a little better due to auto-import and zpool export/import, is all.

I don't have extra SATA bays available, they are all used by the primary backups. For the umpteenth time, this is not used for a primary backup.

Too many people have thought they had good backups on USB drives with ZFS, only to find they couldn't recover their data later. So what was the point? That's all I'm saying. :P

Well, for one, this isn't a backup. I consider it merely a "shadowcopy" using rsync. We have our backups on mirrored SATA drives. Once again, this system is Auxiliary. And it's UFS.

"The point" is using this system we were able to add an additional redundant solution. The primary backups are on mirrored SATA drives, as the mobo is maxed out on SATA, adding a USB external allowed us to add this shadowcopy feature. This isn't our fail-safe solution, the SATA drives are. This, again, is auxiliary. Someone else you communicated with on another thread mentioned they used USB externals so they could unplug and place directly in a fire-safe. There are plenty of legitimate needs for a simple write up for this.

Yeah, I'm not sure about that. I had no problem auto-importing a degraded zpool before. It may be related to USB, or something else. It might be worth putting in a ticket at support.freenas.org and provide step-by-step instructions on how to reproduce your problem.

Auto-import isn't available for UFS, to the best of my knowledge. Hence my method.

That really doesn't matter much. The biggest problem with writing to USB is if the drive drops out of the USB root hub its attached to.

No, you are missing a key issue here: The volume status is degraded. The volume is virtually disabled and any associated shares are disabled. FreeNAS lists it as "Error: can not retrieve available space", and even trying to access it via shell returns "not found" errors. I see no indication that during this circumstance anything is ever being written to the drives... So that concern is also not a concern here. It retains power, and USB is disconnected while no read/write operations are being performed. I see no issue here.

Usually USB commands are cached to increase performance. But if a USB drive encounters all but the most minor of problems, the drive might disconnect and reconnect. The second it disconnects all the data in cache is discarded because you have no guarantee if that drive will reattach, if it will be the same device, etc. So you just lost data and you may not get a warning at all. If you get any warning it will probably be on the local console only. I don't know about you, but I never look at it.

Nope. Again, the situation I'm solving happens immediately after a shutdown or reboot. After booting, the drive is virtually non-existent, as I explained. After boot, the drive is never read from or written to by FreeNAS. It sees it as an unavailable USB device. The cache is already empty in this situation. No succesful read/write operations are performed with after the reboot until after my reattaching process.

-a doesn't run any tests at all.

It retrieves the most recent results.

-a, --all
Prints all SMART information about the disk, or TapeAlert information about the tape drive or changer.
and for SCSI, this is equivalent to
'-H -i -A -l error -l selftest'.
...
-H, --health
Check: Ask the device to report its SMART health status or pending TapeAlert messages.
If the device reports failing health status, this means either that the device has already failed, or that it is predicting its own failure within the next 24 hours. If this happens, use the '-a' option to get more information, and get your data off the disk and to someplace safe as soon as you can.
...
-l selftest
Any errors detected in the self testing will be shown in the SMART self-test log, which can be examined using the '-l selftest' option.
...
The third category of testing (and the only category for which the word 'testing' is really an appropriate choice) is "self" testing. This third type of test is only performed (immediately) when a command to run it is issued. The '-t' and '-X' options can be used to carry out and abort such self-tests; please see below for further details.
...

Note:The words "Self-test" are used in connection with the third category.

I have -O set for automatic SMART selftesting, every 4 hours. I should probably mention that.

Well, the problem (especially so in this forum) is that people don't do thing "by-the-book", but have no understanding of what is in "the-book" and then cry when they lose their data. It happens so frequently if I got $100 for each person that's lost everything I'd be a millionaire. Everyone in IT thinks they are the exception and not the rule. I'd bet I can find 5 threads just this month of people trying to use USB and having issues with storing or recovering their data.

Yea, and most of the threads I've read are people that just can't remove the "DEGRADED" status, or re-import the volume, which this solved for me. Again, I don't recommend relying on this for a primary backup. I'm not using it as a primary backup. This drive could die tomorrow and I wouldn't let out so much as a sigh. It is not our primary backup, or a backup at all really. You're failing to realize this... not to mention the example earlier of someone backing up their primary back ups on a external USB occasionally, to physically place in a fire safe. There are legitimate needs for this, whether you personally feel so or not.

You gotta do what the manual says if you want your data to be safely stored.

Safety of this drive is of no concern.

Also, you said yourself:

I have found a couple of very big deal mistakes in the latest FreeNAS manual. For instance, FreeNAS 8.3.1 allegedly supports zfs v30(only Oracle supports that version) if you examine table 1.4a and the manual front page says 8.3.1 is based on FreeBSD 8.3.1(which doesn't exist as far as I know).

Via FreeNAS Forums.

Perhaps relying only on a manual to do things isn't always best?

Now can we both agree that this isn't a reliable set-up, and that there is still use for it to some? As long as FreeNAS supports mounting USB drives, people will continue to try. I think the # of posts you mentioned referring to this issue is a good example. I don't see the harm in writing up this how-to if I am clear on any risks associated with it. You have major gripes against external USB, and that's fine, but there is no reason to say "NO" 100% of the time.

I don't mean to be hostile, but you can't speak for everyone either.

 

[cyberjock]

I can agree that it isn't a reliable setup, and that some people might use it either once or regularly. But, I'd limit that to people that can do it on their own with no forum help, etc. Some stuff, such as ESXi, are possible to do safely. The real question(like so many others on this forum) is who should do it. Too many people think they have the knowledge and don't. The drawback with reliability is that when people do dumb things(whether they ignored the warnings or didn't know about them), when they lose their data alot of them will immediately jump to the conclusion that "FreeNAS sucks and is not ready for production". I could link 4 threads from the last 7 days right now, but I won't to protect the guilty.

Also, some of my comments, such as "You gotta do what the manual says if you want your data to be safely stored" wasn't just this topic, but a general best way to handle FreeNAS until you are knowledgeable enough with FreeBSD to do it yourself. Nothing is fool proof, and the manual isn't perfect. But take my case as an example. Until January 2012 I had no clue what FreeBSD was, how it related to linux(or if it related), what ZFS was, etc. I was a Windows guy 100%. Now I'm pretty darn knowledgeable in FreeBSD, submitted some of my own code thats in the current versions of FreeNAS(and I don't have any formal training on whatever language its in), and run Linux on 2 of my machines. In spite of having no clue if I was doing anything right with FreeNAS, I followed the manual, read lots of forum posts, and I got everything right the first time. None of this "not enough RAM" daily thread, no USB, didn't use a RAID controller, etc. The only thing I'd change with that system today would be to use ECC RAM, but I knew of the warnings with not using ECC and I was recycling hardware because I had it. I wasn't expecting it to be used long term. I thought it was my proof of concept and would be disassembled just a few short weeks later. Every system since then has been ECC RAM.

As for those manual mistakes, the only real killer one I've found (I think) that could potentially cost you your data after its too late would be https://support.freenas.org/ticket/2178#ticket. Currently there doesn't appear to be any way to restore the "recovery key" to a failed and replaced disk. So all you have is the key+pass option forever once you replace any disks. And there's been no comments from anyone regarding what either I've done wrong or what is wrong with FreeNAS or its manual(assuming its one of those). I'm convinced the manual is missing steps because I'd hate to think that this scenario was overlooked somehow. I know the developers are darn smart. William is pretty awesome IMO.

Does it really matter if you think FreeNAS is based on FreeBSD 8.3.1 and 8.3.1 doesn't exist? Not particularly but you'd look dumb if you told native FreeBSD users you are using 8.3.1. Plenty of people show up here weekly asking how to load their linux software and drivers on FreeBSD because they think FreeBSD is linux. I'd also argue similarly with the v30 support in FreeNAS. If you are familiar enough with ZFS versions and use Oracle you are informed enough to recognize that ZFS v30 only exists in Oracle's implementation. I doubt many people are going to go to FreeNAS, realize they thing FreeBSD is easier than FreeNAS and then complain that they can't import their encrypted zpool directly into FreeBSD without running extra commands to decrypt it.

 

[kriswithak]

...when people do dumb things(whether they ignored the warnings or didn't know about them), when they lose their data alot of them will immediately jump to the conclusion that "FreeNAS sucks and is not ready for production".

That certainly is a legitimate problem. I sympathize with you there; but I still think a write-up isn't too out of the question, if FreeNAS allows mounting USB. I'm sure this discussion might even reinforce those concerns for some doing google research in the near future.

I followed the manual, read lots of forum posts, and I got everything right the first time. None of this "not enough RAM" daily thread, no USB, didn't use a RAID controller, etc.

I worked similarly. I researched for a few weeks prior to my set-up. I got it up and running in just a few short hours... and I've had it running recently with about a 3 week uptime, until I replaced a drive (for more space). No ram issues... tho I began with only 2gb ram. I had a minor issue with copying large folder structures over SMB but that has been resolved. A week later the other 2gb came in and I installed it, which is when I first encountered this USB-UFS Degraded issue, due to the reboot. Performance reports all look awesome, I'm barely running at 50% capability at peak. Speeds over our GigabitE are wonderful.

I have to say tho, you mentioned FreeBSD being a linux derivative? I'd say UNIX, more accurately.

I know the developers are darn smart. William is pretty awesome IMO.

That is without Question! I'm in love with FreeNAS, and they are very open to community support, which is a huge resource. They've done a great job. I do look forward to a version based on FreeBSD 9.0, then I might get Plex to work. I must admit, I have not checked to see if they are planning to do this currently, but I expect it will happen someday. Although I'm sure it may require a GUI overhaul, which can be extensive.

Plenty of people show up here weekly asking how to load their linux software and drivers on FreeBSD because they think FreeBSD is linux.

Right, FreeNAS a derivative of FreeBSD, which is in turn based off UNIX.

FreeNAS is great, don't get me wrong. I swear by it. But I saw a need and decided to help out anyone doing the same google searches I had. Again, I do still appreciate you pointing out those concerns, for those who might not know.

 

[jgreco]

yikes

 

[cyberjock]

I have to say tho, you mentioned FreeBSD being a linux derivative? I'd say UNIX, more accurately.

Not sure where you are getting that from. I didn't say anything of the sort. I worded it just like I did on purpose, and if you want to make that assumption feel free. I've corrected plenty of people on the forum for that mistake, so I got plenty of evidence that I know better. :P

yikes

Yeah, exactly.

 

[kriswithak]

I've corrected plenty of people on the forum for that mistake, so I got plenty of evidence that I know better. :P
.

Haha, fair enough. I must not have understood you correctly there.

Anyways, using USB externals is great for my application, as it is only meant to be a 1-hour, or 24-hour window into our file history. As I mentioned with the external backups being stored in a fire safe example, I can see other people benefiting from my method.

Sorry if I annoyed or offended anyone, I just wanted to contribute to a complete lack of resources on the topic. In my research, I have seen numerous one-sided answers by cyberjock on the topic in various threads, but I feel it is possible and sustainable to configure external USB for certain applications. I only intended to offer a contrasting opinion. I'd hope this would be considered a good thing about FreeNAS 8.3.1's capabilities...

 

[cyberjock]

You'll never offend me. I'm just trying to look out for people's data. I add people to my ignore list when they show their ignorance by blaming FreeNAS for their lost data. It's like they've never heard of the word "backups" and didn't do their homework before starting to use Linux.. err... FreeBSD :D

See what I did there? :P

I use USB flash drives only, and I only use them when I'm taking something to/from a friend's house. That's about it. I'd never trust them with backups, etc. I even keep a backup of my "universal boot key" that has various tools I've used over the years.

 

[ProtoSD]

Yikes....

Cyberjock, I think everyone knows where you stand on USB storage. I think this discussion has gone too far.