CRITICAL: The volume Volume2 (ZFS) status is DEGRADED help

[taekwonjudo]

Hey guys. First off, sorry for Necroing a VERY old thread.

I have recently only JUST set up my first FreeNas system and so far am LOVING everything about this thing. I am very new to linux but am quite competent with windows & mac & the maintenance of the systems.

I'm running FreeNAS-9.2.1.5-RELEASE-x64 on an AMD Athlon II x4 620 with 4gigs of Ram.

Just this morning I got this error message and have followed paleoN's instruction above.. but I'm having trouble seeing the failed drive. If anyone could spare a second and set their eyes on this I'll paste a copy of my SSH here. I need some advice on what I'm looking at in these reports, and some help finding the trouble drive :/.

Thanks for looking guys & for having such a great community to fall back on for help :)

 

[cyberjock]

paleoN is long gone and you shouldn't be using data from 2 years ago... so I moved you to your own thread.

None of your info makes sense since you included no information on your pool and what it was or is supposed to be. I see a bunch of drives and multiple pools and nothing telling me what disks should/do belong to the pools.

I will tell you this though... as it may help:

1. The disk is "REMOVED". This may mean that the disk failed and is no longer detected by the system at all. What that means is you have to get all the serial numbers for the disks the system does detect, pull your server apart and find the disk that you don't have in your list. That will be the bad disk.
2. If gpart show is giving you segmentation fault errors like what I think I see in the first screen shot you have problems and may want to backup your data while it's still there as a precaution.
3. All of your disks are 39C or higher in temperature. They shouldn't go above 40C normally for various reasons, but no doubt your disks are warmer than they should be. Consider adding fans to your case blowing directly on the disks to keep them cool.

 

[taekwonjudo]

Thanks Cyberjock. I appreciate the feedback. The strange thing is that all 6 of the drives I have in my machine are listed there.. That was also part of my confusion.

Is there something I can add here to give you more info that you'd need to help me troubleshoot this issue? I think I'll take your advice regarding the temperatures of the drives, right now I know they are stacked in pretty close proximity to each other. I'll spread them out a bit and see if that helps.

 

[diedrichg]

There is a GUI view of your volume status:
Storage>click the degraded volume>click the right-most icon at the bottom that says Volume Status. On the new page you will be given the status of the volume's drives. This is also the screen where you will take a drive offline to replace it and then to reattach the new drive for it to resilver.

http://doc.freenas.org/index.php/Volumes#Replacing_a_Failed_Drive

 

[taekwonjudo]

Ok... here's a screen cap of that Volume Status screen.... I'm really confused now. The only thing that has changed is that I have done a re-start of the FreeNAS system as everything was going haywire... any suggestions on what next to try? I'm also attaching a capture of the alert status in case that helps at all as well.

 

[cyberjock]

Well, ada1 looks like it's having problems. It has a bunch of checksum errors. So that disk may need to be replaced. You should check the drive temp(high temps can cause errors), and SMART will tell you if the drive is failing or not. You may always want to try replacing the SATA cable.

Unrelated comment: You also are taking some major risks by not having 8GB of RAM with ZFS. There's a chance the box can crash and destroy your pool permanently.

 

[taekwonjudo]

Ooo. I knew it was recommended to have 8gigs for ZFS but I didn't realize it was THAT imperative. Ok. Good to know. I'll grab some more ram this week. So based on what you see here then aside from the checksum errors reported here does it look like everything has sorted itself out for the time being? I looked again at the SMART reports I posted earlier and still didn't see anything that jumps out at my inexperiences eyes as something to concerned with (aside from the temperature issue which I will deal with tonight).

Thanks for the advice and help guys. I'll figure out which drive is ada1 and pull it and test in another machine to see if I find anything else there.

I'll post back anything I find.

 

[cyberjock]

Well, once the disk detaches as it had before you aren't likely going to get SMART data from it. Depending on your hardware(which you didn't provide even though it's requested in the forum rules) your SMART data may have been cached in the SATA/SAS controller. Also if you look at our hardware recommendations thread you'll notice we didn't recommend AMDs at all. There's a reason for that.. they can have random unexplained problems.

Since you just posted a bunch of stuff with no explanation of when you did it, freenas version, hardware, etc I can't make heads or tails of it. So i gave you the only watered down answer I could give.. ada1 is having problems and you need to figure out what that problem is. ;)

 

[taekwonjudo]

Depending on your hardware(which you didn't provide even though it's requested in the forum rules)

Since you just posted a bunch of stuff with no explanation of when you did it, freenas version, hardware, etc I can't make heads or tails of it.

I'm running FreeNAS-9.2.1.5-RELEASE-x64 on an AMD Athlon II x4 620 with 4gigs of Ram.

??? Am I missing something? Not sure what other hardware specs you would like there Cyberjock... there's nothing else in the system... except the HDDs.. which are listed in the screen caps... I'm not running a raid controller, these are all mounted right from the mobo... I'm happy to fill any blanks I've left.

If need it, my mobo is an MSI 785GTM-E45
RAM is Comstar PC2-6400u-555 2GB 800MHz CL5 Non-ECC

Sorry man. I'm trying to be as compliant as I can. Last thing I want to do is irritate the people who volunteer their time to help out poor schlubs like me ;p

 

[cyberjock]

mobo is important since that's what runs all of your hardware. Knowing your CPU doesn't help with any of the meat-and-potatoes(SATA controller, NIC, chipset, etc.)

Schlubs.. LOL. I haven't heard that word before. I though "schmuck" and "scrub" Naa, it's cool. I'll just ignore you if I get annoyed. ;)

That board isn't what we'd ever recommend(despite being AMD). It's not server grade and has all that desktop stuff that can cause its own problems(sound card, etc.).

But looking at the MSI website it looks like it might not support 64-bit, so having >4GB of RAM might not help. But don't quote me. I didn't open the manual though. I just saw that for OSes the webpage said 32-bit only.

Unfortunately, there's no red flag specific for your exact situation. Your disk could be bad, you might have a PSU that's causing problems on that disk, your SATA cable could be bad, your SATA controller might be had, you might have a compatibility problem with your SATA controller. Some you can rule out on your own easily(like the PSU or SATA cable). But the rest are kind of expensive to rule out. Unfortunately that's about all the advice I can give at the present time. I don't know of any easy way to narrow down the problem except to buy better stuff that we know works and see if the issue continues. :(

 

[taekwonjudo]

Well, the good news is that it IS a 64-bit system (I can confirm that as it was running windows 7 pro - 64 bit prior to becoming my freenas) Never had any PSU issues in the past, system has always run great.

Yeah I know it's not a real server setup but I figured I'd try my had at this stuff with components I had laying around to see if I could even do it before laying out the big bucks on a proper system. :) I'll slowly upgrade as I can.

Thanks for the advice & input. It's much appreciated. I'll pull the one drive out & replace it and put it through the paces on another system to see I can get it to fail. At least you've pointed out a few noobish mistakes I've already made that I can correct ie: drives running too hot. Gonna space them out better tonight.

I'll post back if I find anything else out in case this thread helps anyone else out.

Cheers!

 

[taekwonjudo]

Ok. Very strange & I don't know if it's related or not. Cyberjock I may be experiencing that random crappiness you were mentioning earlier regarding AMD systems... for some reason I lost the shares before trying to do anything further to the system... my plex media server still worked using the web interface but through any of my DLNA devices... so again I tried a restart.... this was where everything started going nuts. I started getting a "filesystem full" error message which after a quick search I believe cyberjock you pointed out in another thread that somehow my USB was getting full.... strange. Simple solution. Create a new install on a new USB key, reload a config file... ok. Back up with the system but the shares still seem to not want allow access.

Can someone point me to what screens I need to pull now to try to figure this one out?

 

[cyberjock]

Which filesystem was full?

Is your pool still mounted? are you sure the CIFS shares point to places that still exist?

 

[taekwonjudo]

Both the CIFS as well as the AFP's are not connecting. Everything looks ok in the share menu..

Hmmm ok. I'm getting that filesystem full message on this new USB as well..:

impactmedia kernel: pid 1908 (syslogd), uid 0 inumber 246 on /mnt: filesystem full


I see the same error repeated a few minutes later withuid 80 inumber 262 as well... rinse and repeat along with:

impactmedia last message repeated 5 times
impactmedia last message repeated 18 times
impactmedia last message repeated 16 times

so on and so on....

(The above I only see because I have a screen attached to the NAS (I don't know if that makes any difference or not...)

 

[cyberjock]

that means you've attempted to copy/move files to /mnt instead of /mnt/<poolname>. The drive is full and that is probably meaning your symbolic link to the zpool isn't being created, hence the shares won't work.

So you need to go to /mnt and delete whatever files you created there. But be careful and don't delete your pool!

 

[taekwonjudo]

:9... I'm confused... I haven't tried to put anything in /mnt... could an instruction be corrupt that I need to rectify?

 

[taekwonjudo]

Ok... I've taken a closer look at things here & I think I see where my issue is... Prior to having problems, I was running 4 2TB drives mirrored (I think that's the correct term?... essentially allowing 4TB of space with the other 2 drives for redundancy and safety... please correct my symantics if I'm wrong there.) Anyways... now when looking at the storage tab, I see that my media volume only has 1.6TB of space... that means that 2 of the drives are no longer recognized in the pool doesn't it? How do I get them back as part of the pool? (Just to clarify on these screens, I only have the 4 drives that are part of this pool attached to the system right now. That's why Holder & Temp are showing as UNKNOWN in status.

 

[taekwonjudo]

Ok... very strange. I re-attached the other 2 drives thinking I would alleviate some of the space on the media Volume.... now all of the shares are working again.... I don't get it.