Please help! Started up FreeNAS, suddenly voluime storage (ZFS) status unknown?!

[ProtoSD]

PaleoN is in a different timezone, so I'm not sure when he'll be online. I think he still has a few more things for you to try.

I'm kind of wondering which device it thinks is unavailable, or why.

 

[HHawk]

PaleoN is in a different timezone, so I'm not sure when he'll be online. I think he still has a few more things for you to try.
I'm kind of wondering which device it thinks is unavailable, or why.

Yeah no problem, I thought so. So you are both in different time-zones?

Exactly I am wondering the same thing; it said that before as well, with the other (onboard) controller. It couldn't be the USB stick right?
It's a different one than the one with FreeNAS on it though. And it's pretty new.

It's annoying that it doesn't give more feedback about such kind of errors or messages. :(

Any commands I can still give a go? For example the one with the -X (which also takes a long while)...?

- - - Updated - - -

I meant this command:

Code:

time zpool import -fFX storage

paleoN mentioned this earlier and said I should run it twice.
I guess now would be a good time since he ain't online soon and I am not doing much else on the NAS...?

 

[paleoN]

The first command took a long time...

Very long. I wasn't accounting for vfs.zfs.recover being set to 1.

I think he still has a few more things for you to try.

I'm kind of wondering which device it thinks is unavailable, or why.

I have a few more. Including one that may show us which it thinks is unavailable. A reboot is likely in order to reset vfs.zfs.recover to 0 and not have imports take 9 hours.

I meant this command:

Code:

time zpool import -fFX storage

I'd wait on that one.

- - - Updated - - -

I'd wait on that one.

Statements such as these me want to wait until later:

As a final desperation move, or after you've managed to import the pool readonly and copy everything off you can try:

# zpool import -R /mnt -FX poolname

The above command has a good chance of grenading your pool, but it also has a chance of importing it.

 

[HHawk]

Okay thanks for the update.

So I will have to reboot now and do nothing for now?
I don't know if I can access it by SSH after reboot though. As mentioned I am at work. I did a quick configuration in regarding to forwarding the correct port...

 

[ProtoSD]

The -X and -T commands are related, -T tells it to go back to a specific point in time, hopefully a time when things weren't damaged. The problem could be that ZFS doesn't always store files across all of the disks in your pool, so there might be some files it can't rebuild because it thinks one of the disks is missing. I think we need to really try to figure out what changed, and when, and why it thinks a disk is missing.

 

[paleoN]

So I will have to reboot now and do nothing for now?
I don't know if I can access it by SSH after reboot though. As mentioned I am at work. I did a quick configuration in regarding to forwarding the correct port...

That's probably the safer way to go. In addition I could use another few hours myself. I was just checking in.

 

[HHawk]

The -X and -T commands are related, -T tells it to go back to a specific point in time, hopefully a time when things weren't damaged. The problem could be that ZFS doesn't always store files across all of the disks in your pool, so there might be some files it can't rebuild because it thinks one of the disks is missing. I think we need to really try to figure out what changed, and when, and why it thinks a disk is missing.

Ah okay... It's weird though, that smartctl checks (short or long) both shows all drives are working correctly. And also it's not the (onboard) controller as I am using a brand new SATA controller.
If a drive was "missing" it should have mentioned it earlier during checks right?

The only thing which "changed" in recent time (as in past year) was that one disk went bad, had to replace it and resilver it (as mentioned in the other thread), after that everything was working fine and I didn't have any problems and / or errors till now.

- - - Updated - - -

That's probably the safer way to go. In addition I could use another few hours myself. I was just checking in.

Than I will just shutdown the nas, no use to leave it up running = waste of energy.

//update

Turned it off now...

 

[ProtoSD]

Can you remember specifically when you replaced it?

Also, can you remember the last time you did a scrub, exactly?

I'm not sure how, but those answers might be helpful.


I think I'm going to retire for the night. I have trouble with my neck and it's VERY painful. Spending time at the computer makes it worse....

 

[HHawk]

Can you remember specifically when you replaced it?

Also, can you remember the last time you did a scrub, exactly?

I'm not sure how, but those answers might be helpful.

I think I'm going to retire for the night. I have trouble with my neck and it's VERY painful. Spending time at the computer makes it worse....

Looking at this thread, which I created I would say the drive was replaced on the 8th of December 2012.

Last srub would be a guess. I am guessing the last scrub was between 20 or 30 days. A new scrub was due one of these days.

I used to do it every 15 days, but in that same thread paleon said scrubbing once a month was sufficient along with SMART tests.

IMHO, scrubs every 5 days is excessive. In fact with higher parity levels you can argue that you can scrub less often. The statistical likelihood of having an unrecoverable read on 3 separate disks for the same block of ZFS data is... unlikely to say the least. Other failure modes begin to dominate.

I would still scrub at least once a month and regularly run some long SMART tests to check for read errors on the drives. The SMART tests are faster, check any unused sectors on the drive and are less stressful on the drives.

Anyways have a good night's rest and I hope your neck will feel better soon.



//sidenote

Oh and I don't know if it matters or not; "IF" there is a fix and this would mean I would lose all my data after my last scrub, this is not a problem for me, because I would still have all my important stuff, etc... Probably isn't important, but just letting you guys know.

 

[cyberjock]

I'm still here. Watching this thread and only getting more and more confused. The zpool seems to be convinced that another device should exist. Have you ever had a ZIL or L2ARC?

Also, I'm impressed that you haven't gone off doing things you shouldn't do. This is by far the longest I've seen an OP go without deciding to doing something stupid despite everyone telling you not to do anything without direction. It's also the most troubleshooting I think I've ever seen for someone's data. I'm sure the two are related.

I'm really hoping we can figure out what is going on and get you your data back. I'm not sure if you're being patient because you know your wife will kill you if she figures out how bad things are right now, or if you are that "cool headed". In any case, keep up the good work. I'm rooting for you!

 

[paleoN]

Looking at this thread, which I created I would say the drive was replaced on the 8th of December 2012.

It was the 4[sup]th[/sup], 2012-12-04.07:43:18 zpool replace storage 6201553240551106299 gptid/3dc2f956-3de6-11e2-8af1-00151736994a.

Last srub would be a guess. I am guessing the last scrub was between 20 or 30 days. A new scrub was due one of these days.

Last scrub was the 23[sup]rd[/sup], 2013-02-23.23:23:34 zpool scrub storage.

Boot up mfsBSD normally and run:

Code:

zpool import -V storage

If that "imports", given the other error messages, the pool will most likely be considered faulted. Which also means you can't actually do anything with it.

Code:

zpool status -v

Ought to show us what devices it considers missing.

 

[HHawk]

I'm still here. Watching this thread and only getting more and more confused. The zpool seems to be convinced that another device should exist. Have you ever had a ZIL or L2ARC?

Also, I'm impressed that you haven't gone off doing things you shouldn't do. This is by far the longest I've seen an OP go without deciding to doing something stupid despite everyone telling you not to do anything without direction. It's also the most troubleshooting I think I've ever seen for someone's data. I'm sure the two are related.

I'm really hoping we can figure out what is going on and get you your data back. I'm not sure if you're being patient because you know your wife will kill you if she figures out how bad things are right now, or if you are that "cool headed". In any case, keep up the good work. I'm rooting for you!

In regards to your comment: "ZIL or L2ARC" I can answer no. I looked into it, but you needed a SSD or something, which I didn't haven and it was to much work at that time (in my opinion), so I didn't do that.

Well the data is pretty important to me, that's why I am happy with the help I am getting currently, though my posts reveal otherwise, I am still kinda anxious. However I am currently considering the data last, thought I didn't tell the wife that just yet of course. And about being "cool headed", I was pretty down and semi angry past weekend when this stuff appeared to be gone in the first place. However what use is it to anyone, especially to everyone who is trying to help me out here, to get short tempered or anything...? Not much. And as mentioned above; I am currently seriously considering everything to be lost. So even the smallest "result" would mean a (small) victory in this particular case.

It was the 4[sup]th[/sup], 2012-12-04.07:43:18 zpool replace storage 6201553240551106299 gptid/3dc2f956-3de6-11e2-8af1-00151736994a.

Last scrub was the 23[sup]rd[/sup], 2013-02-23.23:23:34 zpool scrub storage.

Boot up mfsBSD normally and run:

Code:

zpool import -V storage

If that "imports", given the other error messages, the pool will most likely be considered faulted. Which also means you can't actually do anything with it.

Code:

zpool status -v

Ought to show us what devices it considers missing.

Now here is the weird part (maybe not for you, but for me it is)... I have ran the command "zpool import -V storage" already a few times earlier (few posts back), however every time I run it it doesn't report anything back at all.
Just the prompt.

Code:

root@mfsbsd:/root # zpool import -V storage
root@mfsbsd:/root # 

This happened also before...?


The other command shows this:

Code:

root@mfsbsd:/root # zpool status -v
  pool: storage
 state: FAULTED
status: The pool metadata is corrupted and the pool cannot be opened.
action: Destroy and re-create the pool from
        a backup source.
   see: http://illumos.org/msg/ZFS-8000-72
  scan: scrub repaired 0 in 3h50m with 0 errors on Sun Feb 24 03:13:57 2013
config:

        NAME                                            STATE     READ WRITE CKSUM
        storage                                         FAULTED      0     0     2
          raidz2-0                                      ONLINE       0     0     8
            gptid/19177fb9-25fa-11e2-9ab0-00151736994a  ONLINE       0     0     0
            gptid/19b5ec3a-25fa-11e2-9ab0-00151736994a  ONLINE       0     0     0
            gptid/3dc2f956-3de6-11e2-8af1-00151736994a  ONLINE       0     0     0
            gptid/1aefa3e9-25fa-11e2-9ab0-00151736994a  ONLINE       0     0     0
            gptid/1b8f2b64-25fa-11e2-9ab0-00151736994a  ONLINE       0     0     0
            gptid/1c2d6a74-25fa-11e2-9ab0-00151736994a  ONLINE       0     0     0
root@mfsbsd:/root # 

Don't know if this helps.

- - - Updated - - -

Huh...

This is new right?

Code:

root@mfsbsd:/root # zpool status
  pool: storage
 state: FAULTED
status: The pool metadata is corrupted and the pool cannot be opened.
action: Destroy and re-create the pool from
        a backup source.
   see: http://illumos.org/msg/ZFS-8000-72
  scan: scrub repaired 0 in 3h50m with 0 errors on Sun Feb 24 03:13:57 2013
config:

        NAME                                            STATE     READ WRITE CKSUM
        storage                                         FAULTED      0     0     2
          raidz2-0                                      ONLINE       0     0     8
            gptid/19177fb9-25fa-11e2-9ab0-00151736994a  ONLINE       0     0     0
            gptid/19b5ec3a-25fa-11e2-9ab0-00151736994a  ONLINE       0     0     0
            gptid/3dc2f956-3de6-11e2-8af1-00151736994a  ONLINE       0     0     0
            gptid/1aefa3e9-25fa-11e2-9ab0-00151736994a  ONLINE       0     0     0
            gptid/1b8f2b64-25fa-11e2-9ab0-00151736994a  ONLINE       0     0     0
            gptid/1c2d6a74-25fa-11e2-9ab0-00151736994a  ONLINE       0     0     0
root@mfsbsd:/root # 

This didn't show before...?

Indeed...!

I looked up previous posts in this thread and it would show this instead:

Code:

[root@freenas] ~# zpool status
 no pools available

Does this mean anything?!

 

[paleoN]

Does this mean anything?!

Yes, the zpool import -V is a different command. It is showing pool metadata corruption which isn't good. Let's try another -T import:

Code:

zpool export storage

zpool import -T 732362 -o rdonly=on storage

 

[HHawk]

Yes, the zpool import -V is a different command. It is showing pool metadata corruption which isn't good. Let's try another -T import:

Code:

zpool export storage
zpool import -T 732362 -o rdonly=on storage

Well I also meant just "zpool status" without anything behind is now also showing the pool. This wasn't before...

Anyways ran the commands:

Code:

root@mfsbsd:/root # zpool export storage
root@mfsbsd:/root # zpool import -T 732362 -o rdonly=on storage
cannot import 'storage': one or more devices is currently unavailable

I also ran zpool status again (since that is harmless) and now it's gone again. Probably cause we exported it with the previous command...

 

[paleoN]

Code:

root@mfsbsd:/root # zpool export storage
root@mfsbsd:/root # zpool import -T 732362 -o rdonly=on storage
cannot import 'storage': one or more devices is currently unavailable

OK, try:

Code:

zpool import -FT 732362 -o rdonly=on storage

 

[HHawk]

Code:

root@mfsbsd:/root # zpool import -FT 732362 -o rdonly=on storage
cannot import 'storage': one or more devices is currently unavailable

Probably it's a stupid question, but I am going to ask anyways. This cannot be related in anyway that we are using a different FreeBSD right? In contrary to the FreeNAS usb stick. Because it's always complaining about "one or more devices is currently unavailable"...?

 

[paleoN]

This cannot be related in anyway that we are using a different FreeBSD right? In contrary to the FreeNAS usb stick. Because it's always complaining about "one or more devices is currently unavailable"...?

It's not. The error message could be improved.

Reboot mfsBSD:

Code:

set vfs.zfs.recover=1
set vfs.zfs.debug=1
boot

Try:

Code:

time zpool import -FXT 732362 storage

This will likely take hours again.

 

[HHawk]

It's not. The error message could be improved.

Reboot mfsBSD:

Code:

set vfs.zfs.recover=1
set vfs.zfs.debug=1
boot

Try:

Code:

time zpool import -FXT 732362 storage

This will likely take hours again.

Well I cannot try that, I am at work and currently working remotely... :S
Has to wait until I am back home 3 or 4 hours or so...

//update

Will it not say something similar to this:

Code:

cannot import 'storage': one or more devices is currently unavailable

As previously happened?

 

[HHawk]

Well in the meantime I have been searching as well. I don't know if this is useful, but I am sharing it anyways.

Link #1

Link #2

Another guess; wouldn't booting up with an Oracle Solaris 11.1 Live Media for x86 USB/CD and run commands do anything?
Probably not, but couldn't hurt to ask..

 

[HHawk]

Bleh...

I got home, had diner, booted the NAS, pressed 2 and enter the commands.

Than I ran the command. Which was done within seconds... :S

Code:

root@mfsbsd:/root # time zpool import -FXT 732362 storage
cannot import 'storage': one or more devices is currently unavailable
0.007u 3.797s 0:06.76 56.0%     112+2677k 240+0io 91pf+0w

And I redid it once more to make sure, but same thing:

Code:

root@mfsbsd:/root # time zpool import -FXT 732362 storage
cannot import 'storage': one or more devices is currently unavailable
0.000u 3.309s 0:05.78 57.0%     111+2649k 240+0io 0pf+0w