'zpool clear' or 'zpool replace'.

[swamphox1]

Can someone please help me understand how to find out what the problem is here. I have no real idea what to do with this...

• WARNING: The volume swamphoto (ZFS) status is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'.

 

[SweetAndLow]

What does the smart data say about your driver's and please read the forum rules and provide that info.

 

[swamphox1]

I need some help with this. I built this machine a couple years ago and I have had no issues all all. I do not know how to anything really. Here is a little data I was able to acquire.

Aug 20 11:00:00 swampdrive newsyslog[12668]: logfile turned over due to size>100K
Aug 20 11:00:00 swampdrive autosnap.py: [tools.autosnap:57] Popen()ing: /sbin/zfs snapshot swamphoto@auto-20150820.1100-2d
Aug 20 11:00:00 swampdrive autosnap.py: [tools.autosnap:244] Failed to create snapshot 'swamphoto@auto-20150820.1100-2d': cannot open 'swamphoto': pool I/O is currently suspended usage: snapshot [-r] [-o property=value] ... <filesystem@snapname|volume@snapname> ... For the property list, run: zfs set|get For the delegated permission list, run: zfs allow|unallow
Aug 20 11:00:11 swampdrive kernel: hwpstate0: set freq failed, err 6
Aug 20 11:00:32 swampdrive last message repeated 6 times
Aug 20 11:01:01 swampdrive autosnap.py: [tools.autosnap:57] Popen()ing: /sbin/zfs snapshot swamphoto@auto-20150820.1101-2d
Aug 20 11:01:01 swampdrive autosnap.py: [tools.autosnap:244] Failed to create snapshot 'swamphoto@auto-20150820.1101-2d': cannot open 'swamphoto': pool I/O is currently suspended usage: snapshot [-r] [-o property=value] ... <filesystem@snapname|volume@snapname> ... For the property list, run: zfs set|get For the delegated permission list, run: zfs allow|unallow
Aug 20 11:01:08 swampdrive kernel: hwpstate0: set freq failed, err 6

and this is the current alert

• WARNING: The volume swamphoto (ZFS) status is ONLINE: One or more devices are faulted in response to IO failures.Make sure the affected devices are connected, then run 'zpool clear'.

I either need to be walked through what to or point in the right direction to find out on my own.

THanks

 

[danb35]

You need some help, and we need information to have any chance of giving you that help. @SweetAndLow asked on Tuesday for some of that information, and the forum rules (which you said you read and agreed to when you signed up on Monday--there's a link to them in red at the top of the page, in case you've forgotten) give quite a bit more detail about what we're likely to need and why.

 

[swamphox1]

At this point I am limited in the amount of info I can retrieve. Are you able to point me to info to help collect the data that the rule require???

System Information
Hostname swampdrive.local Edit
Build FreeNAS-9.2.0-RELEASE-x64 (ab098f4)
Platform AMD FX(tm)-4130 Quad-Core Processor
Memory 16261MB
System Time Thu Aug 20 11:14:00 EST 2015
Uptime 11:14AM up 1:51, 0 users
Load Average 0.17, 0.20, 0.17

 

[SweetAndLow]

Hmmm... Does 9.2 have the debug file option? I think that is the only way to help this person.

 

[swamphox1]

I am having trouble connecting to the machine even though it shows up on my DHCP Client table. I am thinking the disk that freenas runs on may be a problem. I am looking for suggestions to diagnose. Would it be possible to pull the ZFS drives and reinstall Freenas and then reconnect the ZFS drives?

 

[danb35]

Would it be possible to pull the ZFS drives and reinstall Freenas and then reconnect the ZFS drives?

Yes, it certainly would. It would also be possible to download your configuration file from your existing installation and upload it to the new installation. In that way, you'd preserve all your users, shares, and any other settings you've made.

 

[swamphox1]

Just reading and understanding the rule is not getting me anywhere. Definitely over my head to have set up this machine to begin with but I have using it without fail and addind drives for 2or 3 years. I need baby steps to diagnose the problems. I am able to get back intothe drive now. I did pull the drive that is running freenas and it was quite hot. Once it cooled down I was able to get back in. I am showing a critical error and the drives as degraded

 

[danb35]

Are you able to connect to your server using SSH? If so, run the following commands and provide the output:
'zpool status'
'camcontrol devlist'

 

[swamphox1]

I would really like to be able to help you help me with this but I can't seem to be able to get any of the info that the rules require me to get.

 

[swamphox1]

Good question. I may be able to but I don't know how.

 

[danb35]

Well, that's something you really should learn. It's not critical with this immediate step, but it will be important very soon. The manual covers configuring your FreeNAS server for SSH in section 8.12 (check the "login as root with password" box). On the client side, it will depend what OS you're running. If you're using Windows, PuTTY is the old standby, and I'm hearing some love for Bitvise recently. If MacOS or any flavor of Linux (or *BSD for that matter), just open a terminal window and type "ssh root@ip.of.freenas.server".

But for now, you can access the shell from the web GUI. Log in, and you'll see a menu item on the left for "Shell". Click that, and it will bring up a window connecting you to the shell on your server. Type the commands I gave in my previous post, and post the output here.

You can also go into the web GUI, under Settings -> Advanced, and click the Save Debug button. Post that output here as well.

 

[swamphox1]

I think these are supposed to be formatted or bracketed. I will do so in the future.

status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 14h13m with 0 errors on Sun Aug 9 14:13:32 2015
config:

NAME STATE READ WRITE CKS
UM
swamphoto DEGRADED 0 0
0
mirror-0 DEGRADED 0 0
0
gptid/7a4f52af-733e-11e3-ae8b-d850e6db3a84 ONLINE 0 0
0
8933907702982819623 UNAVAIL 0 0
0 was /dev/gptid/7ace49e7-733e-11e3-ae8b-d850e6db3a84
gptid/30906aab-740b-11e3-ae8b-d850e6db3a84 ONLINE 0 0
0
gptid/2f78032f-ae28-11e4-9bd9-d850e6db3a84 ONLINE 0 0

---------------------------------------------------------------------------------------------------------------

errors: 1 data errors, use '-v' for a list
[root@swampdrive ~]# camcontrol devlist
<WDC WD1600JD-40GBB2 08.05K08> at scbus0 target 0 lun 0 (ada0,pass0)
<WDC WD30EZRX-00DC0B0 80.00A80> at scbus1 target 0 lun 0 (ada1,pass1)
<WDC WD30EZRX-00DC0B0 80.00A80> at scbus2 target 0 lun 0 (ada2,pass2)
<WDC WD30EZRX-00DC0B0 80.00A80> at scbus3 target 0 lun 0 (ada3,pass3)
<WDC WD30EFRX-68EUZN0 80.00A80> at scbus4 target 0 lun 0 (ada4,pass4)

 

[danb35]

Yes, it's better if this stuff is put in code tags. To do that, type {code}your output{/code}, replacing the {} with []. Yes, that's kind of a convoluted way of explaining it, but I can't enter the tags accurately without it interpreting them. If you have a long block of output, as you do, you only need the {code} tag at the beginning, and {/code} at the end (you don't need the tags for each line). The result should look like this:

Code:

your output

Now try with the output of 'zpool status -v'. And if you can get the debug file from your server, post that too--it may have a lot of the information we'll need.

 

[swamphox1]

Code:

action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 14h13m with 0 errors on Sun Aug 9 14:13:32 2015
config:

NAME STATE READ WRITE CKS
UM
swamphoto DEGRADED 0 0
0
mirror-0 DEGRADED 0 0
0
gptid/7a4f52af-733e-11e3-ae8b-d850e6db3a84 ONLINE 0 0
0
8933907702982819623 UNAVAIL 0 0
0 was /dev/gptid/7ace49e7-733e-11e3-ae8b-d850e6db3a84
gptid/30906aab-740b-11e3-ae8b-d850e6db3a84 ONLINE 0 0
0
gptid/2f78032f-ae28-11e4-9bd9-d850e6db3a84 ONLINE 0 0
0

errors: Permanent errors have been detected in the following files:

<metadata>:<0xcb>
Code:

Not sure how to get the debug file. I have looking but not able to find it or really how to get it.

 

[swamphox1]

Looks like I did something not quite right...

 

[danb35]

You were close. You use [/code] at the end of your output, though.

To get the debug file, when you log in to the web GUI of your server, you'll click on the Advanced tab, and then at the bottom you should see a button marked "save debug".

 

[swamphox1]

Here is a link

swamphox.com/debug-swampdrive-20150820142558.tgz

 

[danb35]

Great. That's given a lot of information, but interestingly it doesn't contain SMART output from your disks. Is the SMART service enabled? From the web GUI, select Services from the top row of icons. There will be a list of services, and SMART should be one of them. If it's off, turn it on and generate another debug file.

From what I'm seeing, though, you have two problems which are going to require you to rebuild your pool:

1. You have corruption in the pool metadata which ZFS is unable to repair.
2. Your pool configuration is dangerously short on redundancy. What you have is a two-disk mirror, striped with two more disks. Even after you replace the failed disk (which you definitely need to do), when (not if) either of the striped disks fails, you will lose your pool and all data on it. I'm assuming you care about your data (since you have some redundancy in the first place), so this is a situation you'll want to correct sooner, rather than later.

Is your boot device an actual hard drive? Although that's "legal" for FreeNAS, it's very unusual, and tends to be discouraged. A USB stick or a small SSD is generally preferred.