RAID Z2 Cannot Import Volume I/O Error

[nubrun]

HI there we recently moved the freenas box that we store company files on. Everything has been up for a week with no issues, but came back to work this morning to find my dataset missing. I detached the volume and tried to reattach, but now I am getting an I/O error. Reviewing zpool import I get:

Code:

[root@freenas-primary] ~# zpool import -f
   pool: Data2
	 id: 13429026748135288590
  state: FAULTED
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
		devices and try again.
		The pool may be active on another system, but can be imported using
		the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

		Data2										   FAULTED  corrupted data
		  raidz2-0									  DEGRADED
			13892280858991059497						FAULTED  corrupted data
			17219115969745373477						UNAVAIL  cannot open
			gptid/9101c72b-f45d-11e4-84a0-40167e27da43  ONLINE
			gptid/918221dc-f45d-11e4-84a0-40167e27da43  ONLINE
			gptid/9205ecac-f45d-11e4-84a0-40167e27da43  ONLINE
			gptid/fd57cd8e-9c81-11e7-9192-6805ca5a7a98  ONLINE

I have found the unavailable disc to be disc 4, but with RaidZ2 I should be able to withstand 2 disc failures right? Please help, as our backup is a couple of days old and I don't want to lose that much work if I can avoid it.

System information:


Build FreeNAS-9.3-STABLE-201505040117

Platform AMD A6-5400K APU with Radeon(tm) HD Graphics

Memory 15531MB

 

[nubrun]

I also tried mounting it read only and nothing as well:

Code:

[root@freenas-primary] ~# zpool import -f -o readonly=on Data2
cannot import 'Data2': I/O error
		Destroy and re-create the pool from
		a backup source.
[root@freenas-primary] ~# zpool import -o readonly=on -fNR /mnt Data2
cannot import 'Data2': I/O error
		Destroy and re-create the pool from
		a backup source.

 

[danb35]

What does zpool import -Fn Data2 return?

 

[nubrun]

What does zpool import -Fn Data2 return?

It did something but can't tell what exactly:

Code:

[root@freenas-primary] ~# zpool import -Fn Data2
[root@freenas-primary] ~#

I can't see the volume in the GUI.

 

[danb35]

That's odd; I would have expected it to say something. See if it imported first with zpool status. If it hasn't, try zpool import -F Data2.

 

[nubrun]

That's odd; I would have expected it to say something. See if it imported first with zpool status. If it hasn't, try zpool import -F Data2.

Code:

[root@freenas-primary] ~# zpool status
  pool: freenas-boot
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
		the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Sep 10 03:45:33 2018
config:

		NAME											STATE	 READ WRITE CKSUM
		freenas-boot									DEGRADED	 0	 0	 0
		  mirror-0									  DEGRADED	 0	 0	 0
			13782661019866149478						UNAVAIL	  0	 0	 0  was /dev/da2p2
			gptid/222bb86a-995a-11e7-a0cd-6805ca5a7a98  ONLINE	   0	 0	 0

errors: No known data errors
[root@freenas-primary] ~#

And after zpool import -F Data2

Code:

[root@freenas-primary] ~# zpool import -F Data2
cannot import 'Data2': I/O error
		Destroy and re-create the pool from
		a backup source.
[root@freenas-primary] ~#

 

[danb35]

Not looking too good. I know more recent FreeBSD versions have some code that may let them do better in importing pools, particularly when degraded. Can you try installing 11.2-BETA3 on a fresh USB stick, booting from that, and seeing if you can import the pool from there?

 

[nubrun]

Thanks for the idea Dan I am preparing 11.2 Beta now. To confirm though, the RAIDZ2 protocol should have handled a two disk failure correct? As I rebuild the pool i don't want to make the same mistake (if there was a mistake) again.

 

[danb35]

Yes, RAIDZ2 provides two-disk parity. But if data's gotten corrupted somewhere else...

 

[nubrun]

Hmm, I'm starting to think it might be motherboard or hardware related. I'm having trouble running Freenas 11.1 and 11.2 BETA wouldn't load for whatever reason.

 

[nubrun]

Not looking too good. I know more recent FreeBSD versions have some code that may let them do better in importing pools, particularly when degraded. Can you try installing 11.2-BETA3 on a fresh USB stick, booting from that, and seeing if you can import the pool from there?

Ok, I got 11.2-Beta3 installed on a fresh USB stick, ran "zpool import of Data2" and got the following:

Code:

root@freenas[~]# zpool import -f Data2
internal error: Unknown error: 122
zsh: abort (core dumped)  zpool import -f Data2
root@freenas[~]#

Not sure if that's better then I/O error, or if its a newer version of the same thing.

 

[danb35]

The -f flag isn't needed, and isn't doing anything for you. Try again with -Fn and see if anything changes.

 

[nubrun]

I moved the drives to a new box because I was having hardware issues on the original (perhaps the source of the errors?). It wouldn't let me run it without the -f flag since hte pool had been last accessed on a different box.

But I may have made a little progress, after running zpool import -fFX /mnt Data2 I can't see the volume imported in the GUI of 11.2-Beta 3 but I get this with zpool status which I wasn't getting before:

Code:

root@freenas[~]# zpool status -v
  pool: Data2
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
		corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
		entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: resilvered 198G in 0 days 00:43:44 with 4 errors on Tue Oct  2 04:29:10 2018
config:

		NAME											STATE	 READ WRITE CKSUM
		Data2										   DEGRADED	 0	 0	 4
		  raidz2-0									  DEGRADED	 0	 0	 8
			gptid/900127b2-f45d-11e4-84a0-40167e27da43  DEGRADED	 0	 0  157K  too many errors
			17219115969745373477						UNAVAIL	  0	 0	 0  was /dev/gptid/90840fc4-f45d-11e4-84a0-40167e27da43
			gptid/9101c72b-f45d-11e4-84a0-40167e27da43  DEGRADED	 0	 0	 0  too many errors
			gptid/918221dc-f45d-11e4-84a0-40167e27da43  DEGRADED	 0	 0	 0  too many errors
			gptid/9205ecac-f45d-11e4-84a0-40167e27da43  DEGRADED	 0	 0	 0  too many errors
			gptid/fd57cd8e-9c81-11e7-9192-6805ca5a7a98  DEGRADED	 0	 0	 0  too many errors

errors: Permanent errors have been detected in the following files:

		Data2/.system/samba4:/locking.tdb
		Data2/.system/samba4:/smbXsrv_open_global.tdb

Not good I know, but it still seems like forward progress.

 

[danb35]

I can't see the volume imported in the GUI of 11.2-Beta 3

That's to be expected; pools imported at the CLI don't appear in the GUI. You should be able to export the pool at the CLI, and then import it in the GUI.

 

[nubrun]

That's to be expected; pools imported at the CLI don't appear in the GUI. You should be able to export the pool at the CLI, and then import it in the GUI.

So it looks like I was able to mount the pool and am getting the data off it. It seems like it repaired itself enough to be functional enough. I appreciate all of your help here Dan.