Error getting available space - missing devices

Status
Not open for further replies.
B

BWF

Cadet
Joined
Jun 27, 2018
Messages
2
Hi,

I have inherited a FreeNas that now has an issue and would really appreciate some advice on what the best steps to take to get the pool back online.

Here is what has happened, the computer it runs on was non-responsive (unable to login via ssh or GUI) pressed the reset button!
Now the backuppool is in error.

Version:
FreeNAS-11.1-U5

GUI Storage/Volumes/View volumes
upload_2018-6-27_17-45-18.png


Code:
zpool status -x
all pools are healthy

# zpool status -v
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:01:27 with 0 errors on Wed Jun 27 03:46:27 2018
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da0p2	 ONLINE	   0	 0	 0

errors: No known data errors

# zpool import
   pool: backupPool
	 id: 10059534821358208646
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
		devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
 config:

		backupPool									  UNAVAIL  missing device
		  raidz2-0									  ONLINE
			gptid/a5ff1a2b-ee1d-11e4-b10f-0cc47a448c0f  ONLINE
			gptid/160ddc95-da63-11e7-99e1-0cc47a448c0f  ONLINE
			gptid/a6260283-ee1d-11e4-b10f-0cc47a448c0f  ONLINE
			gptid/a639f16a-ee1d-11e4-b10f-0cc47a448c0f  ONLINE
			gptid/213b0d33-d4c4-11e7-a77f-0cc47a448c0f  ONLINE
			gptid/a6622d87-ee1d-11e4-b10f-0cc47a448c0f  ONLINE
			gptid/a6781d76-ee1d-11e4-b10f-0cc47a448c0f  ONLINE
			gptid/a689a2d9-ee1d-11e4-b10f-0cc47a448c0f  ONLINE
			gptid/a69ea424-ee1d-11e4-b10f-0cc47a448c0f  ONLINE
			gptid/a6b7415f-ee1d-11e4-b10f-0cc47a448c0f  ONLINE
		  gptid/e145ec7f-da63-11e7-99e1-0cc47a448c0f	ONLINE
		  gptid/e1b360fe-da63-11e7-99e1-0cc47a448c0f	ONLINE

		Additional devices are known to be part of this pool, though their
		exact configuration cannot be determined.

  # camcontrol devlist
<LSI MR9271-4i 3.41>			   at scbus0 target 0 lun 0 (pass0,da0)
<LSI MR9271-4i 3.41>			   at scbus0 target 1 lun 0 (pass1,da1)
<LSI MR9271-4i 3.41>			   at scbus0 target 2 lun 0 (pass2,da2)
<LSI MR9271-4i 3.41>			   at scbus0 target 3 lun 0 (pass3,da3)
<LSI MR9271-4i 3.41>			   at scbus0 target 4 lun 0 (pass4,da4)
<LSI MR9271-4i 3.41>			   at scbus0 target 5 lun 0 (pass5,da5)
<LSI MR9271-4i 3.41>			   at scbus0 target 6 lun 0 (pass6,da6)
<LSI MR9271-4i 3.41>			   at scbus0 target 7 lun 0 (pass7,da7)
<LSI MR9271-4i 3.41>			   at scbus0 target 8 lun 0 (pass8,da8)
<LSI MR9271-4i 3.41>			   at scbus0 target 9 lun 0 (pass9,da9)
<LSI MR9271-4i 3.41>			   at scbus0 target 10 lun 0 (pass10,da10)
<LSI MR9271-4i 3.41>			   at scbus0 target 11 lun 0 (pass11,da11)
<LSI MR9271-4i 3.41>			   at scbus0 target 12 lun 0 (pass12,da12)
<LSI SAS2X28 0e12>				 at scbus1 target 4 lun 0 (ses0,pass13)

# glabel status
									  Name  Status  Components
gptid/eea9118b-a6e8-11e4-b26b-0cc47a448c0f	 N/A  da0p1
gptid/a6781d76-ee1d-11e4-b10f-0cc47a448c0f	 N/A  da1p2
gptid/a6b7415f-ee1d-11e4-b10f-0cc47a448c0f	 N/A  da2p2
gptid/213b0d33-d4c4-11e7-a77f-0cc47a448c0f	 N/A  da3p2
gptid/a689a2d9-ee1d-11e4-b10f-0cc47a448c0f	 N/A  da4p2
gptid/a6622d87-ee1d-11e4-b10f-0cc47a448c0f	 N/A  da5p2
gptid/a6260283-ee1d-11e4-b10f-0cc47a448c0f	 N/A  da6p2
gptid/a639f16a-ee1d-11e4-b10f-0cc47a448c0f	 N/A  da7p2
gptid/a5ff1a2b-ee1d-11e4-b10f-0cc47a448c0f	 N/A  da8p2
gptid/160ddc95-da63-11e7-99e1-0cc47a448c0f	 N/A  da9p2
gptid/a69ea424-ee1d-11e4-b10f-0cc47a448c0f	 N/A  da10p2
gptid/e145ec7f-da63-11e7-99e1-0cc47a448c0f	 N/A  da11p2
gptid/e1b360fe-da63-11e7-99e1-0cc47a448c0f	 N/A  da12p2
gptid/a6b19147-ee1d-11e4-b10f-0cc47a448c0f	 N/A  da2p1
gptid/a673efad-ee1d-11e4-b10f-0cc47a448c0f	 N/A  da1p1

# gpart show
=>	   34  975699901  da0  GPT  (465G)
		 34	   1024	1  bios-boot  (512K)
	   1058		  6	   - free -  (3.0K)
	   1064  975698864	2  freebsd-zfs  (465G)
  975699928		  7	   - free -  (3.5K)

=>		34  5859442621  da1  GPT  (2.7T)
		  34		  94	   - free -  (47K)
		 128	 4194304	1  freebsd-swap  (2.0G)
	 4194432  5855248216	2  freebsd-zfs  (2.7T)
  5859442648		   7	   - free -  (3.5K)

=>		34  5859442621  da2  GPT  (2.7T)
		  34		  94	   - free -  (47K)
		 128	 4194304	1  freebsd-swap  (2.0G)
	 4194432  5855248216	2  freebsd-zfs  (2.7T)
  5859442648		   7	   - free -  (3.5K)

=>		40  5859442608  da3  GPT  (2.7T)
		  40		  88	   - free -  (44K)
		 128	 4194304	1  freebsd-swap  (2.0G)
	 4194432  5855248208	2  freebsd-zfs  (2.7T)
  5859442640		   8	   - free -  (4.0K)

=>		34  5859442621  da4  GPT  (2.7T)
		  34		  94	   - free -  (47K)
		 128	 4194304	1  freebsd-swap  (2.0G)
	 4194432  5855248216	2  freebsd-zfs  (2.7T)
  5859442648		   7	   - free -  (3.5K)

=>		34  5859442621  da5  GPT  (2.7T)
		  34		  94	   - free -  (47K)
		 128	 4194304	1  freebsd-swap  (2.0G)
	 4194432  5855248216	2  freebsd-zfs  (2.7T)
  5859442648		   7	   - free -  (3.5K)

=>		34  5859442621  da6  GPT  (2.7T)
		  34		  94	   - free -  (47K)
		 128	 4194304	1  freebsd-swap  (2.0G)
	 4194432  5855248216	2  freebsd-zfs  (2.7T)
  5859442648		   7	   - free -  (3.5K)

=>		34  5859442621  da7  GPT  (2.7T)
		  34		  94	   - free -  (47K)
		 128	 4194304	1  freebsd-swap  (2.0G)
	 4194432  5855248216	2  freebsd-zfs  (2.7T)
  5859442648		   7	   - free -  (3.5K)

=>		34  5859442621  da8  GPT  (2.7T)
		  34		  94	   - free -  (47K)
		 128	 4194304	1  freebsd-swap  (2.0G)
	 4194432  5855248216	2  freebsd-zfs  (2.7T)
  5859442648		   7	   - free -  (3.5K)

=>		40  5859442608  da9  GPT  (2.7T)
		  40		  88	   - free -  (44K)
		 128	 4194304	1  freebsd-swap  (2.0G)
	 4194432  5855248208	2  freebsd-zfs  (2.7T)
  5859442640		   8	   - free -  (4.0K)

=>		34  5859442621  da10  GPT  (2.7T)
		  34		  94		- free -  (47K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  5855248216	 2  freebsd-zfs  (2.7T)
  5859442648		   7		- free -  (3.5K)

=>		40  1952448432  da11  GPT  (931G)
		  40		  88		- free -  (44K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  1948254032	 2  freebsd-zfs  (929G)
  1952448464		   8		- free -  (4.0K)

=>		40  1952448432  da12  GPT  (931G)
		  40		  88		- free -  (44K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  1948254032	 2  freebsd-zfs  (929G)
  1952448464		   8		- free -  (4.0K)
 
Last edited by a moderator:
danb35

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
backupPool appears to be (and I say appears because it's impossible to be certain, since you didn't post your output in code tags to preserve formatting) composed of a ten-disk RAIDZ2 vdev, striped with more than two additional disks (how many more is unknown)--a configuration that can most charitably be described as "reckless". You need to get that additional disk (or those additional disks) online, or everything in your pool is gone.
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
i see @danb35 beat me to it.

It looks like you had a RAIDZ2 pool and then added 2 more drives to it (which are additional VDEVs)... maybe another one also (which would be the missing one).

When you have a pool with multiple VDEVs and lose 1 of them, the whole pool is unavailable.

You may or may not be able to force the pool to go online (with data missing).

The lists don't match on da0, which you show with a partition 1, but not a partition 2, so perhaps it's this disk which is the problem? (although it's a completely different size than the others... I note that da11 and 12 are also a different size to the rest in the RAIDZ2 pool).

You have quite a mess there... time to look for your backup and start the pool again from scratch. (exclude/replace the additional disks as they are not the right size)
 
danb35

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
sretalla said:
The lists don't match on da0, which you show with a partition 1, but not a partition 2,
Click to expand...
No, da0 has partitions 1 and 2. It's the boot device, which for some reason is on a 500 GB drive.
sretalla said:
It looks like you had a RAIDZ2 pool and then added 2 more drives to it (which are additional VDEVs)... maybe another one also
Click to expand...
At least one more, maybe two or more.
sretalla said:
You may or may not be able to force the pool to go online (with data missing).
Click to expand...
I don't believe there's any way to do that. If he can't get the missing disk online, AFAIK, the pool is toast.
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
danb35 said:
I don't believe there's any way to do that. If he can't get the missing disk online, AFAIK, the pool is toast.
Click to expand...

What about this thread? seems like it might be possible get to some of the data...

zpool import -f -o readonly=on poolname

I guess it would first be necessary to eliminate/export/remove the pool so it can be imported... no experience with it, so can't recommend the exact steps, but see if the thread gives some clues to something worth trying.

Also, a much older thread, but suggests another option to the import command:

zpool import -o failmode=continue -o rdonly=on poolname
 
Last edited:
danb35

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
sretalla said:
What about this thread? seems like it might be possible get to some of the data...
Click to expand...
Very different situation. Though at this point I guess it couldn't hurt, if there's no way to get the missing disk(s) back online.
 
B

BWF

Cadet
Joined
Jun 27, 2018
Messages
2
Hi,

Thanks for your help with this. I have been back to the data centre where the FreeNas is to get more info.

There is an Adaptec RAID controller, the two 465GB are mirrored by the Adaptec RAID controller, the rest individual in their own drive group.

upload_2018-7-12_11-43-59.png



The one in slot 14 is the bad one, it powers up then clunk, clunk, clunk and powers down. Slot 15 is a global spare.
upload_2018-7-12_11-47-59.png


I am interested in your thoughts now.

Thanks in advance.

BW
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
I guess the cause of the tumbleweeds and crickets here is the appearance of a RAID controller intervening in the story... FreeNAS can't help you when you hide disks from it with a RAID controller in RAID configurations.

If that's the disk that's "missing" from the pool, then it appears you're fresh out of luck, but you could consider the methods I quoted in an earlier post as some kind of last-ditch attempt at getting something out of the pool.

If you really need the data off the disk and your 16th disk is the same type and capacity, you could take the PCB off the good one and see if attaching it to the "bad" one helps (little hope there though, I'm afraid... and voiding any warranty you might have in the process).
 
Elliot Dierksen

Elliot Dierksen

Guru
Joined
Dec 29, 2014
Messages
1,135
In an LSI RAID controller like that, FreeNAS (or any OS) will only be able to see either virtual drives, or drives that are set as JBOD.
 
Status
Not open for further replies.

Similar threads

M
zpool disks showing offline
Replies
1
Views
1K
mroptman
mroptman
T
  • Locked
ZPOOL degraded, hdd failed and so did my attempt to replace
Replies
3
Views
2K
paleoN
paleoN
ghost reaper
  • Locked
zpool unavail, cant import 6TB Data loss
2
Replies
25
Views
4K
jgreco
jgreco
L
  • Locked
Issues with two drives, looking for some help.
Replies
8
Views
1K
lpittman
L
D
  • Locked
ZFS Failure
2
Replies
23
Views
3K
titan_rw
T
Top