Degraded RAID-10 Array

Status
Not open for further replies.

loknar28

Dabbler
Joined
Mar 25, 2017
Messages
23
I have been running FreeNAS 9.10 containing a RAID10 volume comprised of six 3TiB WD datacenter drives for about a year now. A few days ago I saw that one of the mirrors had degraded causing the array to degrade. So I added a new 7th disk and instructed freenas through the GUI to replace the drive. Just after selecting the new drive in the presented drop down menu the Webportal went offline. However, I saw that the array had become very active, so assuming that the mirror was replicating to the new drive I left the system alone for awhile.
When I came back to the array about a day later the portal was still unavailable, so I ran 'zpool status' from the cli which provided the following output. Looking at this output it a appears I now have two which have been removed from the mirror sets.
Code:
[root@freenas] ~# zpool status
  pool: UberStore
 state: DEGRADED
status: One or more devices has been removed by the administrator.
   Sufficient replicas exist for the pool to continue functioning in a
   degraded state.
action: Online the device using 'zpool online' or replace the device with
   'zpool replace'.
  scan: resilvered 237G in 2h9m with 0 errors on Sun Dec  3 20:28:37 2017
config:

   NAME											STATE	 READ WRITE CKSUM
   UberStore									   DEGRADED	 0	 0	 0
	 mirror-0									  DEGRADED	 0	 0	 0
	   gptid/d90afcc5-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   1235246432796506539						 REMOVED	  0	 0	 0  was /dev/gptid/d9a3adfe-19ae-11e7-b492-d05099c1cb7b
	 mirror-1									  DEGRADED	 0	 0	 0
	   4075242487269000551						 REMOVED	  0	 0	 0  was /dev/gptid/dcba356c-19ae-11e7-b492-d05099c1cb7b
	   gptid/dd6c9ecd-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	 mirror-2									  ONLINE	   0	 0	 0
	   gptid/de0d95b2-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/54304a63-d880-11e7-bc9b-d05099c1cb7b  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Dec  4 03:45:19 2017
config:

   NAME		STATE	 READ WRITE CKSUM
   freenas-boot  ONLINE	   0	 0	 0
	 ada5p2	ONLINE	   0	 0	 0

errors: No known data errors


I have been researching this hoping I would not need to bother the community but I really could use some help.
Additionally, I am fine proceeding with the cli for now but at some point I would like to get the GUI running again. I am completely unfamiliar with nginx, so I will probably need help with that as well.
 
Last edited:

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
First, you should back this pool up immediately. A single additional drive loss has a 66.67% chance in rendering your pool unrecoverable.
The GUI going unavailable is weird... someone more knowledgeable will have to comment on that one. Honestly, I'd just reboot the thing... AFTER you back it up.

As far as what's going on, it appears that the 7th drive was never added to anything. You haven't provided any system information, so it's hard to know what you're working with... but I would be somewhat suspicious of a hardware failure. List your hardware specs (per forum rules) and that'll give people a better clue.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
GUI non response would be explained by it being swapped out to the failed disk
 

loknar28

Dabbler
Joined
Mar 25, 2017
Messages
23
Thank you for the quick reply. As requested here are my system specs.....See signature block..
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
That's far from the full specs. Are you using an HBA or connecting directly to the MB? Case? Power supply? Do you have enough headroom on your power supply to accommodate adding that 7th drive?
 

ethereal

Guru
Joined
Sep 10, 2012
Messages
762
Thank you for the quick reply. As requested here are my system specs.....See signature block..
many people using phones cannot see the signature - always better to provide full system specs in the post
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
many people using phones cannot see the signature - always better to provide full system specs in the post
Yes. When asking for help that benefits from the system specs, always include them explicitly in your posts.

It helps those on mobile see signatures (even if phablets can often use landscape to get the desktop layout, it's a pain in the ass) and it doesn't ruin threads down the line after you change out your system.
 

rs225

Guru
Joined
Jun 28, 2014
Messages
878
Does each drive have it's own power connection, or are any of them using splitters?
 

loknar28

Dabbler
Joined
Mar 25, 2017
Messages
23
I am using a case and powersupply designed for this purpose. The Silverstone DS380 chassis which has a backplane 8 hotswap bays. The power supply used is a 300W Silverstone ST30SF.
Based on the drive specs this should be enough since each drive is rated for 10-11Watts
Average power requirements (W)
Sequential read = 10.2
Sequential write =9.7
Random read/write = 10.9
Idle = 8.6

The Avoton CPU is soldered to the motherboard
 
Last edited:
Joined
May 10, 2017
Messages
838
The drives will require the most power during spin-up, still and since it's got a 25A single 12v rail it's enough, though I didn't check the specs each disk shouldn't need more than 2/2.5A for spin-up, but not a lot to spare.
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
300-watt supply? I think you've found your problem. Read jgreco's treatise on power supply sizing here:
https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/

In short, I suspect 6 drives was pushing your power supply to the max. You added #7, and have now introduced a heisen-bug.

I would:
A. back up your pool - you're playing a dangerous game here with already-limited redundancy
B. replace your power supply with something more properly sized... Seasonic being the hands-down favorite
C. restart your troubleshooting
 

loknar28

Dabbler
Joined
Mar 25, 2017
Messages
23
For those of you who are unable to see my signature I am posting a copy of it here.
FreeNAS 9.10
ASRock C2750D4I AVOTON
Crucial 16GB Kit (8GBx2) DDR3/DDR3L-1600MT/s (PC3-12800) DR x8 ECC UDIMM
6 x WD Re Datacenter drives(WD3000FYYZ)
Silverstone DS380 Case w/300w Silverstone ST30SF power supply

I have a Silverstone SFX 650SX-G on the way.
 
Last edited:

loknar28

Dabbler
Joined
Mar 25, 2017
Messages
23
So I replaced the power supply and restarted the NAS. This seems to have fixed everything but the 7th disk that I added which is not part of the pool is not visible. If I want to make it a hotspare for the array do I have to do a zpool import first?

Code:
pool: UberStore
 state: ONLINE
  scan: resilvered 8.09G in 0h8m with 0 errors on Mon Dec 11 22:39:00 2017
config:

   NAME											STATE	 READ WRITE CKSUM
   UberStore									   ONLINE	   0	 0	 0
	 mirror-0									  ONLINE	   0	 0	 0
	   gptid/d90afcc5-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/d9a3adfe-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	 mirror-1									  ONLINE	   0	 0	 0
	   gptid/dcba356c-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/dd6c9ecd-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	 mirror-2									  ONLINE	   0	 0	 0
	   gptid/de0d95b2-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/54304a63-d880-11e7-bc9b-d05099c1cb7b  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Dec  4 03:45:19 2017
config:

   NAME		STATE	 READ WRITE CKSUM
   freenas-boot  ONLINE	   0	 0	 0
	 ada5p2	ONLINE	   0	 0	 0

errors: No known data errors
 

loknar28

Dabbler
Joined
Mar 25, 2017
Messages
23
Here are all of the disks. I am not clear on how to identify which of these is not part of the pool in the previous post.

Code:
[root@freenas] ~# camcontrol devlist
<WDC WD3000FYYZ-01UL1B1 01.01K02>  at scbus0 target 0 lun 0 (pass0,ada0)
<WDC WD3000FYYZ-01UL1B1 01.01K02>  at scbus1 target 0 lun 0 (pass1,ada1)
<WD3000FYYX 00.0D1K4>			  at scbus2 target 0 lun 0 (pass2,ada2)
<WDC WD3000FYYZ-01UL1B0 01.01K01>  at scbus3 target 0 lun 0 (pass3,ada3)
<WD3000FYYX 00.0D1K4>			  at scbus5 target 0 lun 0 (pass4,ada4)
<Marvell Console 1.01>			 at scbus9 target 0 lun 0 (pass5)
<SAMSUNG SSD UM410 Series 2.5" 16GB VAM12D1Q>  at scbus10 target 0 lun 0 (pass6,ada5)
<WDC WD3000FYYZ-01UL1B0 01.01K01>  at scbus14 target 0 lun 0 (pass7,ada6)
<WDC WD3000FYYZ-01UL1B0 01.01K01>  at scbus15 target 0 lun 0 (pass8,ada7)
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
If I want to make it a hotspare for the array do I have to do a zpool import first?
That's a weird question. The pool you're adding a spare to has to be online during the process, yes.

I am not clear on how to identify which of these is not part of the pool in the previous post.
There are some documents in the Resources section that should help.
 

loknar28

Dabbler
Joined
Mar 25, 2017
Messages
23
That's a weird question. The pool you're adding a spare to has to be online during the process, yes.
The pool is already online. I'm asking about adding the extra disk to the pool. Then hopefully having it designated as a hot spare. Maybe zpool import is not the command to be referencing. I have been searching in the documentation(RTFM) but not finding anything that clearly explains what I am looking to accomplish. I realize freeNAS works differently than a typical Hardware RAID NAS so I may just not be understanding how hotspares are defined.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
I was going to link the manual section, but it turns out that it doesn't actually tell you how to add a hot spare. Someone who's done it before will probably show up and tell you the exact process, so that you don't have to play around looking for it.

Maybe zpool import is not the command to be referencing.
No. The command would be zpool add -n poolname spare driveID, but don't do that or you'll make FreeNAS sad.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
Someone who's done it before will probably show up and tell you the exact process,
I haven't done it, but I'm 99% sure it's just another form of extending a pool. Volume Manager, select pool to extend, add the disk, select spare from the drop-down, and you're good.
 

loknar28

Dabbler
Joined
Mar 25, 2017
Messages
23
select spare from the drop-down, and you're good.
Nice...That is exactly what I needed!!
Code:
[root@freenas] ~# zpool status -v
  pool: UberStore
 state: ONLINE
  scan: resilvered 8.09G in 0h8m with 0 errors on Mon Dec 11 22:39:00 2017
config:

   NAME											STATE	 READ WRITE CKSUM
   UberStore									   ONLINE	   0	 0	 0
	 mirror-0									  ONLINE	   0	 0	 0
	   gptid/d90afcc5-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/d9a3adfe-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	 mirror-1									  ONLINE	   0	 0	 0
	   gptid/dcba356c-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/dd6c9ecd-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	 mirror-2									  ONLINE	   0	 0	 0
	   gptid/de0d95b2-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/54304a63-d880-11e7-bc9b-d05099c1cb7b  ONLINE	   0	 0	 0
   spares
	 gptid/43326693-dfb4-11e7-b8eb-d05099c1cb7b	AVAIL   

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Dec  4 03:45:19 2017
config:

   NAME		STATE	 READ WRITE CKSUM
   freenas-boot  ONLINE	   0	 0	 0
	 ada5p2	ONLINE	   0	 0	 0

errors: No known data errors
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
I haven't done it, but I'm 99% sure it's just another form of extending a pool. Volume Manager, select pool to extend, add the disk, select spare from the drop-down, and you're good.
So was I, but I like to steer people away from the Volume Manager (such a horrible name, too) unless I'm certain that's what they need.

Anyway, there's a docs bug for this now. I'm on mobile, so no link until the morning.
 
Status
Not open for further replies.
Top