ZFS Degraded - Drive Shows Unavailable

[ericv]

Ok, I think I know what I need to do, but I’d figure I’d ask the experts to be sure. Most of the text below is out from smartctl. Just looking to know if I should be doing something else before I swap out the disk tomorrow night...

I believe I should RMA my disk and get a new one.

That said...

I got an email this morning stating the following with no other details.

“The volume Storage (ZFS) state is DEGRADED: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state."

I checked my server storage tab for the disks and didn’t see any issues (not sure if I should here).

--------------------------------------------------------------------------------------------------

I then ran the following command: “zpool status -v” and got the following results:

pool: Storage
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: none requested
config:

NAME STATE READ WRITE CKSUM
Storage DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/b3fd6950-cffe-11e4-87af-0cc47a49c8de ONLINE 0 0 0
gptid/b4466aee-cffe-11e4-87af-0cc47a49c8de ONLINE 0 0 0
gptid/b4919038-cffe-11e4-87af-0cc47a49c8de ONLINE 0 0 0
7009220469486630142 UNAVAIL 0 0 0 was /dev/gptid/b4dc117a-cffe-11e4-87af-0cc47a49c8de
gptid/b521019e-cffe-11e4-87af-0cc47a49c8de ONLINE 0 0 0
gptid/b566e33b-cffe-11e4-87af-0cc47a49c8de ONLINE 0 0 0

errors: No known data errors

--------------------------------------------------------------------------------------------------

Now after about an hour there is a drive missing from the list in the GUI

I am assuming based on the following image I need to replace ada7, correct?

 

[BigDave]

Z303A73M is the serial number that seems to be missing.
That drive ***WAS*** listed as ada5, but you should only
pay attention to serial numbers.
Follow the FreeNAS manual for drive replacement

 

[ericv]

Thanks, I'll be headed over to the local Microcenter to pick up a new drive tomorrow. I'd do it today but they are closed for the holiday. These drives are only 20-30 days old. I purchased 6 from 3 different stores so they wouldn't all be from the same batch... 2xAmazon, 2xNewegg, 2xMicrocenter.

 

[DataKeeper]

I'd pickup 2 drives just in case.

 

[ericv]

I'd pickup 2 drives just in case.

Probably a good idea. They only have 1 in stock. I'll probably get the other from Amazon.

 

[BigDave]

FYI, I have 6 - 3TB WD Red drives purchased from my local Microcenter.
I've never had one bit of trouble with them.
I don't like buying drives from e-tailers that ship long distances, due to the beating
the containers take during transit. You pay for a new drive, it gets trashed,
then you RMA the drive and receive a refurbished one in exchange

 

[ericv]

FYI, I have 6 - 3TB WD Red drives purchased from my local Microcenter.
I've never had one bit of trouble with them.
I don't like buying drives from e-tailers that ship long distances, due to the beating
the containers take during transit. You pay for a new drive, it gets trashed,
then you RMA the drive and receive a refurbished one in exchange

I love living by a Microcenter. I just moved to Maryland, before in Pennsylvania I didn't have one.

 

[Ericloewe]

I really wish we had something like Microcenter around here.

A couple of years ago, I had the chance to go to the one in Boston. It really put the local crummy computer parts store to shame.

FYI, I have 6 - 3TB WD Red drives purchased from my local Microcenter.
I've never had one bit of trouble with them.
I don't like buying drives from e-tailers that ship long distances, due to the beating
the containers take during transit. You pay for a new drive, it gets trashed,
then you RMA the drive and receive a refurbished one in exchange

At least in Europe, Amazon ships drives (at least WD drives) in individual cardboard boxes, suspended by plastic shock absorbers. Never had a problem with them.

 

[danb35]

I don't like buying drives from e-tailers that ship long distances, due to the beating the containers take during transit. You pay for a new drive, it gets trashed, then you RMA the drive and receive a refurbished one in exchange

One nice thing about Amazon is that if your product is defective within their return window, they'll advance exchange a new disk (and pay for return shipping) rather than making you deal with the manufacturer.

 

[BigDave]

At least in Europe, Amazon ships drives (at least WD drives) in individual cardboard boxes, suspended by plastic shock absorbers. Never had a problem with them.

Good packaging and a 30 day return policy makes a difference for sure. I will always make a local trip
for a retail purchase when the choice is mine.
I'll go with the retailer that receives they're drives sealed in a factory packed shipping carton that is
way to heavy for a package delivery service to launch through the air :D

One nice thing about Amazon is that if your product is defective within their return window, they'll advance exchange a new disk (and pay for return shipping) rather than making you deal with the manufacturer.

 

[ericv]

I'll also report that I tried reseating the power connector and replacing the SATA cable with no luck.

BIOS also not detecting the drive.

I hope it is the drive and not the SATA port...

 

[BigDave]

Easy enough to test before buying a new hard drive.
Shut down your server and switch data connections with another drive.
If it's the port, the other drive will not show in the bios...

 

[ericv]

Plugged another drive I had in the port and the BIOS saw the device. Sounds like the drive is the issue.

I can't stress the importance of marking both sides of the cables so you can tell which drive is connected to what port on the motherboard. I have my cable hidden pretty well, I would have hated to undo all that to replace the drive. Also, labeling the drives so that the serial number is visible was also a good move. I'm glad I did both.

 

[BigDave]

If your pool is healthy, you could shut off your server and switch all the drives data cables.
ZFS has all the id numbers in memory/meta and would not skip a beat upon reboot.
Your drives can sometimes change "order" from just rebooting for an update.
This is why it's so important to keep track of the disk's serial number so you can replace
the proper disk that's failing and/or failed.

 

[ericv]

If your pool is healthy, you could shut off your server and switch all the drives data cables.
ZFS has all the id numbers in memory/meta and would not skip a beat upon reboot.
Your drives can sometimes change "order" from just rebooting for an update.
This is why it's so important to keep track of the disk's serial number so you can replace
the proper disk that's failing and/or failed.

I didn't realize that. I also didn't know enough to want to risk it so I used a new drive. I guess it would have to work considering you can import a pool.

 

[ericv]

Ok, drive replaced. I ended up ordering 2 drives. 1 from Amazon and 1 from TigerDirect as Amazon only had 1. Look how the drive from TigerDirect came. I already did the RMA.

Tons of packaging... Yea right...