Trying to Determine serial number of failed HDD in Pool

[CLEAR RTC]

Hi,

I have one or more failed drives but I am having a hard time figure out how to determine what they are, I have read you can get the serial number off the drive but I am not sure how to find that. I read a few posts but I wonder if it was on an older version because I cannot find the location they specify. I am running FreeNAS 9.3

Code:

Checking status of zfs pools:
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
Pool1         12.6T  3.50T  9.12T         -     9%    27%  1.00x  DEGRADED  /mnt
freenas-boot   464G  2.20G   462G         -      -     0%  1.00x  ONLINE  -

  pool: Pool1
state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
    the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 5h24m with 0 errors on Sun Jan 10 05:24:13 2016
config:

    NAME                                            STATE     READ WRITE CKSUM
    Pool1                                           DEGRADED     0     0     0
      raidz3-0                                      DEGRADED     0     0     0
        gptid/70db6547-b7bf-11e4-9ca6-003048632b1c  ONLINE       0     0     0
        gptid/7152ab76-b7bf-11e4-9ca6-003048632b1c  DEGRADED     0     0 8.98K  too many errors
        gptid/71cf4b22-b7bf-11e4-9ca6-003048632b1c  ONLINE       0     0     0
        gptid/723e8b59-b7bf-11e4-9ca6-003048632b1c  ONLINE       0     0     0
        gptid/72b40e05-b7bf-11e4-9ca6-003048632b1c  ONLINE       0     0     0
        gptid/732b7e78-b7bf-11e4-9ca6-003048632b1c  ONLINE       0     0     0
        gptid/73a24179-b7bf-11e4-9ca6-003048632b1c  ONLINE       0     0     0
      raidz3-1                                      DEGRADED     0     0     0
        gptid/742542e7-b7bf-11e4-9ca6-003048632b1c  ONLINE       0     0     0
        gptid/74a1794f-b7bf-11e4-9ca6-003048632b1c  ONLINE       0     0     0
        gptid/751c3fba-b7bf-11e4-9ca6-003048632b1c  ONLINE       0     0     0
        gptid/7591c998-b7bf-11e4-9ca6-003048632b1c  DEGRADED     0     0  472K  too many errors
        16663824247642782574                        UNAVAIL      0     0     0  was /dev/gptid/760c5097-b7bf-11e4-9ca6-003048632b1c
        gptid/7688afcb-b7bf-11e4-9ca6-003048632b1c  ONLINE       0     0     0
        gptid/76fe2fc9-b7bf-11e4-9ca6-003048632b1c  ONLINE       0     0     0

errors: No known data errors

-- End of daily output --

 

[Jailer]

Click on the storage tab and click the view disks button at the top row of the window.

Those are checksum errors. Check your cabling first before you go any further.

Also list of your hardware, all of it, would help greatly.

 

[Yatti420]

smartctl -a /dev/ada? should give you all the info on the drive in question.. There should be other methods aswell.. 7 drive raid z3 pools? Alot of redundancy there! What kind of case you running?

 

[CLEAR RTC]

Thanks for the quick reply

Click on the storage tab and click the view disks button at the top row of the window.

Those are checksum errors. Check your cabling first before you go any further.

Also list of your hardware, all of it, would help greatly.

I Checked all cables and they all seem properly secure.



Ok so I don't see the drive labelled "16663824247642782574"

Just not seeing how to reconcile the large GPTID or the unavailable drive with an ada number to identify the drive. What am I missing?

System Details:
FreeNAS-9.3-STABLE-201602031011
Intel Xeon E5310
SuperMicro X7DVL-E Motherboard
24GB RAM
SuperMicro SAT2-MV8 Rev 3.0 Controllers
15 Seagate 1TB Drives (Will be replacing failed drives with 1TB WD)

 

[Mirfster]

Edit, nvm... Was posting info already done.

Thought you had 15 disks, but the "Volume Status" is missing a couple?

 

[CLEAR RTC]

Edit, nvm... Was posting info already done.

Thought you had 15 disks, but the "Volume Status" is missing a couple?

Thanks for the reply. However if you check the screen shots all the drives in Volume Status say online and there arent any missing or different than in View disks. Except for the extra entry in Volume Status that says "16663824247642782574" Instead of something like "ada112p" as an example.

 

[CLEAR RTC]

smartctl -a /dev/ada? should give you all the info on the drive in question.. There should be other methods aswell.. 7 drive raid z3 pools? Alot of redundancy there! What kind of case you running?

Hey, I have a SuperMicro chassis, it used to be a CORAID box, but that software was sooo slow. I installed a laptop HDD to run the freenas install.

 

[Yatti420]

I need a new case to support 12+ drives eventually.. Somehow a regular computer case isn't cutting it anymore :).. Did you recap with sanyo (legit?) caps where the drives connect to cables? (backplane?)

 

[Mirfster]

May have physically check to see which disk does NOT match the Serial Numbers shown under "View Disks".

Personally, I keep a record for situations like this, old example (bit too much info recorded):

 

[CLEAR RTC]

May have physically check to see which disk does NOT match the Serial Numbers shown under "View Disks".

Personally, I keep a record for situations like this, old example (bit too much info recorded):
View attachment 10629

I actually saw this while researching my issue and will be creating a similar spreadsheet. I don't think that's to much Data, it looks just right :)

This device is not supposed to be supporting any prod data and isn't "really" but my 3 VM hosts have local storage right now which is being moved to a QNAP, the local disks are being removed and installed in the QNAP (fairly new Intel enterprise SSD's).

So in the mean time I have to move the VMs somewhere during the migration and my FreeNAS test box is the only spot I have that is large enough. The joys of IT where you need Enterprise equipment but have a small business budget.

 

[Mirfster]

I don't think that's to much Data, it looks just right :)

Yeah, but that one contains the Name (ex. DA11 or DA11P2); those can actually change dynamically so you wouldn't really want to depend on them. Best to go by Serial Number.

 

[gpsguy]

You might want to run one of @Bidule0hm 's scripts to display all this information on one page.

Look in his signature for a link to the FreeNAS thread listing the scripts.


Sent from my phone

 

[Bidule0hm]

Just not seeing how to reconcile the large GPTID or the unavailable drive with an ada number to identify the drive. What am I missing?

As gpsguy said you may want to use my script, look at the "Useful Scripts" link in my signature ;)

 

[random003]

Use "glabel status" to correlate gptid to drive name. Then you can use smartctl to find the serial name from the drive name. Below is an example.

Code:

        NAME                                                STATE     READ WRITE CKSUM
        tank                                                ONLINE       0     0     0
          raidz3-0                                          ONLINE       0     0     0
            gptid/f5aa938b-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/f5f0eca7-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/f62bdcb9-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/f66959f4-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/f6b49d1a-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/f6f53882-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/275116e0-a5eb-11e5-9af4-002590f0651a.eli  ONLINE       0     0     0
            gptid/f79a86d7-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/f7eda8f1-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/f84fb4de-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/f98b3547-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
          raidz3-1                                          ONLINE       0     0     0
            gptid/fd7ca0b8-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/fede538a-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/ff331c03-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/ff84f64e-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/ffc7e57e-334d-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/000c4634-334e-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/013377e1-334e-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/02731a9c-334e-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/02c4337e-334e-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/0317bd15-334e-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0
            gptid/036539af-334e-11e5-b15f-002590f0651a.eli  ONLINE       0     0     0


Let's pretend that I want to replace the first drive in raidz3-0 with the following gptid.

gptid/f5aa938b-334d-11e5-b15f-002590f0651a


We need to correlate gptid to the name of the drive. Glabel status will give you the output of all the drives but it's annoying to sort through all the info so we use grep to find the drive we are looking for.


[root@nas] ~# glabel status | grep gptid/fd7ca0b8-334d-11e5-b15f-002590f0651a
gptid/fd7ca0b8-334d-11e5-b15f-002590f0651a     N/A  da5p1


Then we can use smartctl to find the serial number of da5.

[root@nas] ~# smartctl -a /dev/da5 | grep Serial
Serial Number:    ML4220F312XR3K

Unfortunately for drives that are completely dead and no longer respond to the system (like 16663824247642782574), you need to make a list of serial numbers of all the drives that are online then by process of elimination find the drive that is not on your list. Or if your chassis has activity led for each drive you can put some load on the array and see which drive lacks activity. For most accurate results I would do a combination these things.

 

[CLEAR RTC]

Use "glabel status" to correlate gptid to drive name. Then you can use smartctl to find the serial name from the drive name. Below is an example.

Code:

Let's pretend that I want to replace the first drive in raidz3-0 with the following gptid.

gptid/f5aa938b-334d-11e5-b15f-002590f0651a


We need to correlate gptid to the name of the drive. Glabel status will give you the output of all the drives but it's annoying to sort through all the info so we use grep to find the drive we are looking for.


[root@nas] ~# glabel status | grep gptid/fd7ca0b8-334d-11e5-b15f-002590f0651a
gptid/fd7ca0b8-334d-11e5-b15f-002590f0651a     N/A  da5p1


Then we can use smartctl to find the serial number of da5.

[root@nas] ~# smartctl -a /dev/da5 | grep Serial
Serial Number:    ML4220F312XR3K

Unfortunately for drives that are completely dead and no longer respond to the system (like 16663824247642782574), you need to make a list of serial numbers of all the drives that are online then by process of elimination find the drive that is not on your list. Or if your chassis has activity led for each drive you can put some load on the array and see which drive lacks activity. For most accurate results I would do a combination these things.

So even though 16663824247642782574 was dead it showed me its old gptid. I have the serial numbers and locations of all the bad drives.

Thank you to everyone who commented and helped.

I am about to place new drives in the freenas box, once they are there are there anything I need to do to initiate a rebuild or will it start automatically?

TIA

Carl

 

[Mirfster]

I am about to place new drives in the freenas box, once they are there are there anything I need to do to initiate a rebuild or will it start automatically?

In "Volume Manager", you would need to select the missing drive and choose [Replace]; then select the new drive. It will handle the rest from there (re-silver). Wait until one is competed its re-silvering before replacing another (if needed) if they are in the same vdev (at least I would, but you have a RaidZ3...).

 

[gpsguy]

Please read the directions in the documentation regarding disk replacement.

 

[CLEAR RTC]

In "Volume Manager", you would need to select the missing drive and choose [Replace]; then select the new drive. It will handle the rest from there (re-silver). Wait until one is competed its re-silvering before replacing another (if needed) if they are in the same vdev (at least I would, but you have a RaidZ3...).

Thanks dude. The pool is now resilvering. Replacing disks was slick and easy. Thanks again for all your help.

 

[CLEAR RTC]

I need a new case to support 12+ drives eventually.. Somehow a regular computer case isn't cutting it anymore :).. Did you recap with sanyo (legit?) caps where the drives connect to cables? (backplane?)

To answer your second question, no I haven't had them recapped, wasn't aware it was an issue with the SM backplanes, have you heard of issues?

 

[Yatti420]

Nope if they are original SM stuff those will be good caps..

To answer your second question, no I haven't had them recapped, wasn't aware it was an issue with the SM backplanes, have you heard of issues?