Multipath Errors - drive shelf replaced.

[mellman]

Hey All,
Here are my specs:

Dell R720XD
384GB RAM
2x Xeon E5-2670 CPU
HP H220 running LSI firmware
1x NetApp DS4243 shelf
FREENAS 11.1-U7

1 pool (called tank), consists of 3 x 12 disk z3 volumes

Edited the title as an update, latest post has all the relevant info.

Still in need of some assistance here. After further investigation, I determined that something was wrong with my disk shelf (only random disk lights were on, the shelf ID was blank, there was very sporadic information about the disks being reported to freenas)

So I installed all the disks in a new shelf. During bootup I get the attached errors.
Multipath errors - corrupt or invalid GPT / GPT rejected
also secondary GPT table is corrupt or invalid, using the primary only.

In addition to the bad disk shelf, I had some odd behavior with my internal HBA, so decided to move the internal disks into the external shelf. Those disks show up fine - and are the only disk group that is listed when I do a zpool import. groups 1 and 2 do not show up, but the disks do show up when I do a camcontrol devlist all.


I was no longer able to reach the server this morning, including the admin console, could not ping the IP, etc. from the console, i wasn't able to ping the gateway either. I checked my physical cables, replaced them, performed a reboot, rebooted the switches, all to no avail. I ended up doing a "restore default configuration" option from the console. This worked, but upon setup, the pool wasn't loaded. Reviewing the disks in the GUI, all show up but 1 has a blank Disk Size.

when I run zpool import I see this:

Code:

root@freenas:/ # zpool import
   pool: tank
     id: 907423887126384953
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
config:

        tank                                            UNAVAIL  missing device
          raidz3-0                                      DEGRADED
            gptid/e6f85169-5136-11e8-b376-bc305bf48148  ONLINE
            731087118901044337                          UNAVAIL  cannot open
            gptid/181f4f1a-5137-11e8-b376-bc305bf48148  ONLINE
            gptid/24da255f-5137-11e8-b376-bc305bf48148  ONLINE
            gptid/325a8e49-5137-11e8-b376-bc305bf48148  ONLINE
            gptid/4a4df2c1-5137-11e8-b376-bc305bf48148  ONLINE
            gptid/6a52ee8d-5137-11e8-b376-bc305bf48148  ONLINE
            gptid/7bc6a747-5137-11e8-b376-bc305bf48148  ONLINE
            gptid/8d306d6b-5137-11e8-b376-bc305bf48148  ONLINE
            gptid/a82f0ce2-5137-11e8-b376-bc305bf48148  ONLINE
            gptid/bb5df1ae-5137-11e8-b376-bc305bf48148  ONLINE
            gptid/db7bb774-5137-11e8-b376-bc305bf48148  ONLINE
          raidz3-2                                      ONLINE
            gptid/bc7a19f9-c4cb-11e8-b538-bc305bf48148  ONLINE
            gptid/bd626022-c4cb-11e8-b538-bc305bf48148  ONLINE
            gptid/bef543fe-c4cb-11e8-b538-bc305bf48148  ONLINE
            gptid/c0e17dc6-c4cb-11e8-b538-bc305bf48148  ONLINE
            gptid/c27fc9e2-c4cb-11e8-b538-bc305bf48148  ONLINE
            gptid/c365fdbb-c4cb-11e8-b538-bc305bf48148  ONLINE
            gptid/c46b5d52-c4cb-11e8-b538-bc305bf48148  ONLINE
            gptid/c60d3684-c4cb-11e8-b538-bc305bf48148  ONLINE
            gptid/c7a6bc30-c4cb-11e8-b538-bc305bf48148  ONLINE
            gptid/c9416b43-c4cb-11e8-b538-bc305bf48148  ONLINE
            gptid/ca343c33-c4cb-11e8-b538-bc305bf48148  ONLINE
            gptid/cb3f2f2c-c4cb-11e8-b538-bc305bf48148  ONLINE

        Additional devices are known to be part of this pool, though their
        exact configuration cannot be determined.

I have a larger disk I can use to swap the failed one, but before I break things, I'd like to validate my process for getting this back online. I did see a few bug reports related to being unable to import degraded pools into FreeNAS, so I'm hoping that's all that's at play here.

Questions:
1) Do I need to fix the degraded pool before I can import it?
2) If so, is the best way to do that to power down, swap the drive (after validating the correct serial number of the failed drive) and then go in via the GUI to "replace disk"?
3) Why does raidz3-1 not show in zpool import list? Is that what the last blurb is indicating, that it knows they are there but because it's degraded it can't tell me anything else?

 

[styno]

Before you can import & fix your pool you should be able to see all 3 raidz3 devices (they can be in a degraded state).
Start with checking what components are dedicated to the missing raidz3-1, it could be as easy as a faulty cable or missing connection.

 

[mellman]

Before you can import & fix your pool you should be able to see all 3 raidz3 devices (they can be in a degraded state).
Start with checking what components are dedicated to the missing raidz3-1, it could be as easy as a faulty cable or missing connection.

But the disks in that missing z3-1 show up in the gui, that's what confused me here.

 

[mellman]

Any help here? The cables and hba ports seem fine, and the disks show up in the gui.

Do I simply need to replace the failed disks and can someone confirm that's the proper steps to do so?

 

[mellman]

Hi All,

Still in need of some assistance here. After further investigation, I determined that something was wrong with my disk shelf (only random disk lights were on, the shelf ID was blank, there was very sporadic information about the disks being reported to freenas)

So I installed all the disks in a new shelf. During bootup I get the attached errors.
Multipath errors - corrupt or invalid GPT / GPT rejected
also secondary GPT table is corrupt or invalid, using the primary only.

In addition to the bad disk shelf, I had some odd behavior with my internal HBA, so decided to move the internal disks into the external shelf. Those disks show up fine - and are the only disk group that is listed when I do a zpool import. groups 1 and 2 do not show up, but the disks do show up when I do a camcontrol devlist all.

When I've moved shelves before I did not have any issue with the cables being connected differently, so does where the cables plugin to the shelf matter?