Multipath Corrupted Multiple Disks and Possibly Destroyed Pool - Please Help

[Brosif_My_Nif]

THE PROBLEM: I had a perfectly running volume for 3+ years that may be lost due to Sata drives being introduced to a brand new Supermicro SAS3 multipath, dual expander backplane environment. Any assistance that can be offered would be greatly appreciated.

OLD SERVER:

• FreeNAS-11.1-U6
• Norco 4224 Chassis
• Supermicro X10SRH-CLN4F motherboard
• Xeon E5-1630v3
• 196 GB Samsung DDR4 ECC RAM
• LSI 9201-16i

VOLUME 1 (21 Hard Drives):
The volume was made up of 3 VDEV's (7-Disk, RAID-Z3).

• x7 - WD SE 4TB Sata HDD's (Raid Z-3) VDEV1
  
• x7 - WD RED 8TB Sata HDD's (Raid Z-3) VDEV2
• x7 - WD RED 8TB Sata HDD's (Raid Z-3) VDEV3

These Sata drives have been running perfectly for the past 3+ years without any problems in the Norco 4224 chassis.
This Norco chassis was not handling cooling well, so I wanted to upgrade to a Supermicro Chassis. I also needed to add additional storage as I was nearly at 80% capacity of my existing volume.


UPGRADED SERVER:

• FreeNAS-11.1-U6
• Supermicro X10SRH-CLN4F motherboard
• Xeon E5-1630v3
• 196 GB Samsung DDR4 ECC RAM
• All 21 Sata drives from old volume

• NEW: Supermicro SuperChassis 847BE2C-R1K28LPB (FRONT: 24-port SAS3 dual-expander backplane + BACK: 12-port SAS3 dual-expander backplane)
  
• NEW: VOLUME 2, x7 - HGST 12TB SAS HDD's (Raid Z-3)
• NEW: LSI 9305-24I
• NEW: LSI 9300-8I (Onboard the Supermicro X10SRH-CLN4F motherboard)

I settled on the Supermicro SuperChassis 847BE2C-R1K28LPB because I liked the idea of multipath access (for redundancy) to some new 12TB HGST SAS drives I purchased. I transplanted my reliable Supermicro X10SRH-CLN4F motherboard, Xeon E5-1630v3, and 196 GB Samsung DDR4 ECC RAM into the new chassis and started testing the new 12TB HGST SAS drives with the motherboard's onboard 3008 based LSI HBA and another LSI 9305-24i HBA I purchased for this build. I updated the firmware on both of these LSI SAS3 HBA's to 16.00.01.00. Everything looked great, multipath worked awesome during my torture testing with these new HGST 12TB SAS drives.

This probably is not important, but I wired the HBA's to the front backplane and then to the rear backplane exactly as it is described in this screenshot below. When I only had the new HGST 12TB SAS drives in this box, it was showing 4 connections to each drive. 3 being Passive and 1 being active. It worked perfectly.





Then I proceeded to move my old pool of 21 sata disks into the new chassis. This is where the problems began.

I was doing this testing on a fresh installation of FreeNAS-11.1-U6. After placing the 21 sata disks into the new SAS3 dual-expander backplane of the Supermicro 847, and powering on the machine, I was met with a system alert. See image below.

I shut down the machine, removed the new 12TB HGST SAS disks that were still being tested and had no data on them, to make sure I was only dealing with my 21 old sata disks.

Upon reboot, the same error popped up again. So I immediately went to the "View Multipaths" tab and found some of the Sata drives were detecting multipath access and some were not. I did not expect the Sata drives to have ANY multipath access because I thought that only SAS drives had multipath capabilities. I was assuming that the old Sata drives would not even show up in the "View Multipaths" tab of the webgui. I thought it was strange that some of the drives were showing "Optimal" while some disks were showing as "Degraded".



MAJOR PROBLEM:
multipath/disk X: corrupt or invalid GPT detected.
multipath/disk X: GPT rejected -- may not be recoverable.

Multiple drives are showing GPT problems.

I then tried to import my existing volume that has been working great for 3+ years on my old hardware, but no volume is recognized.

If anyone can provide any assistance, it would truly mean the world to me.



Please see images below...

Code:

Oct 19 21:22:40 freenas GEOM_MULTIPATH: disk1 created
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da0 added to disk1
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da0 is now active path in disk1
Oct 19 21:22:40 freenas GEOM: multipath/disk1: corrupt or invalid GPT detected.
Oct 19 21:22:40 freenas GEOM: multipath/disk1: GPT rejected -- may not be recoverable.
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da21 added to disk1
Oct 19 21:22:40 freenas GEOM_MULTIPATH: disk2 created
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da1 added to disk2
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da1 is now active path in disk2
Oct 19 21:22:40 freenas GEOM: multipath/disk2: corrupt or invalid GPT detected.
Oct 19 21:22:40 freenas GEOM: multipath/disk2: GPT rejected -- may not be recoverable.
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da22 added to disk2
Oct 19 21:22:40 freenas GEOM_MULTIPATH: disk3 created
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da2 added to disk3
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da2 is now active path in disk3
Oct 19 21:22:40 freenas GEOM: multipath/disk3: corrupt or invalid GPT detected.
Oct 19 21:22:40 freenas GEOM: multipath/disk3: GPT rejected -- may not be recoverable.
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da23 added to disk3
Oct 19 21:22:40 freenas GEOM_MULTIPATH: disk4 created
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da3 added to disk4
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da3 is now active path in disk4
Oct 19 21:22:40 freenas GEOM: multipath/disk4: corrupt or invalid GPT detected.
Oct 19 21:22:40 freenas GEOM: multipath/disk4: GPT rejected -- may not be recoverable.
Oct 19 21:22:40 freenas GEOM_MULTIPATH: da24 added to disk4
Oct 19 21:22:41 freenas GEOM_MULTIPATH: disk5 created
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da4 added to disk5
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da4 is now active path in disk5
Oct 19 21:22:41 freenas GEOM: multipath/disk5: corrupt or invalid GPT detected.
Oct 19 21:22:41 freenas GEOM: multipath/disk5: GPT rejected -- may not be recoverable.
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da25 added to disk5
Oct 19 21:22:41 freenas GEOM_MULTIPATH: disk6 created
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da5 added to disk6
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da5 is now active path in disk6
Oct 19 21:22:41 freenas GEOM: multipath/disk6: corrupt or invalid GPT detected.
Oct 19 21:22:41 freenas GEOM: multipath/disk6: GPT rejected -- may not be recoverable.
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da26 added to disk6
Oct 19 21:22:41 freenas GEOM_MULTIPATH: disk7 created
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da6 added to disk7
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da6 is now active path in disk7
Oct 19 21:22:41 freenas GEOM: multipath/disk7: corrupt or invalid GPT detected.
Oct 19 21:22:41 freenas GEOM: multipath/disk7: GPT rejected -- may not be recoverable.
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da27 added to disk7
Oct 19 21:22:41 freenas GEOM_MULTIPATH: disk8 created
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da7 added to disk8
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da7 is now active path in disk8
Oct 19 21:22:41 freenas GEOM: multipath/disk8: corrupt or invalid GPT detected.
Oct 19 21:22:41 freenas GEOM: multipath/disk8: GPT rejected -- may not be recoverable.
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da28 added to disk8
Oct 19 21:22:41 freenas GEOM_MULTIPATH: disk9 created
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da8 added to disk9
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da8 is now active path in disk9
Oct 19 21:22:41 freenas GEOM: multipath/disk9: corrupt or invalid GPT detected.
Oct 19 21:22:41 freenas GEOM: multipath/disk9: GPT rejected -- may not be recoverable.
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da29 added to disk9
Oct 19 21:22:41 freenas GEOM_MULTIPATH: disk10 created
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da9 added to disk10
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da9 is now active path in disk10
Oct 19 21:22:41 freenas GEOM: multipath/disk10: corrupt or invalid GPT detected.
Oct 19 21:22:41 freenas GEOM: multipath/disk10: GPT rejected -- may not be recoverable.
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da30 added to disk10
Oct 19 21:22:41 freenas GEOM_MULTIPATH: disk11 created
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da10 added to disk11
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da10 is now active path in disk11
Oct 19 21:22:41 freenas GEOM: multipath/disk11: corrupt or invalid GPT detected.
Oct 19 21:22:41 freenas GEOM: multipath/disk11: GPT rejected -- may not be recoverable.
Oct 19 21:22:41 freenas GEOM_MULTIPATH: da31 added to disk11
Oct 19 21:22:42 freenas GEOM_MULTIPATH: disk12 created
Oct 19 21:22:42 freenas GEOM_MULTIPATH: da11 added to disk12
Oct 19 21:22:42 freenas GEOM_MULTIPATH: da11 is now active path in disk12
Oct 19 21:22:42 freenas GEOM: multipath/disk12: corrupt or invalid GPT detected.
Oct 19 21:22:42 freenas GEOM: multipath/disk12: GPT rejected -- may not be recoverable.
Oct 19 21:22:42 freenas GEOM_MULTIPATH: disk13 created
Oct 19 21:22:42 freenas GEOM_MULTIPATH: da12 added to disk13
Oct 19 21:22:42 freenas GEOM_MULTIPATH: da12 is now active path in disk13
Oct 19 21:22:42 freenas GEOM: multipath/disk13: corrupt or invalid GPT detected.
Oct 19 21:22:42 freenas GEOM: multipath/disk13: GPT rejected -- may not be recoverable.
Oct 19 21:22:42 freenas GEOM_MULTIPATH: disk14 created
Oct 19 21:22:42 freenas GEOM_MULTIPATH: da13 added to disk14
Oct 19 21:22:42 freenas GEOM_MULTIPATH: da13 is now active path in disk14
Oct 19 21:22:42 freenas GEOM: multipath/disk14: corrupt or invalid GPT detected.
Oct 19 21:22:42 freenas GEOM: multipath/disk14: GPT rejected -- may not be recoverable.
Oct 19 21:22:42 freenas GEOM_MULTIPATH: disk15 created
Oct 19 21:22:42 freenas GEOM_MULTIPATH: da14 added to disk15
Oct 19 21:22:42 freenas GEOM_MULTIPATH: da14 is now active path in disk15
Oct 19 21:22:42 freenas GEOM: multipath/disk15: corrupt or invalid GPT detected.
Oct 19 21:22:42 freenas GEOM: multipath/disk15: GPT rejected -- may not be recoverable.
Oct 19 21:22:42 freenas GEOM_MULTIPATH: disk16 created
Oct 19 21:22:42 freenas GEOM_MULTIPATH: da15 added to disk16
Oct 19 21:22:42 freenas GEOM_MULTIPATH: da15 is now active path in disk16
Oct 19 21:22:42 freenas GEOM: multipath/disk16: corrupt or invalid GPT detected.
Oct 19 21:22:42 freenas GEOM: multipath/disk16: GPT rejected -- may not be recoverable.
Oct 19 21:22:43 freenas GEOM_MULTIPATH: disk17 created
Oct 19 21:22:43 freenas GEOM_MULTIPATH: da16 added to disk17
Oct 19 21:22:43 freenas GEOM_MULTIPATH: da16 is now active path in disk17
Oct 19 21:22:43 freenas GEOM: multipath/disk17: corrupt or invalid GPT detected.
Oct 19 21:22:43 freenas GEOM: multipath/disk17: GPT rejected -- may not be recoverable.
Oct 19 21:22:43 freenas GEOM_MULTIPATH: disk18 created
Oct 19 21:22:43 freenas GEOM_MULTIPATH: da17 added to disk18
Oct 19 21:22:43 freenas GEOM_MULTIPATH: da17 is now active path in disk18
Oct 19 21:22:43 freenas GEOM: multipath/disk18: corrupt or invalid GPT detected.
Oct 19 21:22:43 freenas GEOM: multipath/disk18: GPT rejected -- may not be recoverable.
Oct 19 21:22:43 freenas GEOM_MULTIPATH: disk19 created
Oct 19 21:22:43 freenas GEOM_MULTIPATH: da18 added to disk19
Oct 19 21:22:43 freenas GEOM_MULTIPATH: da18 is now active path in disk19
Oct 19 21:22:43 freenas GEOM: multipath/disk19: corrupt or invalid GPT detected.
Oct 19 21:22:43 freenas GEOM: multipath/disk19: GPT rejected -- may not be recoverable.
Oct 19 21:22:43 freenas GEOM_MULTIPATH: disk20 created
Oct 19 21:22:43 freenas GEOM_MULTIPATH: da19 added to disk20
Oct 19 21:22:43 freenas GEOM_MULTIPATH: da19 is now active path in disk20
Oct 19 21:22:43 freenas GEOM: multipath/disk20: corrupt or invalid GPT detected.
Oct 19 21:22:43 freenas GEOM: multipath/disk20: GPT rejected -- may not be recoverable.
Oct 19 21:22:43 freenas GEOM_MULTIPATH: disk21 created
Oct 19 21:22:43 freenas GEOM_MULTIPATH: da20 added to disk21
Oct 19 21:22:43 freenas GEOM_MULTIPATH: da20 is now active path in disk21
Oct 19 21:22:43 freenas GEOM: multipath/disk21: corrupt or invalid GPT detected.
Oct 19 21:22:43 freenas GEOM: multipath/disk21: GPT rejected -- may not be recoverable.

 

[sretalla]

I'm not sure how experimental you're willing to be, but in my experience with multipath, I was just able to destroy all of the multipath disks ( gmultipath destroy disk1 ...etc) and everything worked normally from there.

I ended up adding a tunable to unload the gmultipath kernel module at startup to prevent it from trying to create multipath disks every startup.

 

[mav@]

Your problem here is mostly not in SATA drives, but in fact that you should not reconfigure disks with existing pool into multipath. GEOM_MULTIPATH code stores its metadata at the last sector of each disk to make sure it is really a multipath. The problem is that the same sector is used by the second copy of GPT partition table. Unless you did some extra destructive actions, you should be able to get data back by deconstructing multipath cabling and destroying gmultipath instances with `gmultipath destroy`, as recommended above.

It is not a problem to connect SATA disk to SAS backplane, one expander will simply not see them. Multipath appeared here since you used two HBAs connected to each expander. If you have two HBAs, I would recommend you to just dedicate each of them to its own JBOD instead of cascading them. It would simplify SAS fabric operation and made system more robust even though seems like having more single points of failure.

 

[Brosif_My_Nif]

I'm not sure how experimental you're willing to be, but in my experience with multipath, I was just able to destroy all of the multipath disks ( gmultipath destroy disk1 ...etc) and everything worked normally from there.

I ended up adding a tunable to unload the gmultipath kernel module at startup to prevent it from trying to create multipath disks every startup.

I should have added a tunable to unload gmultipath before moving the old volume of SATA disks into the multipath environment. That's very good advice. Thank you. I am going to try gmultipath destroy as you suggested and hope that I can recover my pool. Thank you again for taking the time to respond and offer assistance!

 

[Brosif_My_Nif]

Your problem here is mostly not in SATA drives, but in fact that you should not reconfigure disks with existing pool into multipath. GEOM_MULTIPATH code stores its metadata at the last sector of each disk to make sure it is really a multipath. The problem is that the same sector is used by the second copy of GPT partition table. Unless you did some extra destructive actions, you should be able to get data back by deconstructing multipath cabling and destroying gmultipath instances with `gmultipath destroy`, as recommended above.

It is not a problem to connect SATA disk to SAS backplane, one expander will simply not see them. Multipath appeared here since you used two HBAs connected to each expander. If you have two HBAs, I would recommend you to just dedicate each of them to its own JBOD instead of cascading them. It would simplify SAS fabric operation and made system more robust even though seems like having more single points of failure.

Thank you for taking the time to respond and sharing your knowledge! I am currently deconstructing and simplifying the cabling inside the chassis as you recommended. I will then try gmultipath destroy on each of the SATA disks and hopefully be able to then import the volume.

 

[Brosif_My_Nif]

Well, GREAT NEWS! I reconfigured the front 24-port backplane as a single expander backplane. I re-installed my original 21 SATA HDD's into the front backplane, powered on the FreeNAS, and low and behold, the most beautiful Green Light in the upper right hand corner appeared on the webgui. I didn't even have to use gmultipath destroy. I was able to successfully import my volume and everything is back to normal.

Thank you both for your assistance and taking the time to respond.