Pool destroyed - corrupt or invalid secondary GPT table

[SubnetMask]

First, let me say that no data was lost - this was a test instance and all pools were empty.

Yesterday, I decided to give TrueNAS another go, and installed TrueNAS 13U1.1 onto a Poweredge 510 that had 18 disks attached, some internal and some in an external Supermicro JBOD enclosure. That went as well as it could, still giving me the same VMFS6 issues related to extent block size. After that, I decided to move the setup from the R510 to a PE R620, and move the 3TB SAS disks internal to the R510 into another Supermicro enclosure. Both enclosures have dual expanders SAS2 backplanes, connected to dual HBAs on the host, etc. I didn't re-use the install from the R510, instead, I made a backup of the config, then I installed a fresh copy onto the pair of SAS disks internal to the R620, then once that was complete, I restored the config I had backed up to it and made the required NIC adjustments as the NIC setup is a little different than it was on the R510.

But what I noticed on boot after restoring the config were a bunch of disks were giving the messages:

GEOM: daXX: the secondary GPT table is corrupt or invalid.
GEOM: daXX: using the primary only -- recovery is suggested.

Looking in the UI, the pool is offline, and all of the disks are listed in 'disks', but with no pool association. I did not enable encryption on any of these pools, although when I did the backup of the config, I did select to export the secret seed with the backup.

While I've done pretty much the exact same thing before with TrueNAS installations - one in particular where the newer versions were randomly rebooting, but 11.1U7 was rock solid, so I had to do a clean install to get it back to stable operation, but I've never seen this before, and never failed to either restore a config and be back in business or import the pools manually and be back in business. This corruption of the GPT table is very concerning.

Any ideas what caused this, and how to prevent it? After thinking about it for a bit, I had a thought that maybe it was related to the prompt I got, which I answered yes to, to create 16GB swap partitions, but I don't really think that's it because it didn't ask me that when I installed it to the R510 on a 16GB USB stick, and all of the disks were attached during that install as well, and if that was it, I would think it would have done it to all disks, not just one pool of disks.

 

[sretalla]

What firmware are you running on those HBAs? are they LSI chips?

 

[SubnetMask]

The HBAs are LSI 9207-4i4e's, firmware version 20.00.07.00-IT. This is actually the entire machine I was running FreeNAS on for my environment until I moved it over to run on a FC630 blade.

 

[sretalla]

If I'm not wrong, that's far from the latest firmware... maybe there's something in that...

 

[SubnetMask]

Best I can tell searching Broadcom's site, what's on them is the latest for these HBAs.

 

[AlexGG]

Something may be wrong either with disk sizes, or with sector sizes. I more thinking disk sizes. See, your primary GPT holds locations for all the partitions, and also the location for the backup (secondary) GPT table. The backup GPT happens to be stored in the last sector of the disk. The locations are specified in physical sectors.

So, if you change physical sector size, say, going from 512 to 4096 or vice versa, the entire GPT table is kaput, because all locations are invalid, and all pointers are all over the place. Sector size changes with moving to or from some USB-to-SATA adapters (with no notification), or can be changed on some RAID controllers via the config. I do not know if HBAs do that, silently or otherwise.

Another possibility, common on RAID controllers, is that the controller does not present the full size of the disk to the host. Instead, the available size is rounded down to some boundary (often 1GB or 10GB). This is done to accommodate slight difference in physical disk sizes by different manufacturers, so the drive by one manufacturer can be later replaced with a drive from another manufacturer (as in cold spare). If you change the controller from one to another, and the rounding method changes, it can eat your backup GPT and tail-end ZFS pool labels. Again, I am not sure if it applies to HBAs. Also, this issue normally does not affect the primary GPT and any partitions which entirely fit the newly limited disk space.

What you can do is to read the disk physical sector size and number of sectors on disk from the printed label on the disk itself (or from the spec, if the label does not have this information), and compare to what is reported in software.

 

[SubnetMask]

Something may be wrong either with disk sizes, or with sector sizes. I more thinking disk sizes. See, your primary GPT holds locations for all the partitions, and also the location for the backup (secondary) GPT table. The backup GPT happens to be stored in the last sector of the disk. The locations are specified in physical sectors.

So, if you change physical sector size, say, going from 512 to 4096 or vice versa, the entire GPT table is kaput, because all locations are invalid, and all pointers are all over the place. Sector size changes with moving to or from some USB-to-SATA adapters (with no notification), or can be changed on some RAID controllers via the config. I do not know if HBAs do that, silently or otherwise.

Another possibility, common on RAID controllers, is that the controller does not present the full size of the disk to the host. Instead, the available size is rounded down to some boundary (often 1GB or 10GB). This is done to accommodate slight difference in physical disk sizes by different manufacturers, so the drive by one manufacturer can be later replaced with a drive from another manufacturer (as in cold spare). If you change the controller from one to another, and the rounding method changes, it can eat your backup GPT and tail-end ZFS pool labels. Again, I am not sure if it applies to HBAs. Also, this issue normally does not affect the primary GPT and any partitions which entirely fit the newly limited disk space.

What you can do is to read the disk physical sector size and number of sectors on disk from the printed label on the disk itself (or from the spec, if the label does not have this information), and compare to what is reported in software.

Hmmm... interesting... I never thought of it, but I can understand RAID controllers reducing the reported size for the reason you mention, but I wouldn't think a HBA would do that. That being said, the HBA that these were connected to is a PERC H200A that I purchased from Art Of Server (I believe he's a member here) and is flashed with 20.00.07.00-IT firmware, that I believe some part of was modified so that it would function in the internal storage slot. The other HBA in the system, that the other disks were attached to is a 9207-8e. Being IT mode firmware, I wouldn't think there would be any RAID artifacts on it.

But that being said, if only the backup GPT table was roached, and it was using the primary, shouldn't the disks still have come up right and the pool show up, but with some sort of error related to the GPT table issue?

 

[SubnetMask]

Not that I NEED to recover these disks, but I'd like to see if I can - I've been searching for quite a bit trying to figure out how, and from everything I've found, running 'gpart recover /dev/da25' should recover disk da25 (and so on) - however, when I run it, I get 'gpart: arg0 'da25': Invalid argument'. Nothing that I've tried has resulted in anything other than gpart erroring because it 'didn't have enough information'. Any suggestions?

 

[AlexGG]

try

Code:

gpart show da25

(with no /dev/)

 

[SubnetMask]

Everything returns 'gpart: No such geom:'. No matter that permutation I try - daXX, diskXX, multipath/diskXX, but I do get returns on the ones that are part of the 'happy' pools.

 

[SubnetMask]

Ok, so this is odd and annoying. I attempted to put the affected disks back on the R510 without moving the eight drives from the Supermicro trays to the Dell trays, and 'rigged' a connection between the enclosure they were in and the H200 - didn't really work well because everything was annoyed by the R510's backplane not being connected to the SAS card. So I bit the bullet and moved them all back to the Dell trays, restored the config that I backed up back to the R510, and while the machine was working on posting, before initializing the HBAs, I slid all of the drives into the internal bays and reconnected the SAS cables to the external enclosures (the one the affected drives were in is empty right now). After it booted, did its thing and finally came online.... the pool was there. One disk was missing, possibly related to the fact that I did something I always say to NEVER do, and spanned the pool across the internal bays and external bays, and I MAY have wiped a disk in trying to do 'something' ( was at a point where I was like, what the heck, let's try to make a new pool on the disks, but it failed - it didn't want to touch the affected disks), but, it was there. So I killed the pool and recreated it with just the eight disks in the internal bays, and I'm going to continue trying to see if I can figure out what the deal is.

I think next step will be shut it down and move the disks from external enclosure 1 to external enclosure 2 and see of they go screwy - if they do, then there's probably something screwy with enclosure 2 - if they don't, it's kind of back to head scratching.

And again, to be clear, I wouldn't be doing 'lets try this' tinkering if there was important data - this is just testing stuff, and if it mushroom clouds, it doesn't matter. I'd like to try to figure out what happened - what caused these disks to not be recognized when moved to the new system in the different enclosure? I've done several similar moves in the past with FreeNAS and never had an issue - after moving the disks to the new hardware/HBAs, it just worked.

 

[AlexGG]

With old config and new config both, get parameters of the disk as reported to the system, sector size and number of sectors, and compare. See if there is any difference, and also how it compares to physical specification as printed on the disk label.