ZFS Pool died.

[dgarner-cg]

Good afternoon, everyone.
I know this has been posted a million times here, so through my use of search function, I will attempt to give all details needed to prevent redundant questions having to be asked..

Today, after attempting to update my HPE server firmware running my NAS, my pool has died.

I am running z2.
This pool initially was 8 active drives with 4 (FOUR) hot spares ... so I'm somewhat at a loss for how this has so many catastrophic failures at the exact same time...

When I visit the GUI, the only thing I can find about the pool is it being showed as exported under Storage > Disks..

It is not available under storage > import pool although it's recognized in several places as being part of z2..

in console,

$z2 status

Code:

pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:15 with 0 errors on Thu Sep 21 03:45:17 2023
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sda3      ONLINE       0     0     0

errors: No known data errors

  pool: system
 state: ONLINE
  scan: scrub repaired 0B in 00:00:36 with 0 errors on Sun Sep 10 00:00:38 2023
config:

        NAME                                      STATE     READ WRITE CKSUM
        system                                    ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            80e5349f-da1e-4e77-a6d8-a8c2c1ce76a9  ONLINE       0     0     0
            b66c7018-9097-454b-89d9-4d638ffb2e85  ONLINE       0     0     0

errors: No known data errors

$z2 import

Code:

   pool: z2
     id: 1052080854075599518
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        z2                                          ONLINE
          raidz2-0                                  ONLINE
            spare-0                                 ONLINE
              74bbd376-271a-4e0d-89c5-13da850eee79  ONLINE
              b451bd60-074b-4110-8845-800db5fa85b9  ONLINE
            07c5606f-2055-4bc1-a5e1-f54387968d16    ONLINE
            ab6bedd7-601e-4623-be42-50869b0ca6ed    ONLINE
            8106fd95-8e65-4f64-a5ea-8ce4a451c577    ONLINE
        spares
          b451bd60-074b-4110-8845-800db5fa85b9

Any and all help would be unbelievable .. This is fairly mission critical, hence the 4 hot spares that apparently were 100% all part of a bad batch or something is really goofed up here..

 

[Ericloewe]

This pool initially was 8 active drives with 4 (FOUR) hot spares ... so I'm somewhat at a loss for how this has so many catastrophic failures at the exact same time...

Unless you mean it was a 4-wide RAIDZ2 pool with four additional spares, no it wasn't. It was clearly four disks, any spares aside.

And the pool appears to be fine, you seem to just have exported it somehow. Have you tried to import it yet?

Sidenote: RAIDZ2 with four hotspares is truly bizarre. You'd use RAIDZ3 in lieu of a hotspare, with a single vdev, since it does the same job but better. I also question the wisdom of having six disks' worth of redundancy for two disks' worth of data, it's a six-thousand-hulls sort of situation, what kind of bizarre failure takes out six disks, leaving two intact, giving enough time for rebuilds in the meantime, and doesn't destroy the whole machine outright? I have no reason to doubt the mission-critical nature of your data, but that's just not a good way to achieve your goals.

 

[dgarner-cg]

Unless you mean it was a 4-wide RAIDZ2 pool with four additional spares, no it wasn't. It was clearly four disks, any spares aside.

And the pool appears to be fine, you seem to just have exported it somehow. Have you tried to import it yet?

Sidenote: RAIDZ2 with four hotspares is truly bizarre. You'd use RAIDZ3 in lieu of a hotspare, with a single vdev, since it does the same job but better. I also question the wisdom of having six disks' worth of redundancy for two disks' worth of data, it's a six-thousand-hulls sort of situation, what kind of bizarre failure takes out six disks, leaving two intact, giving enough time for rebuilds in the meantime, and doesn't destroy the whole machine outright? I have no reason to doubt the mission-critical nature of your data, but that's just not a good way to achieve your goals.

You are correct, my apologies. It was a Z2 VDEV spanning 4 total, with additional drives. The reason it wasn't a larger z3 is because at the time I created the pool, I only had 4x 8TB HDD, purchasing more later.. you know ... to avoid this.

I don't know how it was exported in the first place .. allegedly. However.. there is no option to import from the GUI, the pool drop down is blank, as pictured here:

And when I use `zpool import z2` ... I get this beautiful message.

 

[Ericloewe]

Well, that changes things and is not good. You might try to rollback a few transactions to see if that allows the pool to import, but it's a long shot.

Does the error pop up immediately or after a few moments?

 

[sfatula]

Are you using motherboard ports for your drives, or an HBA?

 

[dgarner-cg]

Wow.. Totally meant to post this yesterday, got pulled away and thought I had sent it .. Oops.

But to answer your question, sfatula, this is plugged into... I don't know 1000%.
The drives are housed in an HPE D3600 DAE (JBOD), I presume an HBA, that is then connected to dual Mini SAS HD cables, running into a DL360 Gen 10 with 12Gbps SAS PCIe card.

---

Little more information .. idk how helpful.. All the disks appear online everywhere..

$lsblk

Code:

NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda           8:0    0 447.1G  0 disk
├─sda1        8:1    0     1M  0 part
├─sda2        8:2    0   512M  0 part
├─sda3        8:3    0 430.6G  0 part
└─sda4        8:4    0    16G  0 part
sdb           8:16   0   7.3T  0 disk
├─sdb1        8:17   0     2G  0 part
└─sdb2        8:18   0   7.3T  0 part
sdc           8:32   0   7.3T  0 disk
├─sdc1        8:33   0     2G  0 part
└─sdc2        8:34   0   7.3T  0 part
sdd           8:48   0   7.3T  0 disk
├─sdd1        8:49   0     2G  0 part
└─sdd2        8:50   0   7.3T  0 part
sde           8:64   0   7.3T  0 disk
sdf           8:80   0   7.3T  0 disk
├─sdf1        8:81   0     2G  0 part
└─sdf2        8:82   0   7.3T  0 part
sdg           8:96   0   7.3T  0 disk
├─sdg1        8:97   0     2G  0 part
└─sdg2        8:98   0   7.3T  0 part
sdh           8:112  0   7.3T  0 disk
├─sdh1        8:113  0     2G  0 part
└─sdh2        8:114  0   7.3T  0 part
sdi           8:128  0 465.8G  0 disk
├─sdi1        8:129  0     2G  0 part
│ └─md127     9:127  0     2G  0 raid1
│   └─md127 253:0    0     2G  0 crypt [SWAP]
└─sdi2        8:130  0 463.8G  0 part
sdj           8:144  0 465.8G  0 disk
├─sdj1        8:145  0     2G  0 part
│ └─md127     9:127  0     2G  0 raid1
│   └─md127 253:0    0     2G  0 crypt [SWAP]
└─sdj2        8:146  0 463.8G  0 part
sdk           8:160  0 232.9G  0 disk
├─sdk1        8:161  0   200M  0 part
└─sdk2        8:162  0 232.6G  0 part
sdl           8:176  0 931.5G  0 disk
├─sdl1        8:177  0     2G  0 part
└─sdl2        8:178  0 929.5G  0 part
sdm           8:192  0   7.3T  0 disk
zd0         230:0    0    30G  0 disk
zd16        230:16   0    30G  0 disk

Code:

=== START OF INFORMATION SECTION ===
Model Family:     HGST Ultrastar He8
Device Model:     HGST HUH728080ALE604
Serial Number:    2EHEZYGX
LU WWN Device Id: 5 000cca 23bd47483
Firmware Version: A4ADWDA0


...


199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       126


Error 127 occurred at disk power-on lifetime: 59548 hours (2481 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.


  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0


  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 80 fc 3f 40 00      07:55:07.994  READ FPDMA QUEUED
  60 01 00 7f fc 3f 40 00      07:55:07.993  READ FPDMA QUEUED
  60 01 00 7e fc 3f 40 00      07:55:07.993  READ FPDMA QUEUED
  60 01 00 7d fc 3f 40 00      07:55:07.993  READ FPDMA QUEUED
  60 01 00 7c fc 3f 40 00      07:55:07.993  READ FPDMA QUEUED


Error 126 occurred at disk power-on lifetime: 59548 hours (2481 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.


  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0


  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 80 fa 3f 40 00      07:55:07.970  READ FPDMA QUEUED
  60 01 38 7f 04 00 40 00      07:55:07.962  READ FPDMA QUEUED
  60 01 30 7e 04 00 40 00      07:55:07.962  READ FPDMA QUEUED
  60 01 28 7d 04 00 40 00      07:55:07.962  READ FPDMA QUEUED
  60 01 20 7c 04 00 40 00      07:55:07.962  READ FPDMA QUEUED
 
 Error 133 occurred at disk power-on lifetime: 59440 hours (2476 days + 16 hours)
  When the command that caused the error occurred, the device was active or idle.


  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0


  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 80 fc 3f 40 00      07:55:07.993  READ FPDMA QUEUED
  60 01 08 7f fc 3f 40 00      07:55:07.992  READ FPDMA QUEUED
  60 01 00 7e fc 3f 40 00      07:55:07.992  READ FPDMA QUEUED
  60 01 08 7d fc 3f 40 00      07:55:07.992  READ FPDMA QUEUED
  60 01 00 7c fc 3f 40 00      07:55:07.992  READ FPDMA QUEUED


Error 132 occurred at disk power-on lifetime: 59440 hours (2476 days + 16 hours)
  When the command that caused the error occurred, the device was active or idle.


  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0


  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 80 fa 3f 40 00      07:55:07.952  READ FPDMA QUEUED
  60 01 00 7f 04 00 40 00      07:55:07.822  READ FPDMA QUEUED
  60 01 00 7e 04 00 40 00      07:55:07.821  READ FPDMA QUEUED
  60 01 00 7d 04 00 40 00      07:55:07.821  READ FPDMA QUEUED
  60 01 00 7c 04 00 40 00      07:55:07.821  READ FPDMA QUEUED


Error 131 occurred at disk power-on lifetime: 59440 hours (2476 days + 16 hours)
  When the command that caused the error occurred, the device was active or idle.


  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 00 00 00 00 00  Error: ICRC, ABRT at LBA = 0x00000000 = 0


  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 90 05 00 40 00      07:55:07.798  READ FPDMA QUEUED
  60 01 00 9f 03 00 40 00      07:55:07.798  READ FPDMA QUEUED
  60 01 00 9e 03 00 40 00      07:55:07.798  READ FPDMA QUEUED
  60 01 00 9d 03 00 40 00      07:55:07.798  READ FPDMA QUEUED
  60 01 00 9c 03 00 40 00      07:55:07.798  READ FPDMA QUEUED
 
  SMART Error Log Version: 1
ATA Error Count: 1
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.


Error 1 occurred at disk power-on lifetime: 45855 hours (1910 days + 15 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.


  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  10 41 00 00 00 00 00  Error: IDNF at LBA = 0x00000000 = 0


  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 28 f8 cf 77 40 00  15d+00:54:41.834  WRITE FPDMA QUEUED
  2f 00 01 10 00 00 00 00  15d+00:54:41.834  READ LOG EXT
  61 60 80 00 00 78 40 00  15d+00:54:33.577  WRITE FPDMA QUEUED
  61 00 78 00 fc 77 40 00  15d+00:54:33.577  WRITE FPDMA QUEUED
  61 00 70 00 f8 77 40 00  15d+00:54:33.577  WRITE FPDMA QUEUED


SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     52642         -
# 2  Short offline       Completed without error       00%     52626         -


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[sfatula]

The reason I asked is in your OP, you said "after attempting to update my HPE server firmware". I'm not clear what attempting means, did it fail? It just seems like a big coincidence that you then suffer lots of various disk issues. Unlikely all the disks died at once. And usually UDMA_CRC_Error_Count is a cabling or other hardware issue. I'm more curious what you tried to do exactly and did you adjust any hardware during this process?

 

[dgarner-cg]

Okay, so .. the answer to that is basically no, and somewhat embarrassing, haha.

Earlier that morning I had received notifications on 3 drives at once, out of nowhere.. uhh. Okay, yeah, I anticipated it had to do something with seating of HDD, even though my other zfs pool is online, which is also in same JBOD...

I had been working with HP Enterprise on the phone for a specific issue related to an NVMe drive on a separate server, a DL380 Gen 10. During the phone call on that day, I was performing an upgrade to the DL380 SPP.. it completed successfully.

Onto our subject, once that was wrapped up, while still on the phone, talking through another issue .. I thought it a good idea to go ahead and upgrade the SPP of the DL360 while performing upgrades. I opened ILO, attach the SPP .ISO ... it loads, reboots .. and hangs, the HPE Tech and I are on screen share and after a few minutes I find it curious it's hanging on POST ... because the .ISO was on the D3600 JBOD. smhhh.

Reboot into not having zpool.

So it did not lose power mid-firmware upgrade, it was prior to firmware beginning. I have also taken the case off D3600 and reseated all easily accessible cables into the backplane twice.

 

[dgarner-cg]

Well, that changes things and is not good. You might try to rollback a few transactions to see if that allows the pool to import, but it's a long shot.

Does the error pop up immediately or after a few moments?

Sorry, I guess I overlooked this or something? Heh.

How would I rollback transactions?

After a few moments..

 

[Ericloewe]

zpool import -nF z2 - if the results are promising, remove the n and run it again to try it for real.

 

[dgarner-cg]

zpool import -nF z2 - if the results are promising, remove the n and run it again to try it for real.

I'm installing Windows Server 2022 on bare metal to prepare for disk cloning. Hopefully all goes smoothly as the new disks I purchased are SAS and I .. have no idea if you can use a JBOD to clone. I presume this should be no big deal for Clonezilla.

I will attempt this as soon as this slow install is complete and report back.
Thank you for the feedback.

Knoll Ontrack is sending a quote over after speaking with one of the engineers yesterday at length, they estimate minimum $16,000 to recover the RAID array.
I am also waiting to hear back from Western Digital after speaking with one of the Manager's there as these were recently purchased refurbished disks that were from an "Approved Reseller."

This is genuinely ridiculous. :)

 

[Ericloewe]

I hesitate to blame the disks, sounds a lot more like one of the other moving parts was at fault. I'd be a lot more suspicious of the HBA and/or disk chassis. The quickest way of identifying the HBA would be to look in the output of lspci for the SAS controller.

 

[sfatula]

I agree, with Ericlowewe, this sounds like a hardware issue to me, and I still can't get over this happened after attempting some updates and wonder what could have been a factor. I would also point out, that no amount of redundancy is sufficient to not do backups. You could have raidz10 and I'd still do backups.

 

[dgarner-cg]

zpool import -nF z2 - if the results are promising, remove the n and run it again to try it for real.

What does "promising" look like, exactly?
This operation took approximately 20 seconds to complete.

Code:

root@agrippa[~]# zpool import -nF z2
root@agrippa[~]# 

I just booted on for the first time in a day or two, also had other issues arise...
Order of the day so far ... I just wrote down which S/N were part of the array, am going to throw this in an ML310 Gen 8 v2 and see what happens .. Barring that miraculously working, will begin using Clonezilla to clone the drives onto the new SAS drives to be safe.

I just purchased (10)x 8TB HP Enterprise SAS drives that came in directly after the failure, that were ordered before. Bum timing.

Waiting to hear back from W/D on this issue as the drivers were recently purchased, Ontrack is quoting $16,000 to $57,000 minimum for recovery, depending on how they classify "volume." Also just acquired UFS Explorer Pro which is roughly $700 from vendor and reached out to the DAE vendor, as well, as that was also just purchased within the last 1.5 months. (refurb) #neveragain

I am well adapt at a lot of this, however, with this now being my data, my head is in a spin and I have no idea (facetiously speaking, of course) how to even open command prompt. This is insane-o.

 

[dgarner-cg]

I agree, with Ericlowewe, this sounds like a hardware issue to me, and I still can't get over this happened after attempting some updates and wonder what could have been a factor. I would also point out, that no amount of redundancy is sufficient to not do backups. You could have raidz10 and I'd still do backups.

Agreed ..
I don't believe the update was relevant, just perhaps the camel's back breaking as the OS/array went offline during reboot. It had been throwing errors that morning but nothing seemingly critical still, had new HDD on way ... and obviously the update didn't even have a chance to begin as it couldn't locate the .iso

And you are correct, I was actually in the process of setting up 2 local off-site rbs as this happened, definitely not swimming in the kind of money to afford set up 3x Synology or even .. spare PC running any backup system with 32 TB HDD all at once.

I had actually already planned to purchase a Synology today, as a matter of fact, for off-site RBS. hahha.

You know, timing is golden.

 

[sfatula]

Yeah, timing is golden, happens! I would say as far as your question about what is promising, is no errors for the command is promising. Oh, and I do get your comment about how to open a command prompt, been there done that (am retired now but lots of years to pile up a few of those). Definitely keep those notes! I guess there's hope it could work in the ML310 Gen 8 v2, I wouldn't be shocked.

I feel for you, even if you didn't have a backup!

Heck, I've planned everyhing so carefully in my systems, tested recovery, etc. etc. but who's to say disaster happens and there was one key element that was different or forgotten and I lose stuff. You never actually know in computers. You have good odds maybe, but, "can't happen" is rarely appropriate!

Write back here when you actually try the import with the -n option. Am curious.

 

[Ericloewe]

I would say as far as your question about what is promising, is no errors for the command is promising.

Yeah, no errors is a good start.

 

[dgarner-cg]

So.. uhh.. few updates ... Spent all day doing this as I'm having to play musical machines...

As far as it being the backplane.. Seems like that may not, or is no longer, the case.

I ran $zpool import and grabbed the IDs, matched with serial numbers and pulled the 5 drives that matched. Yanked those out, and put those into my ML310e Gen 8 v2, where the zpool was originally created, actually..

I plugged the four listed, from what I could tell, as the main RAID into the hot swap cage, the spare plugged into the second SATA connection and pulled the boot OS from the server as well just to save time, and fired it up.. same results. z2 failed to import due to IO errors even with.

Took another 30m to swap things back as the ML is now my Firewall, had to yank my previous Firewall out the storage closet ... Okay.

Back into the server, booted up Windows Server 2022 with 8 of the HGST drives and 4 of the brand new SAS drives and ... Windows doesn't see any of them to try and use UFS on .. Even after installing the SAS3008 drivers.

uh k, so can't even mirror disks at this point.. or try and view the filesystem.

Booted back into TrueNAS, same errors.. so uh.. I guess let's give this a shot, what's the worse that could happen, right? ...

 

[dgarner-cg]

root@agrippa[/var/log]# zpool import -F z2

Code:

cannot import 'z2': I/O error
        Destroy and re-create the pool from
        a backup source.

root@agrippa[/var/log]#

 

[dgarner-cg]

Just an update ...

Nothing I've tried so far has worked and I have failed (and am still waiting for Microsoft to get back with me) to get Windows Server 2022 DC to recognize the drives so perform any sort of .. anything.

Currently I have had to revert to direct connection between Testbench M/B to SATA connection and am running through UFS Explorer Scan. I want to say Hetman RAID recovery was quickly unsuccessful.

Now, the down side to this is .. I have no image/clone of these drives as Clonezilla refused to recognize them, and I could not easily get zfs installed to test if that would work. It would not recognize the drives, and at some point, the drives stop being recognized by Linux kernel at all to attempt to run a zpool send or any sort of dd-* ..

I did get a 45mb log from running

Code:

zdb -F -e

, but did not yet run zdb to inspect the Uberblocks.

I did receive a DM of someone that had the exact same issue and was able to successfully recover everything, however, I am currently 18 hours in a UFS scan at 30% with est. 35h left before I can run through this attempted walkthrough.

As far as I can tell now, this is definitely firmware/hardware damage, although I have been pristinely gentle with them under my care, I obviously can't speak for the highly paid retail professionals, logistics companies and whoever else had a hand ON these prior.

• I am reaching out to Amazon's legal team today, which has to be done via USPS, as advised by my lawyer;
• I was finally able to contact the vendor and had my complaint with all logs escalated to Executive Team/Owner;
• Western Digital denied disaster recovery due to x, y, z and I will be pursuing a strategy that has worked prior in having a company cover disaster recovery at no cost (Seagate specifically, $3,500 recovery);

In the event we do not hear back favorably from anyone in the supply chain, or, if data is not recovered successfully by myself within the allotted time period.. well.. being as this is a public forum, will leave it at, I anticipate one of these routes will be successful.. :)