DC Overheated lost 30 of my 90 drives

[Johnny Fartpants]

So this happened the other day. DC cooling failed for a while taking drive temps to approx 45 degrees and the next day one of the 90 bay JBODs dropped its first two rows of drives (30). Power-cycled the JBOD and all drives came back online only to drop off again overnight. New JBOD in moved all 90 drives from one to the other and now crossing fingers ZFS can work its magic. Pool has imported and all data is accessible atm and as you can see a few of my hot-spares kicked in but not much good when you lose 30 drives simultaneously. Im not asking for any help with this just sharing for those that are interested. I assume one of the backplanes didn't like the heat and decided to have a slow death.

Code:

  pool: tank
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Mar 16 05:52:33 2023
    251T scanned at 3.21G/s, 248T issued at 3.17G/s, 663T total
    9.06T resilvered, 37.43% done, 1 days 13:11:57 to go
config:

    NAME                                              STATE     READ WRITE CKSUM
    tank                                              ONLINE       0     0     0
      raidz3-0                                        ONLINE       0     0     0
        spare-0                                       ONLINE       0     0     0
          gptid/8d4baef2-9572-11ed-a956-3cecef0f5436  ONLINE       0     0     0  (resilvering)
          gptid/9a029c28-9572-11ed-a956-3cecef0f5436  ONLINE       0     0     0  (resilvering)
        spare-1                                       ONLINE       0     0     0
          gptid/8d9f172d-9572-11ed-a956-3cecef0f5436  ONLINE       0     0     0  (resilvering)
          gptid/99f33b7e-9572-11ed-a956-3cecef0f5436  ONLINE       0     0     0  (resilvering)
        spare-2                                       ONLINE       0     0     0
          gptid/8d9c81ce-9572-11ed-a956-3cecef0f5436  ONLINE       0     0     0  (resilvering)
          gptid/9a08da32-9572-11ed-a956-3cecef0f5436  ONLINE       0     0     0  (resilvering)
        gptid/8d46a193-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/8d414843-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/8d55d1a7-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/8d492fb2-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/8d50a990-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/8d39a679-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/8d4e346a-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/8d53221b-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/8d3c0eb5-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/8d43e3fe-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/8d127c62-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/8d3ea105-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
      raidz3-1                                        ONLINE       0     0     0
        gptid/8d371e51-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fbd8eff-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/90215a7f-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fca6f43-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fb9ba5f-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fd9bff2-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fd2432d-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fbb051e-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fc2adcf-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fc01c2c-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fd74d47-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fc7c81e-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fa390e4-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fcfa37c-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fc53f59-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
      raidz3-2                                        ONLINE       0     0     0
        gptid/8fd4e29b-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/8fcd136e-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/921c8320-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/92511f29-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/925def51-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9253b365-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9242b4c8-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/92468a2c-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/92607f1d-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/92b60089-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/924fb3df-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9258ce3f-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9244027f-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/925b732c-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/92b37fe9-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
      raidz3-3                                        ONLINE       0     0     0
        gptid/924b96c7-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9248ff58-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/92562b9a-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94ec6630-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94e1f309-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94aed761-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94f1a786-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94e715ba-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94f6b937-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94e48d7c-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94df7041-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94fbbb38-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/954411de-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94ef0605-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94f4294c-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
      raidz3-4                                        ONLINE       0     0     0
        gptid/94f931c0-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/94e9c5fe-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/95469bdf-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/95418455-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/97a25760-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/97f0d589-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/97f3436a-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/979d5ce7-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9795babd-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9791de68-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/97a4df4c-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/979fe07f-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/979acb79-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/97683ff0-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9793456c-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
      raidz3-5                                        ONLINE       0     0     0
        gptid/979853cd-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/978eb7b9-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/978c29ab-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/97aa09b3-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/97a76c57-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/99f89279-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9a4728fa-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/99d517d7-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9a4be6a5-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9a03edb9-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/99fb3448-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/9a068655-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/9a004477-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0  (resilvering)
        gptid/99f5feda-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
        gptid/99fdb2eb-9572-11ed-a956-3cecef0f5436    ONLINE       0     0     0
    spares
      gptid/9a029c28-9572-11ed-a956-3cecef0f5436      INUSE     currently in use
      gptid/99f33b7e-9572-11ed-a956-3cecef0f5436      INUSE     currently in use
      gptid/9a08da32-9572-11ed-a956-3cecef0f5436      INUSE     currently in use
      gptid/9a0b17af-9572-11ed-a956-3cecef0f5436      AVAIL

errors: 11669250 data errors, use '-v' for a list

 

[WI_Hedgehog]

45°C isn't that hot, 11 year old drives alarm at 65°C, newer HGST at 85°C. Cooler is better, but that was a short run at 45°.

 

[Johnny Fartpants]

Yeah only lasted an hour but drives normally ideal at 30 or below. It most likely got hotter than 45 but drives only reached that.

 

[Johnny Fartpants]

Other servers popped so it must have been bad.

 

[WI_Hedgehog]

I wonder if there were also other factors at play...?

 

[Davvo]

More than the max temperature it's the termal shock that should be of concern: a rapid change in temperature can be way more damaging than a long, low paced increase.

 

[Johnny Fartpants]

Seems like it wasn't heat related just a loose internal power connection at assembly. Been reseated and working fine for the last few months.

 

[WI_Hedgehog]

Another member lost his drives running at 45C, followed by the ones at 42 & 43.

 

[Davvo]

It's always those small things that you can't reach with your fingers withouth moving mountains and lakes that cause issues.

 

[Arwen]

...
errors: 11669250 data errors, use '-v' for a list

You may have lost data.

In my opinion, the best way to proceed is to:

1. Wait til all the hot spares have re-silvered
2. Run zpool status -v tank and record the errors
3. Clear the pool errors
4. Run a scrub
5. See if the errors come back

Who knows, maybe you will get lucky and loose nothing.

 

[Johnny Fartpants]

Yeah thanks I got lucky all was ok after resilvers, clear and scrub. It was a backup (replication target) also so wouldn’t have been the end of the world.