Zpool (raidz2-0) offline, log SSDs unavailable

[dadio]

I have a TrueNAS system with Qty=5, 4TB drives in a raidz2-0 configuration. It has one main zpool named "rivrun-1" that now OFFLINE. I noticed that the mirrored log disks (actually SSDs) I included in the pool when I set it up are both UNAVAIL. Seems amazing they both died. How do I go about replacing the log disks and getting my pool back ONLINE. I searched the forum but could not find anything pertinent. Any recommendations would be greatly appreciated.

Code:

root@crequefreenas:~ # zpool import
   pool: rivrun-1
     id: 12522148391907502925
  state: UNAVAIL
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-6X
 config:

        rivrun-1                                        UNAVAIL  missing device
          raidz2-0                                      ONLINE
            gptid/c94cb861-8e1b-11ea-a33a-6805ca103140  ONLINE
            gptid/0d8b4314-32dd-11ea-b85c-6805ca103140  ONLINE
            gptid/a775bce7-3418-11ea-9ee3-6805ca103140  ONLINE
            gptid/a165ee06-3483-11ea-a056-6805ca103140  ONLINE
            gptid/f0854a8d-119d-11eb-a691-6805ca103140  ONLINE
        logs
          mirror-2                                      UNAVAIL  insufficient replicas
            gptid/fe72fa09-2131-11ea-8a26-6805ca103140  UNAVAIL  cannot open
            gptid/a56d7119-2135-11ea-86d2-6805ca103140  UNAVAIL  cannot open

        Additional devices are known to be part of this pool, though their
        exact configuration cannot be determined.
root@crequefreenas:~ #

 

[artlessknave]

my first guess would be that you wore out your log devices, and they are both dead. remember that LOG is basically write only.
I do wonder if you really needed them, but that doesn't help with getting the pool back.
the easiest way would probably to nuke it and restore from backups....you do have backups....right?

I believe there is a way to force it to import the pool and discard the log devices, but I have never needed log devices so have never mucked with them myself.
the people who most likely could help rarely do if you don't, at least, follow the forum rules of posting your hardware setup. many put it as a signature so it's always present.
commendation for using the code tags though!

 

[HoneyBadger]

The extra parameter needed is -m (possibly also -f to force import, or -F to "really force")

But try zpool import -m rivrun-1 first. You will get a warning about discarding data since the last successful transaction; "thems the breaks" though. Suggest a scrub after mount.

my first guess would be that you wore out your log devices, and they are both dead. remember that LOG is basically write only.

Potentially the cause here. Make and model of log drives please @dadio ?

 

[dadio]

Thank you very much all. I discovered my pool offline last evening and was too tired to do a hardware inventory, but I will this evening. I will also try

Code:

zpool import -m rivrun-1

and let y'all know the results, along with hardware setup per the rules.

 

[dadio]

Forcing the import using the -m option brought my pool back online. Thank you all. Here is what my pool status is now:

Code:

root@crequefreenas:~ # zpool status
  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:12:09 with 0 errors on Thu Feb 17 03:57:09 2022
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2       ONLINE       0     0     0

errors: No known data errors

  pool: rivrun-1
 state: DEGRADED
status: One or more devices has been removed by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using zpool online' or replace the device with
        'zpool replace'.
  scan: scrub repaired 0B in 03:56:04 with 0 errors on Thu Feb 17 03:56:05 2022
config:

        NAME                                            STATE     READ WRITE CKSUM
        rivrun-1                                        DEGRADED     0     0 0
          raidz2-0                                      ONLINE       0     0 0
            gptid/c94cb861-8e1b-11ea-a33a-6805ca103140  ONLINE       0     0 0
            gptid/0d8b4314-32dd-11ea-b85c-6805ca103140  ONLINE       0     0 0
            gptid/a775bce7-3418-11ea-9ee3-6805ca103140  ONLINE       0     0 0
            gptid/a165ee06-3483-11ea-a056-6805ca103140  ONLINE       0     0 0
            gptid/f0854a8d-119d-11eb-a691-6805ca103140  ONLINE       0     0 0
        logs
          mirror-2                                      UNAVAIL      0     0 0  insufficient replicas

I also added my rig hardware details to my signature.

 

[HoneyBadger]

Logs: Qty = 2 (mirrored) 120 GByte Crucial BX500 SSDs

Yep, if these were actually absorbing writes (eg: NFS exports, sync=always datasets or zvols) that would have completely blown these out in a very short period as they aren't designed for intense write workloads at all.

Check the link in my signature regarding SLOG devices for good options. For SATA it's basically the Intel DC S37x0 series, but you'll get far better results with an NVMe Optane device.

 

[dadio]

Hey HoneyBadger. Thank you very much! I skimmed your link, which looks awesome, but need to set some time aside to fully absorb it. From what I have read, SSDs are recommended for log devices, but since logs are write only why recommend an SSD if it will wear out? Confusing...

 

[HoneyBadger]

From what I have read, SSDs are recommended for log devices, but since logs are write only why recommend an SSD if it will wear out? Confusing...

Same reason you put high-performance tires on a sports car. Yes, they're expensive. Yes, they're going to wear out. But you won't get the performance you need any other way.

If you buy SSDs that are designed for a write-intensive workload, they survive much longer.

For example, the BX500 120G is rated for 40TB of lifetime writes.
The Intel DC S3700 100G is rated for about 1825TB of lifetime writes, or roughly 45 times as long.

 

[dadio]

Great information. Thanks again. Greatly appreciated.