SOLVED: Expand button added LOG as Pool VDEV ?!

[buckyball]

Edit: Jump down to Post #3 and you'll see that "zpool remove poolname log-vdev" got rid of the broken LOG vdev's confusion.
----------------------
Good reader, long story short, in trying to fix a simple issue I seem to have created a bigger one.

Help me answer this question:
A) Can I fix the problem described below, or,
B) Do I need to back up my pool, destroy it, rebuild it, and restore my data?

Extra credit: Do you think that the UI needs to be fixed to prevent this sort of disaster?

Thanks in advance!
----------------------------
I had a perfectly good system:
TrueNAS SCALE 23.10.1

deadpool (loved the movie)
DATA stripe of 5 mirror VDEVs (10 x 6TB SATA HDD drives)
LOG VDEV Optane SSD (110GB)

That was all working fine and dandy.
Then I got a message that one of the mirror members had an unrecoverable checksum error.

(offending drive is boldfaced)

Code:

 
# zpool status deadpool
pool: deadpool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.

action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub repaired 0B in 09:51:23 with 0 errors on Sat Feb 10 03:51:27 2024

config:

    NAME                                      STATE     READ WRITE CKSUM
    deadpool                                  ONLINE       0     0     0
      mirror-0                                ONLINE       0     0     0
        5ca8ef8d-a70a-11eb-aac5-a8a15937dbc6  ONLINE       0     0     0
        c6702ffc-c408-11eb-a335-a8a15937dbc6  ONLINE       0     0     1
      mirror-1                                ONLINE       0     0     0
        63dc06fb-a70a-11eb-aac5-a8a15937dbc6  ONLINE       0     0     0
        f0fd5e07-c69c-11eb-aea5-a8a15937dbc6  ONLINE       0     0     0
      mirror-2                                ONLINE       0     0     0
        657d1fa3-a70a-11eb-aac5-a8a15937dbc6  ONLINE       0     0     0
        6c234139-c40b-11eb-a335-a8a15937dbc6  ONLINE       0     0     0
      mirror-3                                ONLINE       0     0     0
        391b0432-dd76-11eb-950d-a8a15937dbc6  ONLINE       0     0     0
        393b4155-dd76-11eb-950d-a8a15937dbc6  ONLINE       0     0     0
      mirror-4                                ONLINE       0     0     0
        761be833-daec-4c39-8321-65ee9e8568b6  ONLINE       0     0     0
        448ca77e-8020-41a3-a69e-11b8f09562cb  ONLINE       0     0     0
    logs  
      ecb518e7-03e5-44fb-8bcc-8e1e1b1dbc29    ONLINE       0     0     0

errors: No known data errors

So, I did the following:
- checked the SMART log for that drive - no error
- ran a scrub on the pool (~10 hours) - no errors
- ran both a SMART SHORT and SMART LONG self-test - no errors

- documentation suggests I run "zpool clear"

- THEN I poked the EXPAND button on the Storage dashboard..... BIG MISTAKE

Now it appears that the LOG device has been appended as another DATA vdev!
BUT the SSD had previously been successfully assigned as a LOG on the pool!

Now the system appears to be confused as to whether the SSD is still the LOG device or not:

Instead of showing no Unassigned Disks, I get this:

So I rebooted the server. Now my real problems begin:

Running zpool status gives me this:

Code:

# zpool status
  pool: deadpool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
    invalid.  Sufficient replicas exist for the pool to continue
    functioning in a degraded state.

action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
  scan: scrub repaired 0B in 09:51:23 with 0 errors on Sat Feb 10 03:51:27 2024
config:

    NAME                                      STATE     READ WRITE CKSUM
    deadpool                                  DEGRADED     0     0     0
      mirror-0                                ONLINE       0     0     0
        5ca8ef8d-a70a-11eb-aac5-a8a15937dbc6  ONLINE       0     0     0
        c6702ffc-c408-11eb-a335-a8a15937dbc6  ONLINE       0     0     0
      mirror-1                                ONLINE       0     0     0
        63dc06fb-a70a-11eb-aac5-a8a15937dbc6  ONLINE       0     0     0
        f0fd5e07-c69c-11eb-aea5-a8a15937dbc6  ONLINE       0     0     0
      mirror-2                                ONLINE       0     0     0
        657d1fa3-a70a-11eb-aac5-a8a15937dbc6  ONLINE       0     0     0
        6c234139-c40b-11eb-a335-a8a15937dbc6  ONLINE       0     0     0
      mirror-3                                ONLINE       0     0     0
        391b0432-dd76-11eb-950d-a8a15937dbc6  ONLINE       0     0     0
        393b4155-dd76-11eb-950d-a8a15937dbc6  ONLINE       0     0     0
      mirror-4                                ONLINE       0     0     0
        761be833-daec-4c39-8321-65ee9e8568b6  ONLINE       0     0     0
        448ca77e-8020-41a3-a69e-11b8f09562cb  ONLINE       0     0     0
    logs   
      15381800400422709162                    UNAVAIL      0     0     0  was /dev/disk/by-partuuid/ecb518e7-03e5-44fb-8bcc-8e1e1b1dbc29

errors: No known data errors 

The way I interpret this is that the LOG device is not available, because it is now a DATA vdev, correct? So am I up the creek?

 

[artlessknave]

observations:
it would be impossible, by design, to attach a 110gb disk to a 6TB vdev. zfs simply will not allow it. attaching requires a disk of equal or larger size, and 110gb is clearly not equal to 6tb.
truenas will generally not attach an existing disk in way you appear to be assuming. it knows if a disk is part of a pool, and hides it from selection entirely.

according to your zpool outputs, you have not added any new vdevs, you have simply lost the log vdev.

a LOG device only works for sync writes, and will make your writes slower, not faster, in most cases. it functions as a backup of your writes, which are stored in RAM, so you iirc you can remove it anytime. the only time it would be read is if you had interrupted *sync* writes.

I would just remove the log device from the pool. it looks like it died? im not sure if you can do that from the webUI. might have to CLI it. something like this? (I dont use log devices, completely pointless)
zpool dealpool detach 15381800400422709162
or
zpool dealpool remove 15381800400422709162

you want an output like "glabel status" to check your gpids, as this helps to map gptid to physical disk, but im not sure what it would be on scale.
the "disks" webUI might be enough.
lsblk maybe? I dont have a linux system handy, so you would have to check the man or help pages, but I think something like:
lsblk -o NAME,fstype,hctl,gptid,disk
or maybe just
lsblk -f

you can reattach it if you figure out what its problem is, though I recomend not bothering.

a single checksum failure is not really a big deal, just keep an eye on it and plan for having a replacement ready.

also, while technically correct, "a stripe of 5 mirrors" is not the usual way to state a mirror pool. zfs always stripes across every vdev, so there is no need to specify that. a mirror pool can have only 1 topology, all that changes is the number of disks per vdev.

 

[buckyball]

Thanks for the timely response. I really appreciate your help.

The trick was "zpool remove pool vdev" command.

For completeness, I wanted to report the full results:

The "zpool detach" failed; the "zpool remove" succeeded. Here's the output:

Code:

# zpool detach deadpool 15381800400422709162
cannot detach 15381800400422709162: only applicable to mirror and replacing vdevs

# zpool remove deadpool 15381800400422709162     
(returned with no error) <- I prefer some sort of positive acknowledgement of success/failure. I know that's not friendly for scripting tho.

# zpool status deadpool

  pool: deadpool
 state: ONLINE    <<<<- Yeehah!
status: Some supported and requested features are not enabled on the pool.
    The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 09:51:23 with 0 errors on Sat Feb 10 03:51:27 2024
config:

    NAME                                      STATE     READ WRITE CKSUM
    deadpool                                  ONLINE       0     0     0
      mirror-0                                ONLINE       0     0     0
        5ca8ef8d-a70a-11eb-aac5-a8a15937dbc6  ONLINE       0     0     0
        c6702ffc-c408-11eb-a335-a8a15937dbc6  ONLINE       0     0     0
      mirror-1                                ONLINE       0     0     0
        63dc06fb-a70a-11eb-aac5-a8a15937dbc6  ONLINE       0     0     0
        f0fd5e07-c69c-11eb-aea5-a8a15937dbc6  ONLINE       0     0     0
      mirror-2                                ONLINE       0     0     0
        657d1fa3-a70a-11eb-aac5-a8a15937dbc6  ONLINE       0     0     0
        6c234139-c40b-11eb-a335-a8a15937dbc6  ONLINE       0     0     0
      mirror-3                                ONLINE       0     0     0
        391b0432-dd76-11eb-950d-a8a15937dbc6  ONLINE       0     0     0
        393b4155-dd76-11eb-950d-a8a15937dbc6  ONLINE       0     0     0
      mirror-4                                ONLINE       0     0     0
        761be833-daec-4c39-8321-65ee9e8568b6  ONLINE       0     0     0
        448ca77e-8020-41a3-a69e-11b8f09562cb  ONLINE       0     0     0

errors: No known data errors

 

[buckyball]

Just a few other thoughts:

- I was clear on the expected behavior of ZFS when trying to do illegal things, which is why my jaw dropped.

- My follow-on concern is that apparently the UI in TrueNAS Scale allows this to happen at all.

In my case, the SSD had already been successfully assigned to the pool, and there were 0 drives assignable when this whole thing started. I would expect that unless I had just replaced and re-silvered all mirror pool drives to larger disks and had autoexpand=on in the pool properties, that invoking "Expand" would be a NO-OP (or be grayed out / unselectable?).

Do you think I should open a ticket in Jira?

(Witness prior discussions I've read around "Extend" and Expand", how they show up the UI, and how users interpret what they are intended to do .vs. what they actually do on ZFS.)

- On the nomenclature around ZFS mirror pools - I think it's confusing to people previously familiar with traditional RAID0/1/5/10 concepts.
When I was first learning about ZFS vdev and pool topologies, I recall a "stripe" being akin to "RAID0" (not recommended of course) and a "mirror" being (1+n) drives with the capacity of 1 drive. IIRC, non-ZFS folks would normally refer to a "mirror pool" as a "RAID10".

- I know that progress has been made in the TrueNAS UI to provide structure around vdev and pool creation. Bravo for that.

 

[artlessknave]

The "zpool detach" failed; the "zpool remove" succeeded. Here's the output:

ya thats why I put both. certain things are done with each and I never remember which

On the nomenclature around ZFS mirror pools - I think it's confusing to people previously familiar with traditional RAID0/1/5/10 concepts.

yes, but they are completely different technologies, so its best to learn it. for example, a zfs stripe is analogous to RAID0; it is NOT raid0, and calling it that causes people to expect it to act like a raid0, which it very much is not. it's similar but implemented a very different way.
its like calling a semi a car; yes they both use internal combusion to make it go but how they do so is very different. semi's are usually diesel and cars gas, the engine design is very different, but still makes it go.

I was clear on the expected behavior of ZFS when trying to do illegal things, which is why my jaw dropped.

My follow-on concern is that apparently the UI in TrueNAS Scale allows this to happen at all.

nothing I have seen here so far shows ZFS doing anything "illegal"; your pool is exactly as you described 5x mirror vdevs. the ssd is not attached to any mirror because it *cant* be; its not a matter of just that zfs wont allow it, you cannot attach a smaller disk to a bigger disk; the disk being attached MUST have the same space or it outright cannot work, you simply cannot resilver 1TB onto 110GB.
a list of the disks and their GPTIDs would show the actual state and make it far more clear. without that I can see little more than the gptids, i have no way to see the disks; as what you are saying happened isnt possible, I can only believe it happened if you can prove it, which has not occured.

autoexpand=on

autoexpand means that if you replace all the disks in a vdev with larger disks, zfs will adapt and use the larger space *automatically*. if autoexpand is on, the expand function would just do nothing; it would check for anything to expand, find nothing, and end. if autoexpand is off, the expand button would check for anything to expand, expand those it finds, and then end.
it will not attach disks, and there are no disks to choose (I was confusing expand and extend, because I built my pool before those even existed so I havent really used them)

you started with 5 mirror vdevs of 2 drives+a SLOG. you ended with 5 mirror vdevs of 2 drives and a MIA SSD.
you have had that for each zpool status, none of your GPTIDs have changed (i checked each line for the 3 zpool outputs) which means none of your disks in your pool have moved anywhere.

so. here is what I think happened, based on everything in the thread so far.
you had a checksum error, which is notable but not huge. you then kinda paniced and went through running some of the most drive intensive checks back to back on all your disks, stressing everything.
you clicked expand, zfs checked your pool, expanded anything it could as designed, and your SSD died somewhere around that.
you paniced, assumed facts not in evidence, and came here to ask (good decision)
you have now removed the failed SSD, leaving you with a healthy deadpool

I see no sign of anything to report or do; truenas functioned as its supposed to and did exactly what you told it to.



as LOG device ONLY writes, they typically die fast for any sync writes; you need high endurance. you dont specifiy which optane, and they are usually on the longer lasting side, but it could also have died for other reasons. or the cable/connection could be loose. or gremlims.