Mirrored Boot Pool on Flash Drives degraded after reboot

[Felix_True]

I'm running TrueNAS Core "TrueNAS-12.0-U5.1", booting from "Samsung FIT Plus 32GB Typ-A 200 MB/s USB 3.1 Flash Drive". I have two of them, running a mirrored boot-pool.

After the reboot initiated by updating to 12.0-U5 the system came up, telling me that the boot pool is degraded. Looking closer shows:

Code:

root@vault[~]# zpool status boot-pool
  pool: boot-pool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
  scan: scrub repaired 40.5K in 00:02:22 with 0 errors on Wed Aug 18 03:47:22 2021
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   DEGRADED     0     0     0
          mirror-0  DEGRADED     0     0     0
            da0p2   ONLINE       0     0     0
            da0p2   FAULTED      0     0     0  corrupted data

errors: No known data errors

I'm not surprised that mirroring da0p2 to da0p1 doesn't work. /dev/da1 is not present any more. So I removed the Flash Drive that represents /dev/da1 und plugged it in again. Without any futher interaction, the system recognized /dev/da1 and started doing the right thing:

Code:

root@vault[~]# zpool status boot-pool
  pool: boot-pool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Aug 22 13:14:25 2021
        7.05G scanned at 185M/s, 447M issued at 11.5M/s, 7.05G total
        443M resilvered, 6.19% done, 00:09:50 to go
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0p2   ONLINE       0     0     0
            da1p2   ONLINE       0     0     0  (resilvering)

errors: No known data errors

dmesg shows that da0 was identified at boot time, da1 was not:

Code:

Root mount waiting for: usbus0
ugen0.2: <Samsung Flash Drive FIT> at usbus0
umass0 on uhub0
umass0: <Samsung Flash Drive FIT, class 0/0, rev 3.10/11.00, addr 2> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x4000
umass0:8:0: Attached to scbus8
da0 at umass-sim0 bus 0 scbus8 target 0 lun 0
da0: <Samsung Flash Drive FIT 1100> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 0319521010021458
da0: 400.000MB/s transfers
da0: 30594MB (62656641 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
usb_alloc_device: set address 3 failed (USB_ERR_IOERROR, ignored)
...
usbd_setup_device_desc: getting device descriptor at addr 3 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=3, set address failed! (USB_ERR_IOERROR, ignored)
usbd_setup_device_desc: getting device descriptor at addr 3 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=3, set address failed! (USB_ERR_IOERROR, ignored)
usbd_setup_device_desc: getting device descriptor at addr 3 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=3, set address failed! (USB_ERR_IOERROR, ignored)
...
usbd_setup_device_desc: getting device descriptor at addr 3 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=3, set address failed! (USB_ERR_IOERROR, ignored)
usbd_setup_device_desc: getting device descriptor at addr 3 failed, USB_ERR_IOERROR

When replugging the Flash Drive it was immediately identified:

Code:

ugen0.3: <Samsung Flash Drive FIT> at usbus0
umass1 on uhub0
umass1: <Samsung Flash Drive FIT, class 0/0, rev 2.10/11.00, addr 4> on usbus0
umass1:  SCSI over Bulk-Only; quirks = 0x4000
umass1:9:1: Attached to scbus9
da1 at umass-sim1 bus 1 scbus9 target 0 lun 0
da1: <Samsung Flash Drive FIT 1100> Removable Direct Access SPC-4 SCSI device
da1: Serial Number 0374121030003565
da1: 40.000MB/s transfers
da1: 30594MB (62656641 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>

Any hints what I can do to make both Flsh Drives be identified at boot time?
Why does TrueNAS "replace" the mirror to da0p2 with da0p2 if da1p2 is missing? Is this intended behaviour? Why not something like "disk missing"?