Replacing failed USB boot drive, accidentally removed ZFS mirror

[GTAXL]

Hello, I had a USB Flash drive fail in my freenas-boot pool. It thus resulted in a degraded state for the boot pool mirror. The configuration for the boot drive is x2 SanDisk Cruzer Dial 16GB USB 2.0 Flash drives. These are plugged into the front two 2.0 ports. They are in a ZFS mirror named freenas-boot. I know it is not wise to use flash drives. I did this back in 2018 and was being cheap. I am replacing them with M.2 SATA SSDs in a USB enclosure since I now have the opportunity anyway.

pool: freenas-boot
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q
scan: scrub repaired 0B in 00:11:20 with 0 errors on Sat Dec 2 03:56:21 2023
config:

NAME STATE READ WRITE CKSUM
freenas-boot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
da0p2 ONLINE 0 0 0
da1p2 UNAVAIL 3 103 0 cannot open

errors: No known data errors

I then proceeded to offline the failed flash drive da1p2. I looked around on the Web GUI and couldn't find how to offline or replace the drive via storage pools or disks as it just shows the data pool only, not the boot pool. So I proceeded to offline the drive via the command line, which worked.

zpool offline freenas-boot da1p2
da1p2 OFFLINE 3 103 0

I kept the flash drive plugged in as I didn't know which one had failed due to no serial number written on the drive. I plugged in two M.2 SATA SSD's via a USB enclosure which I plan on moving the boot drive to. These were plugged into different USB ports, 3.0 ports, whilst the existing flash drives including the one failed one remained plugged into 2.0 ports. TrueNAS detected the new drives as such:

Dec 3 10:36:17 freenas.nas.lan.gtaxl.net usb_msc_auto_quirk: UQ_MSC_NO_GETMAXLUN set for USB mass storage device SSK SSK Storage (0x152d:0x0580)
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net usb_msc_auto_quirk: UQ_MSC_NO_PREVENT_ALLOW set for USB mass storage device SSK SSK Storage (0x152d:0x0580)
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net ugen0.5: <SSK SSK Storage> at usbus0
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net umass2 on uhub1
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net umass2: <SSK SSK Storage, class 0/0, rev 3.20/76.01, addr 4> on usbus0
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net umass2: SCSI over Bulk-Only; quirks = 0x8100
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net umass2:10:2: Attached to scbus10
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net da1 at umass-sim2 bus 2 scbus10 target 0 lun 0
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net da1: <SSK 7601> Fixed Direct Access SPC-4 SCSI device
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net da1: Serial Number DD564198838FE
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net da1: 400.000MB/s transfers
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net da1: 122104MB (250069680 512 byte sectors)
Dec 3 10:36:17 freenas.nas.lan.gtaxl.net da1: quirks=0x2<NO_6_BYTE>
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net usb_msc_auto_quirk: UQ_MSC_NO_GETMAXLUN set for USB mass storage device SSK SSK Storage (0x152d:0x0580)
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net usb_msc_auto_quirk: UQ_MSC_NO_PREVENT_ALLOW set for USB mass storage device SSK SSK Storage (0x152d:0x0580)
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net ugen0.6: <SSK SSK Storage> at usbus0
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net umass3 on uhub1
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net umass3: <SSK SSK Storage, class 0/0, rev 3.20/76.01, addr 5> on usbus0
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net umass3: SCSI over Bulk-Only; quirks = 0x8100
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net umass3:11:3: Attached to scbus11
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net da2 at umass-sim3 bus 3 scbus11 target 0 lun 0
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net da2: <SSK 7601> Fixed Direct Access SPC-4 SCSI device
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net da2: Serial Number DD564198838FE
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net da2: 400.000MB/s transfers
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net da2: 122104MB (250069680 512 byte sectors)
Dec 3 10:36:37 freenas.nas.lan.gtaxl.net da2: quirks=0x2<NO_6_BYTE>

I noticed in Storage > Disks of the Web GUI it only showed da1, and not the da2 which also reports detected in logs. Not sure why this is, but the new drives, SSK based enclosure, one of them shows up as da1 which the failed existing drive (SanDisk Cruzer Dial) was da1p2?? I assume p2 means partition two? I'm confused, why would a new drive plugged into a completely different USB port, 3.0 instead of 2.0, even while the failed drive was still physically plugged in, show up as da1 as well?

I said oh well, maybe USB operates on a first come, first serve basis and since I did "offline" the failed drive, maybe the new drive picked up the same name? I then proceeded to try to replace the failed drive with the new one. However that didn't work and got confused and accidentally detached the failed drive, converting my mirror into single disk.

root@freenas:/var/log # zpool replace freenas-boot da1p2 da1
cannot replace da1p2 with da1: already in replacing/spare config; wait for completion or use 'zpool detach'
root@freenas:/var/log # zpool detach
missing pool name argument
usage:
detach <pool> <device>
root@freenas:/var/log # zpool detach freenas-boot da1p2
root@freenas:/var/log # zpool status
pool: cabinet
state: ONLINE
scan: scrub repaired 0B in 03:42:03 with 0 errors on Fri Dec 1 03:42:03 2023
config:

NAME STATE READ WRITE CKSUM
cabinet ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/620e6ee6-dd5d-11e8-ad02-4ccc6aff3353.eli ONLINE 0 0 0
gptid/62be7dea-dd5d-11e8-ad02-4ccc6aff3353.eli ONLINE 0 0 0
gptid/636a448a-dd5d-11e8-ad02-4ccc6aff3353.eli ONLINE 0 0 0
gptid/641dae75-dd5d-11e8-ad02-4ccc6aff3353.eli ONLINE 0 0 0
gptid/65730f0c-dd5d-11e8-ad02-4ccc6aff3353.eli ONLINE 0 0 0
gptid/d27a0e87-8402-11ee-8f0e-00074306cd70.eli ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:11:20 with 0 errors on Sat Dec 2 03:56:21 2023
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors

I feel real silly now.. So my question is how do I add the new drive and put it back into a mirror? Then how would I replace the da0p2, which is the still working flash drive with the 2nd new drive da2? Here are the drives, the da1 and da2 are the new M.2 drives that appear as SSK which is the brand of the USB enclosure. The da0 is the existing flash drive that still works for the boot pool, the other sandisk drive doesn't appear on the list, maybe because I offlined it? It's still physically connected.

root@freenas:/var/log # gpart list
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
efimedia: HD(1,GPT,62015496-dd5d-11e8-ad02-4ccc6aff3353,0x80,0x400000)
rawuuid: 62015496-dd5d-11e8-ad02-4ccc6aff3353
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada0p2
Mediasize: 3998639456256 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e2
efimedia: HD(2,GPT,620e6ee6-dd5d-11e8-ad02-4ccc6aff3353,0x400080,0x1d180be00)
rawuuid: 620e6ee6-dd5d-11e8-ad02-4ccc6aff3353
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 3998639456256
offset: 2147549184
type: freebsd-zfs
index: 2
end: 7814037119
start: 4194432
Consumers:
1. Name: ada0
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
efimedia: HD(1,GPT,62ae023c-dd5d-11e8-ad02-4ccc6aff3353,0x80,0x400000)
rawuuid: 62ae023c-dd5d-11e8-ad02-4ccc6aff3353
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada1p2
Mediasize: 3998639456256 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e2
efimedia: HD(2,GPT,62be7dea-dd5d-11e8-ad02-4ccc6aff3353,0x400080,0x1d180be00)
rawuuid: 62be7dea-dd5d-11e8-ad02-4ccc6aff3353
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 3998639456256
offset: 2147549184
type: freebsd-zfs
index: 2
end: 7814037119
start: 4194432
Consumers:
1. Name: ada1
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5

Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
efimedia: HD(1,GPT,635aa33d-dd5d-11e8-ad02-4ccc6aff3353,0x80,0x400000)
rawuuid: 635aa33d-dd5d-11e8-ad02-4ccc6aff3353
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada2p2
Mediasize: 3998639456256 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e2
efimedia: HD(2,GPT,636a448a-dd5d-11e8-ad02-4ccc6aff3353,0x400080,0x1d180be00)
rawuuid: 636a448a-dd5d-11e8-ad02-4ccc6aff3353
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 3998639456256
offset: 2147549184
type: freebsd-zfs
index: 2
end: 7814037119
start: 4194432
Consumers:
1. Name: ada2
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5

Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
efimedia: HD(1,GPT,640ebb6b-dd5d-11e8-ad02-4ccc6aff3353,0x80,0x400000)
rawuuid: 640ebb6b-dd5d-11e8-ad02-4ccc6aff3353
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada3p2
Mediasize: 3998639456256 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e2
efimedia: HD(2,GPT,641dae75-dd5d-11e8-ad02-4ccc6aff3353,0x400080,0x1d180be00)
rawuuid: 641dae75-dd5d-11e8-ad02-4ccc6aff3353
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 3998639456256
offset: 2147549184
type: freebsd-zfs
index: 2
end: 7814037119
start: 4194432
Consumers:
1. Name: ada3
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5

Geom name: ada4
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada4p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
efimedia: HD(1,GPT,6562c890-dd5d-11e8-ad02-4ccc6aff3353,0x80,0x400000)
rawuuid: 6562c890-dd5d-11e8-ad02-4ccc6aff3353
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada4p2
Mediasize: 3998639456256 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e2
efimedia: HD(2,GPT,65730f0c-dd5d-11e8-ad02-4ccc6aff3353,0x400080,0x1d180be00)
rawuuid: 65730f0c-dd5d-11e8-ad02-4ccc6aff3353
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 3998639456256
offset: 2147549184
type: freebsd-zfs
index: 2
end: 7814037119
start: 4194432
Consumers:
1. Name: ada4
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 30595031
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
Mediasize: 272629760 (260M)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 20480
Mode: r0w0e0
efimedia: HD(1,GPT,b2a087be-c385-11e8-b4e2-4ccc6aff3353,0x28,0x82000)
rawuuid: b2a087be-c385-11e8-b4e2-4ccc6aff3353
rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
label: (null)
length: 272629760
offset: 20480
type: efi
index: 1
end: 532519
start: 40
2. Name: da0p2
Mediasize: 15324893184 (14G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 272650240
Mode: r1w1e1
efimedia: HD(2,GPT,b300cf82-c385-11e8-b4e2-4ccc6aff3353,0x82028,0x1c8b7a8)
rawuuid: b300cf82-c385-11e8-b4e2-4ccc6aff3353
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 15324893184
offset: 272650240
type: freebsd-zfs
index: 2
end: 30463951
start: 532520
Consumers:
1. Name: da0
Mediasize: 15664676864 (15G)
Sectorsize: 512
Mode: r1w1e2

Geom name: ada5
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada5p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
efimedia: HD(1,GPT,d26b4ac9-8402-11ee-8f0e-00074306cd70,0x80,0x400000)
rawuuid: d26b4ac9-8402-11ee-8f0e-00074306cd70
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada5p2
Mediasize: 3998639460352 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e2
efimedia: HD(2,GPT,d27a0e87-8402-11ee-8f0e-00074306cd70,0x400080,0x1d180be08)
rawuuid: d27a0e87-8402-11ee-8f0e-00074306cd70
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 3998639460352
offset: 2147549184
type: freebsd-zfs
index: 2
end: 7814037127
start: 4194432
Consumers:
1. Name: ada5
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5

root@freenas:/var/log # geom disk list
Geom name: ada0
Providers:
1. Name: ada0
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5
descr: WDC WD40EFRX-68N32N0
lunid: 50014ee2652a40f9
ident: WD-WCC7K3PY4A55
rotationrate: 5400
fwsectors: 63
fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5
descr: WDC WD40EFRX-68N32N0
lunid: 50014ee20fd5211b
ident: WD-WCC7K3PY4V2D
rotationrate: 5400
fwsectors: 63
fwheads: 16

Geom name: ada2
Providers:
1. Name: ada2
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5
descr: WDC WD40EFRX-68N32N0
lunid: 50014ee26560b72d
ident: WD-WCC7K4NHXLVJ
rotationrate: 5400
fwsectors: 63
fwheads: 16

Geom name: ada3
Providers:
1. Name: ada3
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5
descr: WDC WD40EFRX-68N32N0
lunid: 50014ee20fd51b04
ident: WD-WCC7K1NDSEHK
rotationrate: 5400
fwsectors: 63
fwheads: 16

Geom name: ada4
Providers:
1. Name: ada4
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5
descr: WDC WD40EFRX-68N32N0
lunid: 50014ee26560dcc9
ident: WD-WCC7K0UX440A
rotationrate: 5400
fwsectors: 63
fwheads: 16

Geom name: ada5
Providers:
1. Name: ada5
Mediasize: 4000787030016 (3.6T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5
descr: WDC WD40EFPX-68C6CN0
lunid: 50014ee2c0764d52
ident: WD-WX82D43732NS
rotationrate: 5400
fwsectors: 63
fwheads: 16

Geom name: da0
Providers:
1. Name: da0
Mediasize: 15664676864 (15G)
Sectorsize: 512
Mode: r1w1e2
descr: SanDisk Cruzer Dial
lunname: SanDisk Cruzer Dial 4C530001320728115411
lunid: SanDisk Cruzer Dial 4C530001320728115411
ident: 4C530001320728115411
rotationrate: unknown
fwsectors: 63
fwheads: 255

Geom name: da1
Providers:
1. Name: da1
Mediasize: 128035676160 (119G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
descr: SSK
lunname: SSK DD56419883897
lunid: 3044564198838650
ident: DD564198838FE
rotationrate: 0
fwsectors: 63
fwheads: 255

Geom name: da2
Providers:
1. Name: da2
Mediasize: 128035676160 (119G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
descr: SSK
lunname: SSK DD56419883897
lunid: 3044564198838650
ident: DD564198838FE
rotationrate: 0
fwsectors: 63
fwheads: 255

These are 120GB M.2 SATA SSD's in a USB 3.0 enclosure. I did do a "zpool set autoexpand=on freenas-boot" so it should resize the pool from 16GB to 120GB. I also deleted old boot environments as the boot pool was nearing 80% capacity on the 16GB flash drives. This is probably why one failed.

Any help would be greatly appreciated and please let me know if you need any more information or what commands you would like me to run.