z2pool gone crazy after stupid tinkering-no data related issues but: inaccessible GUI options, eternal resilver loop and wrong disk identifiers

homer27081990 · Aug 13, 2022

ElGusto said:
As in #37: I cancelled the SMART long test I had initiated, because it was said to take 24 hours.

Yes, but it was canceled 2 times, both at 90% in a row. Did it go to a 24h estimate at about 90%, or was that the estimate from the beginning?

ElGusto · Aug 13, 2022

The response that I got from the WebUI directly after initiating the extended offline test immediately told me so. The progress counter seems to be wrong.

ElGusto · Aug 13, 2022

Eureka! I made some progress:

I found this thread https://www.truenas.com/community/threads/zfs-gptid-has-disappeared.80797/ which pictured the very same issue regarding the GPTIDs (da1 had no GPTID after importing, but it had one with the pool being exported). It had no solution, but somehow made me find this: https://www.reddit.com/r/zfs/comments/f7b3if/mixed_gptid_and_dev_names_in_zpool_status/

As said in the later post, I exported the pool and then did:
glabel refresh /dev/da1
glabel refresh /dev/da2
glabel refresh /dev/da3
glabel refresh /dev/da4

and then imported the pool using:
zpool import -d /dev/gptid z2pool

After this zpool status showed me, that all hard drives where added by their GPTID instead mixed, opposed to what it was like before:

root@truenas[/var/log]# zpool status z2pool
pool: z2pool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Aug 12 19:11:15 2022
291G scanned at 2.85G/s, 1.75M issued at 17.5K/s, 23.3T total
0B resilvered, 0.00% done, no estimated completion time
config:

NAME STATE READ WRITE CKSUM
z2pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/2affd65a-0bc6-11ed-bd90-000c29077bb3 ONLINE 0 0 0 (awaiting resilver)
gptid/10046937-6b06-11eb-9b87-000c29077bb3 ONLINE 0 0 0
gptid/aa7c274d-0bc3-11ed-bd90-000c29077bb3 ONLINE 0 0 0 (awaiting resilver)
gptid/9ce5247f-331b-3148-9b38-c958d2bd057a ONLINE 0 0 0

The WebUI did not show the pool at all at this stage, so I exported it again, rebootet and imported it normally using the WebUI, with this result:

So they are all added by their drive number now, instead of being mixed. I remember this was the 'normal' status for years (until my self-inflicted rampage). Also the WebUI is able to access da1's options again, which did not work before!

So I now have two options, I think:
1. See if the resilver runs through now without instantly restarting
2. Fix that strange partitioning diversity on da4 first, we remember it looked like this:

root@truenas[~]# gpart show /dev/da1
=> 40 15628053088 da1 GPT (7.3T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)

root@truenas[~]# gpart show /dev/da2
=> 40 15628053088 da2 GPT (7.3T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)

root@truenas[~]# gpart show /dev/da3
=> 40 15628053088 da3 GPT (7.3T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 15623858696 2 freebsd-zfs (7.3T)

root@truenas[~]# gpart show /dev/da4
=> 34 15628053101 da4 GPT (7.3T)
34 2014 - free - (1.0M)
2048 15628034048 1 !6a898cc3-1dd2-11b2-99a6-080020736631 (7.3T)
15628036096 16384 9 !6a945a3b-1dd2-11b2-99a6-080020736631 (8.0M)
15628052480 655 - free - (328K)

I don't know how I would fix that, though.. would removing, erasing and and committing a replace operation on da4 fix the block size and partitioning to be in line with the others?

Arwen · Aug 13, 2022

I find it odd that ZFS wants to re-silver 2 disks when manually importing the pool.

Their was some option in newer ZFS to delay replacing a 2nd disk until the first re-silver was complete. The manual page indicates that if disabled, and ZFS is confused on which disk to re-silver, it could be restarting it;

Code:

     resilver_defer
             GUID                  com.datto:resilver_defer
             READ-ONLY COMPATIBLE  yes

             This feature allows ZFS to postpone new resilvers if an existing one is already in progress.  With-
             out this feature, any new resilvers will cause the currently running one to be immediately restarted
             from the beginning.

             This feature becomes active once a resilver has been deferred, and returns to being enabled when the
             deferred resilver begins.

homer27081990 · Aug 13, 2022

Could you run gpart show -p -r /dev/sdaX? (X the drive number, duh).

ElGusto · Aug 13, 2022

Arwen said:
I find it odd that ZFS wants to re-silver 2 disks when manually importing the pool.

Their was some option in newer ZFS to delay replacing a 2nd disk until the first re-silver was complete. The manual page indicates that if disabled, and ZFS is confused on which disk to re-silver, it could be restarting it;

Code:
resilver_defer GUID com.datto:resilver_defer READ-ONLY COMPATIBLE yes This feature allows ZFS to postpone new resilvers if an existing one is already in progress. With- out this feature, any new resilvers will cause the currently running one to be immediately restarted from the beginning. This feature becomes active once a resilver has been deferred, and returns to being enabled when the deferred resilver begins.

Right, I just found this about it:

disabled resilver_defer feature leads to looping resilvers · Issue #9299 · openzfs/zfs

System information smartos 20190718T005708Z Describe the problem you're observing When a disk is replaced with another on a pool with the resilver_defer feature present, but no enabled the resilver...

github.com

But when I checked the features, I saw it's active:

root@truenas[~]# zpool get all z2pool | grep resilver
z2pool feature@resilver_defer active local

Well, maybe disabling could help in this case?

ElGusto · Aug 13, 2022

homer27081990 said:
Could you run gpart show -p -r /dev/sdaX? (X the drive number, duh).

Here it is:

root@truenas[~]# gpart show -p -r /dev/da1
=> 40 15628053088 da1 GPT (7.3T)
40 88 - free - (44K)
128 4194304 da1p1 516e7cb5-6ecf-11d6-8ff8-00022d09712b (2.0G)
4194432 15623858696 da1p2 516e7cba-6ecf-11d6-8ff8-00022d09712b (7.3T)

root@truenas[~]# gpart show -p -r /dev/da2
=> 40 15628053088 da2 GPT (7.3T)
40 88 - free - (44K)
128 4194304 da2p1 516e7cb5-6ecf-11d6-8ff8-00022d09712b (2.0G)
4194432 15623858696 da2p2 516e7cba-6ecf-11d6-8ff8-00022d09712b (7.3T)

root@truenas[~]# gpart show -p -r /dev/da3
=> 40 15628053088 da3 GPT (7.3T)
40 88 - free - (44K)
128 4194304 da3p1 516e7cb5-6ecf-11d6-8ff8-00022d09712b (2.0G)
4194432 15623858696 da3p2 516e7cba-6ecf-11d6-8ff8-00022d09712b (7.3T)

root@truenas[~]# gpart show -p -r /dev/da4
=> 34 15628053101 da4 GPT (7.3T)
34 2014 - free - (1.0M)
2048 15628034048 da4p1 6a898cc3-1dd2-11b2-99a6-080020736631 (7.3T)
15628036096 16384 da4p9 6a945a3b-1dd2-11b2-99a6-080020736631 (8.0M)
15628052480 655 - free - (328K)

homer27081990 · Aug 13, 2022

ElGusto said:
Here it is:

Is da4 the same drive as the rest? There are clear differences...

Code:

root@truenas[~]# gpart show -p -r /dev/da1
=> 40 15628053088 da1 GPT (7.3T)

Code:

root@truenas[~]# gpart show -p -r /dev/da2
=> 40 15628053088 da2 GPT (7.3T)

Code:

root@truenas[~]# gpart show -p -r /dev/da3
=> 40 15628053088 da3 GPT (7.3T)

But...

Code:

root@truenas[~]# gpart show -p -r /dev/da4
=> 34 15628053101 da4 GPT (7.3T)

Other things are different, too. I think it is the block size? I am not sure, but in the first 3 its 128 and da4 has 2048? Just out of instinct, though. I can't tell for sure.

ElGusto · Aug 13, 2022

The drives are all Seagate Archive 8TB bought in one order back then. (One of them has a way different serial number from the others, I remember, though).

I have just found, we can get more info out there using 'gpart list':

Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 15628053127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da1p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
efimedia: HD(1,GPT,29d057e3-0bc6-11ed-bd90-000c29077bb3,0x80,0x400000)
rawuuid: 29d057e3-0bc6-11ed-bd90-000c29077bb3
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: da1p2
Mediasize: 7999415652352 (7.3T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e2
efimedia: HD(2,GPT,2affd65a-0bc6-11ed-bd90-000c29077bb3,0x400080,0x3a3412a08)
rawuuid: 2affd65a-0bc6-11ed-bd90-000c29077bb3
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 7999415652352
offset: 2147549184
type: freebsd-zfs
index: 2
end: 15628053127
start: 4194432
Consumers:
1. Name: da1
Mediasize: 8001563222016 (7.3T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5

Geom name: da4
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 15628053134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da4p1
Mediasize: 8001553432576 (7.3T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e2
efimedia: HD(1,GPT,9ce5247f-331b-3148-9b38-c958d2bd057a,0x800,0x3a380e000)
rawuuid: 9ce5247f-331b-3148-9b38-c958d2bd057a
rawtype: 6a898cc3-1dd2-11b2-99a6-080020736631
label: zfs
length: 8001553432576
offset: 1048576
type: !6a898cc3-1dd2-11b2-99a6-080020736631
index: 1
end: 15628036095
start: 2048
2. Name: da4p9
Mediasize: 8388608 (8.0M)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
efimedia: HD(9,GPT,e632cfa5-0957-6344-b0ae-c7e0ecefabbc,0x3a380e800,0x4000)
rawuuid: e632cfa5-0957-6344-b0ae-c7e0ecefabbc
rawtype: 6a945a3b-1dd2-11b2-99a6-080020736631
label: (null)
length: 8388608
offset: 8001554481152
type: !6a945a3b-1dd2-11b2-99a6-080020736631
index: 9
end: 15628052479
start: 15628036096
Consumers:
1. Name: da4
Mediasize: 8001563222016 (7.3T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e3

Edit:

I just checked the serial numbers. The far out one is da1, not da4. So this isn't the culprit.

homer27081990 · Aug 13, 2022

ElGusto said:
The drives are all Seagate Archive 8TB bought in one order back then. (One of them has a way different serial number from the others, I remember, though).

I have just found, we can get more info out there using 'gpart list':

Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 15628053127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da1p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
efimedia: HD(1,GPT,29d057e3-0bc6-11ed-bd90-000c29077bb3,0x80,0x400000)
rawuuid: 29d057e3-0bc6-11ed-bd90-000c29077bb3
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: da1p2
Mediasize: 7999415652352 (7.3T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e2
efimedia: HD(2,GPT,2affd65a-0bc6-11ed-bd90-000c29077bb3,0x400080,0x3a3412a08)
rawuuid: 2affd65a-0bc6-11ed-bd90-000c29077bb3
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 7999415652352
offset: 2147549184
type: freebsd-zfs
index: 2
end: 15628053127
start: 4194432
Consumers:
1. Name: da1
Mediasize: 8001563222016 (7.3T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r2w2e5 Geom name: da4
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 15628053134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da4p1
Mediasize: 8001553432576 (7.3T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e2
efimedia: HD(1,GPT,9ce5247f-331b-3148-9b38-c958d2bd057a,0x800,0x3a380e000)
rawuuid: 9ce5247f-331b-3148-9b38-c958d2bd057a
rawtype: 6a898cc3-1dd2-11b2-99a6-080020736631
label: zfs
length: 8001553432576
offset: 1048576
type: !6a898cc3-1dd2-11b2-99a6-080020736631
index: 1
end: 15628036095
start: 2048
2. Name: da4p9
Mediasize: 8388608 (8.0M)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
efimedia: HD(9,GPT,e632cfa5-0957-6344-b0ae-c7e0ecefabbc,0x3a380e800,0x4000)
rawuuid: e632cfa5-0957-6344-b0ae-c7e0ecefabbc
rawtype: 6a945a3b-1dd2-11b2-99a6-080020736631
label: (null)
length: 8388608
offset: 8001554481152
type: !6a945a3b-1dd2-11b2-99a6-080020736631
index: 9
end: 15628052479
start: 15628036096
Consumers:
1. Name: da4
Mediasize: 8001563222016 (7.3T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e3

Edit:

I just checked the serial numbers. The far out one is da1, not da4. So this isn't the culprit.
View attachment 57661

Sorry for the delay, I'm troubleshooting my PfSense... So, even more so, something is up with da4. Those numbers should not deviate.

homer27081990 · Aug 13, 2022

da1

ElGusto said:
type: freebsd-swap

da4 (zfs here, but you get the idea):

ElGusto said:
type: !6a898cc3-1dd2-11b2-99a6-080020736631

Your GPT seems messed up in da4

homer27081990 · Aug 13, 2022

Those numbers seem to be the start sector for the partition... Is there an extra partition (between 128 and 2048) in da4?

homer27081990 · Aug 13, 2022

What type of array is it? I am sorry for asking and not looking, but at the moment I try to troubleshoot my PfSense FW with a migraine...

ElGusto · Aug 13, 2022

It's Raid Z2.

homer27081990 · Aug 13, 2022

ElGusto said:
It's Z2.

So, Z2 at 4 drives, you can lose 2, correct?

ElGusto · Aug 13, 2022

Yes, two arbitraty drives. That's why I went for Z2 instead of Raid 1.

homer27081990 · Aug 13, 2022

So, you could try wiping da4 and then try to resilver the whole array as a last resort?

homer27081990 · Aug 13, 2022

ElGusto said:
I don't know how I would fix that, though.. would removing, erasing and and committing a replace operation on da4 fix the block size and partitioning to be in line with the others?

After erasure, run a quick surface test

ElGusto · Aug 13, 2022

It is kind of unsettling to wipe one drive more, when two other drives are being resilvered, though. Won't the pool be degaded then?
Well, as you said: last resort. I think I will let the current resilver finish (which is gonna take about 6 days, because SMR is slow) and see if the looping issue is fixed. If it is, I can happily clear da4.

homer27081990 · Aug 13, 2022

ElGusto said:
It is kind of unsettling to wipe one drive more, when two other drives are being resilvered, though. Won't the pool be degaded then?

I am talking theoretically, last resort, nuclear option. Wait for the resilver, first. I am talking after.

Important Announcement for the TrueNAS Community.

z2pool gone crazy after stupid tinkering-no data related issues but: inaccessible GUI options, eternal resilver loop and wrong disk identifiers

Patron

Explorer

Explorer

MVP

Patron

Explorer

Explorer

Patron

Explorer

Patron

Patron

Patron

Patron

Explorer

Patron

Explorer

Patron

Patron

Explorer

Patron

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "z2pool gone crazy after stupid tinkering-no data related issues but: inaccessible GUI options, eternal resilver loop and wrong disk identifiers"

Similar threads