Problems with VDEV autoexpand on Dragonfish or user error?

[eJonny]

I have a pool that has a VDEV that I am attempting to resize by replacing a single drive at a time. It is a 4x12TB Z1 VDEV that where I have replaced all the 12TB drives with 16TB drives. It is in a pool that has three z1 VDEVS (one 16TBx4, one 12TBx4, and this one, which is a 12TBx4 that I'm trying to upgrade to 16TBx4).

I offlined each drive prior to replacement. Removed the offlined drive (I didn't have a spare slot). I slotted the 16TB drive and triggered the replace process. TrueNAS put the VDEV into degraded status each time, but resilvered and the VDEV was healthy again. I repeated this three more times.

At the end of the process I expected that the pool size would increase. But after the 4th drive finished resilvering the pool size did not change.

I am running Dragonfish-24.04-RC.1.

I've done a fair amount of research on others with the same issue. So I've tried the following:
1. I've rebooted TrueNAS Scale
2. I've confirmed autoexpand is turned on with the zpool get autoexpand command
3. I've turned off and turned back on autoexpand with the zpool set autoexpand=off and on commands
4. I've used the zpool online -e command for each of the drives in the VDEV

None of these solutions have resolved my issue. The one clue that I don't understand is why the GUI is showing two VDEVs with the 16TB drives (14.55TiB), but when I run a zpool list -v bigpool command I see the following:

Code:

NAME                                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bigpool                                    145T   113T  32.9T        -         -    15%    77%  1.00x    ONLINE  /mnt
  raidz1-0                                58.2T  44.2T  14.0T        -         -    15%  76.0%      -    ONLINE
    bd75b86c-cac3-4dcb-b5fe-5dca73e3e7d9  14.6T      -      -        -         -      -      -      -    ONLINE
    6c63cbbb-824b-422e-aa4c-c255ec5828fc  14.6T      -      -        -         -      -      -      -    ONLINE
    6185f4f0-4ce7-4316-932c-c0468926d60f  14.6T      -      -        -         -      -      -      -    ONLINE
    e69dfd88-4cac-4fd1-b209-086fe5c41e4c  14.6T      -      -        -         -      -      -      -    ONLINE
  raidz1-1                                43.6T  34.1T  9.51T        -         -    16%  78.2%      -    ONLINE
    2c876bc1-e25c-4a1e-bbb4-05e137db5f42  10.9T      -      -        -         -      -      -      -    ONLINE
    30fab442-6717-4d7e-b439-ea4a31c564aa  10.9T      -      -        -         -      -      -      -    ONLINE
    959f59b3-f4aa-401e-b9b9-1c1ba38cd545  10.9T      -      -        -         -      -      -      -    ONLINE
    65547adc-d97b-414e-af3a-2830cd3b6d22  10.9T      -      -        -         -      -      -      -    ONLINE
  raidz1-2                                43.6T  34.2T  9.43T        -         -    15%  78.4%      -    ONLINE
    e973d868-6056-45d7-b2d9-d7af129dc4f0  10.9T      -      -        -         -      -      -      -    ONLINE
    6bcac0ea-4128-4a11-946c-ef9028b4afa0  10.9T      -      -        -         -      -      -      -    ONLINE
    49650b25-3f74-49b7-a6a3-c5a02179233f  10.9T      -      -        -         -      -      -      -    ONLINE
    7202cd7d-8f7b-4d1b-a8f8-bd4729b4c8b6  10.9T      -      -        -         -      -      -      -    ONLINE

My research hasn't uncovered anything else to try. Any suggestions? Might this be a Dragonfish bug?

 

[danb35]

Any suggestions?

Check the output of lsblk, in case iX still haven't fixed their partitioning SNAFU.

 

[eJonny]

I think you are right. I have run the lsblk command and sorted the drives in the sequence of the VDEVs (the first 4 are my 1st 16TBx4 which is working fine, the second 4 are the new 16TBx4 which won't expand, and the last 4 are the 12TBx4 which is working fine).

Code:

lsblk -o name,partuuid,fstype,size
NAME        PARTUUID                             FSTYPE             SIZE
sdf                                                                14.6T
├─sdf1      0a939918-cd97-4c82-aa42-efc7cd1c0ced linux_raid_member    2G
│ └─md122                                                             2G
│   └─md122                                      swap                 2G
└─sdf2      6185f4f0-4ce7-4316-932c-c0468926d60f zfs_member        14.6T
sdg                                                                14.6T
├─sdg1      f5f59738-5281-4f02-9f58-f904b0a8f8dc linux_raid_member    2G
│ └─md124                                                             2G
│   └─md124                                      swap                 2G
└─sdg2      e69dfd88-4cac-4fd1-b209-086fe5c41e4c zfs_member        14.6T
sdh                                                                14.6T
├─sdh1      ae763085-19f4-4ea1-9a4e-05be56582154 linux_raid_member    2G
│ └─md123                                                             2G
│   └─md123                                      swap                 2G
└─sdh2      6c63cbbb-824b-422e-aa4c-c255ec5828fc zfs_member        14.6T
sdo                                                                14.6T
├─sdo1      1d32ed54-230c-419e-b5f0-fd02f7aa67b3 linux_raid_member    2G
│ └─md124                                                             2G
│   └─md124                                      swap                 2G
└─sdo2      bd75b86c-cac3-4dcb-b5fe-5dca73e3e7d9 zfs_member        14.6T


sdb                                                                14.6T
├─sdb1      42f77ae2-f0ec-4181-828f-8f697c62630b                      2G
└─sdb2      30fab442-6717-4d7e-b439-ea4a31c564aa zfs_member        10.9T
sdc                                                                14.6T
├─sdc1      089a386b-ea4b-4928-a98a-b0d075faf196                      2G
└─sdc2      959f59b3-f4aa-401e-b9b9-1c1ba38cd545 zfs_member        10.9T
sdd                                                                14.6T
├─sdd1      e4bfec43-a190-4554-b74e-4a5aefd4a564                      2G
└─sdd2      2c876bc1-e25c-4a1e-bbb4-05e137db5f42 zfs_member        10.9T
sde                                                                14.6T
├─sde1      ee0d4dae-dc8c-42ad-b71f-474aaa5b2268                      2G
└─sde2      65547adc-d97b-414e-af3a-2830cd3b6d22 zfs_member        10.9T


sdj                                                                10.9T
├─sdj1      795f9b30-346e-428c-b261-45fbb0e6edf5 linux_raid_member    2G
│ └─md125                                                             2G
│   └─md125                                      swap                 2G
└─sdj2      6bcac0ea-4128-4a11-946c-ef9028b4afa0 zfs_member        10.9T
sdl                                                                10.9T
├─sdl1      5bd69252-33ec-472d-b0ac-76eb66fb4bba linux_raid_member    2G
│ └─md125                                                             2G
│   └─md125                                      swap                 2G
└─sdl2      7202cd7d-8f7b-4d1b-a8f8-bd4729b4c8b6 zfs_member        10.9T
sdn                                                                10.9T
├─sdn1      33aa53d8-230e-4844-ae4f-bcea0f09e2ab linux_raid_member    2G
│ └─md123                                                             2G
│   └─md123                                      swap                 2G
└─sdn2      e973d868-6056-45d7-b2d9-d7af129dc4f0 zfs_member        10.9T
sdq                                                                10.9T
├─sdq1      556ae5a8-a2fe-4307-8755-336214cefff9 linux_raid_member    2G
│ └─md122                                                             2G
│   └─md122                                      swap                 2G
└─sdq2      49650b25-3f74-49b7-a6a3-c5a02179233f zfs_member        10.9T

Based on being able to search on "lsblk" and "expand" I found this thread: https://www.truenas.com/community/t...ty-not-expanding-23-10-1-cobia-solved.116337/.

Based on the results of my lsblk do you think I've run into the same partitioning SNAFU?

Do you recommend the remediation steps from that thread:
1. Reinstalled TrueNAS-SCALE-22.12.4.2.
2. Offline the disk,
3. Readded disk and resilver.

I really need to downgrade my install?

 

[chuck32]

I really need to downgrade my install?

It seems like you don't, refer to @danb35 write up here:

SOLVED - **Scale - Expand to fill new larger disks errors and chasing my tail** *Error: Partition(s) 1 on /dev/sdb have been written, but we have been unable*

Scale. OS Version:TrueNAS-SCALE-23.10.1.1 So, I replaced all the disks in a RaidZ2 8 Disk pool. 3 TB to 10 TB. Autoexpand is ON. Nothing happens. I press "expand" in the GUI. I get this error. [EFAULT] Command partprobe /dev/sdb failed (code 1): Error: Partition(s) 1 on /dev/sdb have been...

www.truenas.com

 

[danb35]

do you think I've run into the same partitioning SNAFU?

It looks like it; /dev/sd{bcde} are partitioned as though they're 12 TB disks rather than 16 TB.

Run parted /dev/sdb resizepart 2 100% and repeat for the remaining affected drives.

 

[eJonny]

It looks like it; /dev/sd{bcde} are partitioned as though they're 12 TB disks rather than 16 TB.

Run parted /dev/sdb resizepart 2 100% and repeat for the remaining affected drives.

Looks like you've literally written the guide to avoid this problem. I sincerely appreciate the help resolving. In your guide to manual drive replacement you mention creating the swap partition. It looks like there is a 2G partition, but is it just not named "swap"?

 

[danb35]

It looks like there is a 2G partition, but is it just not named "swap"?

There is the 2G partition, but not only is it not named "swap", it isn't of the correct filesystem type. I don't see that there's a need to bother with it, though.

 

[eJonny]

There is the 2G partition, but not only is it not named "swap", it isn't of the correct filesystem type. I don't see that there's a need to bother with it, though.

Thanks again for all your help. I ran the

Code:

parted /dev/sdb resizepart 2 100%

for each of the drives and viola the storage pool now has double digit TB more space. But I suspect I'll always feel like this VDEV was abused by TrueNAS and hope that they don't develop some type of trauma related problem in the future.

 

[danb35]

I hope not--it's not exactly like you can have it talk with a therapist. But I had to do this myself a few months back and haven't seen any ill effects yet.