Replaced disks with larger capacity - not expanding - 23.10.1 Cobia (SOLVED)

[jafin]

Hello,

I have a nas running SCALE 23.10.1 cobia
I replaced 2 4TB disks with 2x 8TB drives.
The smallest drive in the array is now 6TB. (these will be replaced with 8TB once budget allows)
Autoexpand is enabled on the pool. After the resilver completed I expected the pool to expand to 6T capacity per drive. It did not, stayed at 3.6T

The closest related issue I can see is this (vdev mirror not expanding)

troubleshooting mirror vdev won't expand

I appear to have a similar situation as @ https://www.truenas.com/community/threads/replaced-mirror-with-larger-drives-but-cant-expand-pool.112702/ but that thread ended with inconclusively. I have a pool, tank1. It is 6 disks, 3x 2-wide mirrors. originally, it was 6x 8tb drives. I replaced...

www.truenas.com

Code:

# zpool get autoexpand
NAME       PROPERTY    VALUE   SOURCE
boot-pool  autoexpand  off     default
tank1      autoexpand  on      local

Code:

# zpool list -v
NAME                                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool                                   95G  4.70G  90.3G        -         -     0%     4%  1.00x    ONLINE  -
  sde3                                    95.3G  4.70G  90.3G        -         -     0%  4.94%      -    ONLINE
tank1                                     21.8T  19.9T  1.82T        -         -    20%    91%  1.00x    ONLINE  /mnt
  raidz2-0                                21.8T  19.9T  1.82T        -         -    20%  91.7%      -    ONLINE
    a55e4fd3-41f5-4557-853b-2c07724d3ad8  3.64T      -      -        -         -      -      -      -    ONLINE
    sdf2                                  7.28T      -      -        -         -      -      -      -    ONLINE
    sdg2                                  5.46T      -      -        -         -      -      -      -    ONLINE
    sda2                                  5.46T      -      -        -         -      -      -      -    ONLINE
    sdc2                                  7.28T      -      -        -         -      -      -      -    ONLINE
    06e895bd-c236-4786-8bd9-70ac185b85f1  3.64T      -      -        -         -      -      -      -    ONLINE

a55e4fd3-41f5-4557-853b-2c07724d3ad8 and 06e895bd-c236-4786-8bd9-70ac185b85f1 are the replacement 8TB drives, still showing at 3.64T

Code:

# lsblk -o name,partuuid,fstype,size
NAME     PARTUUID                             FSTYPE       SIZE
sda                                                        5.5T
├─sda1   fe7b125c-f956-11ed-ab1b-3497f600e007                2G
└─sda2   fe8fb054-f956-11ed-ab1b-3497f600e007 zfs_member   5.5T
sdb                                                        7.3T
└─sdb1   a55e4fd3-41f5-4557-853b-2c07724d3ad8 zfs_member   3.6T
sdc                                                        7.3T
├─sdc1   c3afa5f9-9171-11ee-9c2b-3460f94eb9a8                2G
└─sdc2   c3c6b633-9171-11ee-9c2b-3460f94eb9a8 zfs_member   7.3T
sdd                                                        7.3T
└─sdd1   06e895bd-c236-4786-8bd9-70ac185b85f1 zfs_member   3.6T
sde                                                      111.8G
├─sde1   e3eea168-2e4c-4294-b4f8-730e228c4147                1M
├─sde2   aa69105b-7aa4-48f4-ab0a-f09dc760cc7c vfat         512M
├─sde3   2b96e399-88fa-4498-8d13-219b64ddb5b3 zfs_member  95.3G
└─sde4   2fc39878-08c7-4021-9289-2595ed6bbc59               16G
  └─sde4                                      swap          16G
sdf                                                        7.3T
├─sdf1   0782ce1e-b1c9-11ee-a75b-3460f94eb9a8                2G
└─sdf2   0796cfbd-b1c9-11ee-a75b-3460f94eb9a8 zfs_member   7.3T
sdg                                                        5.5T
├─sdg1   758f0d8f-fa84-11ed-afbe-3497f600e007                2G
└─sdg2   75a2b9a5-fa84-11ed-afbe-3497f600e007 zfs_member   5.5T

Looking at the above, there appears to be no swap partition on the replaced disks.

To my knowledge, I replaced and resilvered all drives via the GUI. The system is an update from truenas core.
I'm uncertain why some of the drives are UUID and others SDX labels.
I am uncertain why the replaced drives do not have swap partitions. (is this a bug?)

Any clues what my next steps are to fix the expand?
Should swap partitions have been created when replaced? (I understand swap was a way to accommodate small differences in similar sized drives, which seems useful)
The forum suggested export/import each drive? Doing this from CLI will this upset truenas?

I have done expands before in core without incident.

 

[jafin]

Oh, it appears I've hit https://www.truenas.com/community/threads/problem-expanding-pool-after-drive-upgrade.115699/
Linked to https://ixsystems.atlassian.net/browse/NAS-126809
Fixed in:
SCALE-24.10-ALPHA.1 (ElectricEel)
SCALE-23.10.2
SCALE-24.04-RC.1 (DragonFish)

 

[jafin]

Reinstalled TrueNAS-SCALE-22.12.4.2.
Offline the disk,
Readded disk and resilver.
Appears the partitions are now the correct size, including the 2G swap.

Code:

sdb                                                                  7.3T
├─sdb1      7435a050-d564-4243-88eb-a15980084cb8                       2G
└─sdb2      a89e655c-19e2-4bfd-bf5b-d70ca75cbe65 zfs_member          7.3T

Will stay on this version for a while until upstream ver gets fixed.