How to set 'missing' GPTID for disk in volume that shows device id (da0) only (TrueNAS-12.0-U1)

[Redcoat]

Try oseek=7809842694

Same result

 

[Patrick M. Hausen]

Got some time to spare? We could wipe the entire partition. I don't get why replace -f would not work in your case.

dd if=/dev/zero of=/dev/da0p2 bs=1m and just let that run until the end. It wipes all data from that particular partition but you still have one level of redundancy, so ...

 

[Redcoat]

Yes, I have time to spare. And I thank you again for spending so much of yours on this with me...

I have Chapter 5 "Repairs and Renovations" of FreeBSD Mastery: ZFS open and have been thinking about possible actions:
I have open drive bays and two cold, tested, new drives on the shelf - could it make sense to cycle those into and then out of the pool is a replacement process for the three outliers? Would that process fix the issue?

And I can certainly try the wipe you suggest - assuming it's a success, what's the step after that? Does a resilver revise the partition info?

 

[Patrick M. Hausen]

You should be able to do the replace operation after the wipe because there will be no ZFS information left on the disk.

 

[Redcoat]

OK, I'm going to give that a go...

Later...

 

[jafin]

Thanks @Patrick M. Hausen

I did as you advised, offline the drive from the pool.
Then I replaced it with itself via the GUI, I tried zpool labelclear but wouldn't execute due to the device still being marked as part of the pool. (I did not try dd the device as I didn't see the thread comments at the time.)

Anyhow, the device is now is resilvering and has been assigned a gptid according to the output of the zpool status.
The drive now shows in gpart list with a freebsd-zfs partition and swap.

Code:

        NAME                                            STATE     READ WRITE CKSUM
        vol1                                            ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/6838364e-5018-11eb-9efa-3ca82a4ba544  ONLINE       0     0     0
            gptid/a987dc98-5204-11eb-ab4d-3ca82a4ba544  ONLINE       0     0     0
            gptid/4a1fd222-b266-11e7-8c66-3ca82a4ba544  ONLINE       0     0     0
            gptid/2e23b965-53d6-11eb-84cb-3ca82a4ba544  ONLINE       0     0     0  (resilvering)
            gptid/962a2678-4e36-11eb-9efa-3ca82a4ba544  ONLINE       0     0     0
            gptid/5448fa00-241f-11e7-8724-3ca82a4ba544  ONLINE       0     0     0

 

[Redcoat]

Well, I'm 16+ hours into the dd, no prompt yet returned in my SSH session. No indication of progress.

One thing I have observed is that, while the periodic snapshots of the degraded pool continue to be made, the replication task to my backup box doesn't run. Is that expected behavior?

EDIT: And I see that scheduled daily Veeam backup jobs for two Windows machines to the degraded pool failed, also.

 

[Patrick M. Hausen]

That is not expected. The pool is online and working, although degraded, so there should be no operational consequences.

As for the progress: press Ctrl-T ...

 

[Redcoat]

Thanks. Ctrl-T returned:

Code:

root@NAS3:~ # dd if=/dev/zero of=/dev/da0p2 bs=1m
dd: /dev/da0p2: short write on character device
dd: /dev/da0p2: end of device
3813400+0 records in
3813399+1 records out
3998639460352 bytes transferred in 33642.505574 secs (118856767 bytes/sec)

Yes, I was doubtful about the non-performing aspects for the pool. I guess that's to be investigated once the pool is healthy again.

Trying the replace again - looks like progress!

Code:

root@NAS3:~ # zpool replace Volume1 da0p2 gptid/9be4e735-bd6a-11e4-8c34-d0509946c5e6
root@NAS3:~ # zpool status -v
  pool: Volume1
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Jan 11 08:01:08 2021
        79.3G scanned at 2.09G/s, 1.01G issued at 27.2M/s, 9.89T total
        0B resilvered, 0.01% done, 4 days 09:48:50 to go
config:

        NAME                                              STATE     READ WRITE CKSUM
        Volume1                                           DEGRADED     0     0     0
          raidz2-0                                        DEGRADED     0     0     0
            replacing-0                                   DEGRADED     0     0     0
              da0p2                                       OFFLINE      0     0     0
              gptid/9be4e735-bd6a-11e4-8c34-d0509946c5e6  ONLINE       0     0     0
            da2p2                                         ONLINE       0     0     0
            gptid/9cd608c6-bd6a-11e4-8c34-d0509946c5e6    ONLINE       0     0     0
            da3p2                                         ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:01:39 with 0 errors on Sat Jan  9 03:46:39 2021
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          ada0p2      ONLINE       0     0     0

errors: No known data errors

So, resilver's in progress -

I'll rinse, repeat for the other two when it's complete.

Thank you again @Patrick M. Hausen

 

[Redcoat]

OK, stage1 objective achieved:

Code:

root@NAS3:~ # zpool status
  pool: Volume1
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: resilvered 2.40T in 11:11:25 with 0 errors on Mon Jan 11 19:12:33 2021
config:

        NAME                                            STATE     READ WRITE CKSUM
        Volume1                                         ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/9be4e735-bd6a-11e4-8c34-d0509946c5e6  ONLINE       0     0     0
            da2p2                                       ONLINE       0     0     0
            gptid/9cd608c6-bd6a-11e4-8c34-d0509946c5e6  ONLINE       0     0     0
            da3p2                                       ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:01:39 with 0 errors on Sat Jan  9 03:46:39 2021
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          ada0p2      ONLINE       0     0     0

Pool's online bur Replication Task won't run - I'll open another thread.

Will work on other two drives meanwhile...

 

[Patrick M. Hausen]

You could try again to speed up the wiping of the old device a bit. After taking e.g. da2p2 offline:
dd if=/dev/zero of=/dev/da2p2 bs=1m count=1; dd if=/dev/zero of=/dev/da2p2 bs=1m oseek=3813300

Let me know if that works. Possibly your drives have 4k sectors natively and did not allow us to seek in units of 512 bytes?

 

[Redcoat]

ds2p2 is resilvering now. I'll try the speedup on da3p2.
Thanks for the suggestion.

 

[Redcoat]

Speedup was not accepted for da3p2. Original dd plan running now.

 

[Redcoat]

Well this exercise was finally completed for the third outlier drive last evening and the pool status now shows 4 drives with gptid's. Thank you again, @Patrick M. Hausen for taking me through that!

I had earlier reported that W10 Veeam backups to the degraded pool and a replication task from it were not functioning while the fix was in progress.

Investigating the backup failure ("target not available") showed that all SMB shares had somehow been toggled to "disabled". Enabling them fixed that.

The existing replication task (which had be running without issue for several months, starting under 11.3) just would not run so I made an exact duplicate task which I started but then realized I had not checked "Start from scratch" so I stopped it, then for good measure deleted all the snapshots on the target to give it a clean start However, this morning it showed Error status and the log contained:
"For task 'task_5' at attempt 5 recoverable replication error RecoverableReplicationError("cannot receive incremental stream: destination 'tank/newnas3backup/10G/iocage/jails/unifi/root' does not exist\nwarning: cannot send 'Volume1/iocage/jails/unifi/root@auto-2021-01-03_20-00-2w': signal received". (Patrick, I note the similarity of this error to that you have been discussing in another thread this morning).

Indeed the "/root" did not exist but surely the replication task should have created it (as it did for some other target locations).

I cleared all the snapshots that had been created in the errored-out replication and restarted the replication task. I have checked on the target and this time the /unifi/root has been created already. This is the first time I ever tried to observe the progress of output of the replication process so I'm not quite sure what to expect (and can't anyway sit here all day watching it...).

 

[Redcoat]

This last replication task rerun looks to have completed cleanly - no errors and with the desired result. In 6 minutes it should replicate the most recent scheduled snapshot task.
And it did so.

 

[denis4o]

@Patrick M. Hausen I have a similar issue, I am trying to replace my 4 x 2TB HDDs with 4 x 8TB HDDs to grow my pool. I was able to offline and replace one of the HDDs which was listed with its gptid. However, the remaining three are listed as dev partitions and I cannot offline and replace them via the GUI or via the CLI:

Code:

zpool status
  pool: freenas-boot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:01:57 with 0 errors on Sun Jan  8 03:46:57 2023
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2       ONLINE       0     0     0

errors: No known data errors

  pool: home_server
 state: ONLINE
  scan: resilvered 828K in 00:00:01 with 0 errors on Thu Jan 12 19:14:18 2023
config:

        NAME                                            STATE     READ WRITE CKSUM
        home_server                                     ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/fb50dea3-9265-11ed-be0b-7085c25f11d1  ONLINE       0     0     0
            ada0p2                                      ONLINE       0     0     0
            ada2p2                                      ONLINE       0     0     0
            ada3p2                                      ONLINE       0     0     0

errors: No known data errors


zpool list -v
NAME                                             SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
freenas-boot                                      14G  5.66G  8.34G        -         -      -    40%  1.00x    ONLINE  -
  da0p2                                         14.1G  5.66G  8.34G        -         -      -  40.4%      -    ONLINE
home_server                                     7.25T  5.88T  1.37T        -         -    26%    81%  1.00x    ONLINE  /mnt
  raidz1-0                                      7.25T  5.88T  1.37T        -         -    26%  81.1%      -    ONLINE
    gptid/fb50dea3-9265-11ed-be0b-7085c25f11d1  7.28T      -      -        -         -      -      -      -    ONLINE
    ada0p2                                      1.82T      -      -        -         -      -      -      -    ONLINE
    ada2p2                                      1.82T      -      -        -         -      -      -      -    ONLINE
    ada3p2                                      1.82T      -      -        -         -      -      -      -    ONLINE

I tried some of the proposed solutions, but to no avail

Code:

root@freenas:~ # zpool labelclear -f ada2p2
failed to open /dev/ada2p2: Operation not permitted

Code:

root@freenas:~ # zpool replace home_server ada3p2 gptid/1b8422cd-b803-11eb-8966-7085c25f11d1
cannot open 'gptid/1b8422cd-b803-11eb-8966-7085c25f11d1': no such device in /dev
must be a full path or shorthand device name

Any help will be really appreciated.