Move from Legacy Encryption?

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Yes. Then the replace command will probably work already. I'd try zpool replace default gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2.eli gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
zpool offline again, first? You had one semi-successful zpool online ...
 

Daisuke

Contributor
Joined
Jun 23, 2011
Messages
1,038
@Patrick M. Hausen I'm sorry I'm confused. If I take the disk offline and run label clear, I get the same error, what exactly do I need to run to start again the resilvering? Sorry about the constant spoon feeding requests, I have no idea how to fix the problem.
Code:
root@nas[~]# zpool offline default gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2
root@nas[~]# zpool status default
  pool: default
state: DEGRADED
  scan: resilvered 0B in 02:38:52 with 0 errors on Sun Oct 25 14:44:20 2020
config:

    NAME                                                  STATE     READ WRITE CKSUM
    default                                               DEGRADED     0     0     0
      raidz2-0                                            DEGRADED     0     0     0
        gptid/47702b65-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/496e1d69-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/4f684965-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/526bcc91-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/5380dfbd-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/4f7e5bea-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/435c778b-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        replacing-7                                       OFFLINE      0     0     0  all children offline
          gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2.eli  OFFLINE      0     0     0
          805706285987834672                              OFFLINE      0     0     0  was /dev/gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2
        gptid/5b2e7414-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0
        gptid/592e5f92-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0
        gptid/5a2e8157-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0
        gptid/56c5b521-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0
root@nas[~]# zpool labelclear -f /dev/da10p2
failed to open /dev/da10p2: Operation not permitted
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Since the is a raidz2, should I delete the actual disk and format it? I don't know how to do this but I'm just thinking how to make that disk disappear, at least I know we deal with da10 disk.
Other than removing it from the box I don't currently know how we would go about formatting. If the system doesn't let you "labelclear", that means the disk is somehow "active yet not quite active" so you cannot write to the device via direct device path. There is a sysctl that would permit to dd and wipe the entire permission table, then probably a reboot would recognise the disk as empty - but ... slowly. First things first, we are playing with your data, here.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Sorry about the constant spoon feeding requests, I have no idea how to fix the problem
Me neither, at least not "these steps are precisely what will fix your problem". Sorry. I never had a device in that state. And I don't know why zpool online will not restart the resilver process ...

I do have a good understanding of the fundamental workings of ZFS in general and block devices in FreeBSD in particular. What will most definitely work is:
  • shutdown
  • pull this disk - double check it's the right one
  • clear disk with zeros in another machine
  • put disk back in
  • partition, zpool replace
But possibly somebody else can help who does know how this online yet not quite online situation can be fixed.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
There is this "clear" command to clean an error state of a device, so possibly reverting your last command
Code:
zpool online default gptid/...
zpool clear default gptid/...
will wack it back into "known good" state.

As for your last question - as far as my understanding goes, once you offlined a disk, you should be able to write to it. So, yes. But the system thinks otherwise.
 

Daisuke

Contributor
Joined
Jun 23, 2011
Messages
1,038
@Patrick M. Hausen I made some progress, now the disk shows as corrupted and unavailable:
Code:
# zpool status default
  pool: default
state: DEGRADED
status: One or more devices could not be used because the label is missing or
    invalid.  Sufficient replicas exist for the pool to continue
    functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
  scan: scrub in progress since Sun Oct 25 17:16:11 2020
    7.81T scanned at 7.11G/s, 1.70T issued at 1.55G/s, 7.81T total
    0B repaired, 21.74% done, 01:07:27 to go
config:

    NAME                                                STATE     READ WRITE CKSUM
    default                                             DEGRADED     0     0     0
      raidz2-0                                          DEGRADED     0     0     0
        gptid/47702b65-68bf-11ea-8287-047d7bd5d6a2      ONLINE       0     0     0
        gptid/496e1d69-68bf-11ea-8287-047d7bd5d6a2      ONLINE       0     0     0
        gptid/4f684965-68bf-11ea-8287-047d7bd5d6a2      ONLINE       0     0     0
        gptid/526bcc91-68bf-11ea-8287-047d7bd5d6a2      ONLINE       0     0     0
        gptid/5380dfbd-68bf-11ea-8287-047d7bd5d6a2      ONLINE       0     0     0
        gptid/4f7e5bea-68bf-11ea-8287-047d7bd5d6a2      ONLINE       0     0     0
        gptid/435c778b-68bf-11ea-8287-047d7bd5d6a2      ONLINE       0     0     0
        gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2.eli  UNAVAIL      0     0     0  corrupted data
        gptid/5b2e7414-68bf-11ea-8287-047d7bd5d6a2.eli  ONLINE       0     0     0
        gptid/592e5f92-68bf-11ea-8287-047d7bd5d6a2.eli  ONLINE       0     0     0
        gptid/5a2e8157-68bf-11ea-8287-047d7bd5d6a2.eli  ONLINE       0     0     0
        gptid/56c5b521-68bf-11ea-8287-047d7bd5d6a2.eli  ONLINE       0     0     0

errors: No known data errors

I have a spare disk that I can use the replace the unavailable one. Is there a way to format the disk in place with CLI? I believe the disk shows unavailable because I cannot add it to an encrypted pool.
 
Last edited:

Daisuke

Contributor
Joined
Jun 23, 2011
Messages
1,038
@Patrick M. Hausen I'm back in business, once the disk was fixed, all I had to do is repeat your procedure. :smile:
The clear command did not worked, what fixed the issue was detaching the /dev/da10p2 partition within GUI (which failed) then setting the disk offline from GUI also.
Code:
root@nas[~]# zpool status default
  pool: default
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Oct 25 17:45:25 2020
    2.96T scanned at 67.2G/s, 825M issued at 18.3M/s, 7.81T total
    0B resilvered, 0.01% done, 5 days 03:56:41 to go
config:

    NAME                                                  STATE     READ WRITE CKSUM
    default                                               DEGRADED     0     0     0
      raidz2-0                                            DEGRADED     0     0     0
        gptid/47702b65-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/496e1d69-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/4f684965-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/526bcc91-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/5380dfbd-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/4f7e5bea-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/435c778b-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        replacing-7                                       DEGRADED     0     0     0
          gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2.eli  OFFLINE      0     0     0
          gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2      ONLINE       0     0     0
        gptid/5b2e7414-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0
        gptid/592e5f92-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0
        gptid/5a2e8157-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0
        gptid/56c5b521-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0

errors: No known data errors
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
OK, you could replace it with a spare, check the output of dmesg to make sure which device the new one is going to be. Let's assume the new one is da10 just like the old one.

1. Find the device for one of your already unencrypted disks, e.g. gptid/435c778b-68bf-11ea-8287-047d7bd5d6a2 Let's assume it's da8.
2. Copy the partition layout from the present disk to the new one: gpart backup da8 | gpart restore -F da10. Double check the devices before doing that.
3. Inspect the partition table of da10 to get the UUID: gpart list da10
4. Do the zpool replace as usual
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740

Daisuke

Contributor
Joined
Jun 23, 2011
Messages
1,038
What did you do? zpool online followed by zpool clear?
I updated the previous post, sorry. I figured is important to let you know. The online and clear commands did not do anything:
Code:
root@nas[~]# zpool offline default gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2
root@nas[~]# zpool online default gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2
warning: device 'gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present
root@nas[~]# zpool clear default gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2
root@nas[~]# zpool status default
  pool: default
state: DEGRADED
  scan: scrub in progress since Sun Oct 25 17:16:11 2020
    7.81T scanned at 14.2G/s, 776G issued at 1.38G/s, 7.81T total
    0B repaired, 9.71% done, 01:27:06 to go
config:

    NAME                                                  STATE     READ WRITE CKSUM
    default                                               DEGRADED     0     0     0
      raidz2-0                                            DEGRADED     0     0     0
        gptid/47702b65-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/496e1d69-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/4f684965-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/526bcc91-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/5380dfbd-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/4f7e5bea-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        gptid/435c778b-68bf-11ea-8287-047d7bd5d6a2        ONLINE       0     0     0
        replacing-7                                       UNAVAIL      0     0     0  insufficient replicas
          gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2.eli  OFFLINE      0     0     0
          805706285987834672                              UNAVAIL      0     0     0  was /dev/gptid/5b43ebbf-68bf-11ea-8287-047d7bd5d6a2
        gptid/5b2e7414-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0
        gptid/592e5f92-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0
        gptid/5a2e8157-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0
        gptid/56c5b521-68bf-11ea-8287-047d7bd5d6a2.eli    ONLINE       0     0     0

errors: No known data errors

The GUI did something to /dev/da10p2 partition, even if it failed as when I set the disk offline from GUI, it was in the state your procedure expected to be.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Glad it worked out.
 

Daisuke

Contributor
Joined
Jun 23, 2011
Messages
1,038
@Patrick M. Hausen thank you for all your precious help, much appreciated. I have all my pools now unencrypted, everything is clean. I don't think I will ever use encryption anymore. When I imported the pools, I was presented with the Encrypt option, which I did not select. Out of curiosity, I cannot see anywhere that Pool Encrypt option anymore. Is this a one time selection, I will have to export the pool and import it, then select the Encrypt option?

1603778719122.png


1603775694663.png
 
Last edited:

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Encryption is per dataset now:

Bildschirmfoto 2020-10-27 um 08.34.19.png
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
You wrote "pool encrypt option" :wink:

So
  • encryption is per dataset
  • it can only be enabled at create time
How should ZFS go about encrypting a dataset full of valuable files after the fact? That would mean rewriting everything - and ZFS has a design policy to never rewrite data at rest. Device removal is already violating that, though.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
  • encryption is per dataset
  • it can only be enabled at create time
Sounds like what you're saying (without actually spelling it out) is to create a new (encrypted) dataset and zfs send | recv into it, then optionally at the end, rename... I guess you could rename, create the encrypted one with the original name, then send | recv.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
@sretalla If you intend to encrypt existing data, yes.
@TECK did not provide a complete user story :wink:
 
Top