Expanding an Encrypted Pool Storage

[okynnor]

I've been googling on how to expand my pool storage by replacing the drives one at a time. It seems that by trying to offline the first drive, TrueNAS gave me the following warning: Warning: Disk cannot be offlined in encrypted pools. [confirm] [cancel] [offline]

According to this post, it suggests to be OK to force offlining the disk drive.

However, from the post, it doesn't mention the said warning though. Therefore, I would like to confirm by asking, what's going to happen if I actually confirm to TrueNAS to offline the encrypted drive (ada1)

Would love it if I could get exact step instructions to remove and upgrade the zpool. The new drives are 12TB Seagate NAS drives.

My TrueNAS specs are as follows:

Code:

    NAME                                                STATE     READ WRITE CKSUM
    FreeNASBIGPOOL                                      ONLINE       0     0     0
      mirror-0                                          ONLINE       0     0     0
        gptid/a46ecd4e-1d0e-11ea-bd47-1c1b0d6be5c4.eli  ONLINE       0     0     0
        gptid/a544a3f5-1d0e-11ea-bd47-1c1b0d6be5c4.eli  ONLINE       0     0     0
      mirror-1                                          ONLINE       0     0     0
        gptid/447bc0f1-1d0f-11ea-bd47-1c1b0d6be5c4.eli  ONLINE       0     0     0
        gptid/49f0aed7-1e0b-11ea-9564-1c1b0d6be5c4.eli  ONLINE       0     0     0

Running TrueNAS-13.0-U3.1
32GB of ECC RAM
All 4 4TB hard drives were put in service in 2019.
Intel(R) Xeon(R) CPU D-1521 @ 2.40GHz

references:

• Docs Hub: Disk Replacement

 

[winnielinnie]

Running TrueNAS-13.0-U3.1

I would be tentative before proceeding.

I'm not entirely sure if TrueNAS Core 13+ will even allow you, via the GUI, to encrypt an entire drive/partition with GELI, in order to add it to a vdev.

EDIT: It might try to do so, but even with earlier versions (i.e, "FreeNAS"), I remember there being a weird song-and-dance about having to "re-key" the encrypted drives if you wanted to replace a drive or expand your pool.

You might have better luck with FreeNAS 11.3 (unless you've already "upgraded" the pool's features).

Even with this disclaimer, I wouldn't feel comfortable.

Although GELI encryption is deprecated, TrueNAS implements GELI encryption during a “GELI-Encrypted (Legacy) pool” disk replacement. TrueNAS uses GELI encryption for the lifetime of that pool, even after replacement.

 

[winnielinnie]

I've been googling on how to expand my pool storage by replacing the drives one at a time.

Is it feasible for you to take advantage of this opportunity (assuming you purchased additional drives), to create a new pool using native ZFS encryption, then replicate all your data to the new pool, and then re-purpose the old drives to expand your new pool?

 

[okynnor]

I would be tentative before proceeding.

I'm not entirely sure if TrueNAS Core 13+ will even allow you, via the GUI, to encrypt an entire drive/partition with GELI, in order to add it to a vdev.

EDIT: It might try to do so, but even with earlier versions (i.e, "FreeNAS"), I remember there being a weird song-and-dance about having to "re-key" the encrypted drives if you wanted to replace a drive or expand your pool.

You might have better luck with FreeNAS 11.3 (unless you've already "upgraded" the pool's features).

Even with this disclaimer, I wouldn't feel comfortable.

Luckily, I haven't upgraded the pool features.
What can be done if I haven't upgraded the pool's features?

PS: I have data in that TrueNAS which I would hate to have to dump onto a backup -- it's a lot of work!
And same as @winnielinnie suggestion of building another TrueNAS server.

 

[winnielinnie]

Not sure why I missed this notification.

I was implying that you can do what I did. Take the opportunity of wanting to expand the pool to do a "shuffle" into a new pool with native ZFS encryption. As long as you can keep the current pool's drives connected, and add two (or however many) new drives.

Then you replicate everything to the new pool using native encryption (might be better to use an SSH + tmux session). After completed, you can unplug the old drives, and then you're good to go. Or re-purpose the old drives to expand the new pool, so long as you have a backup of all your data.

 

[okynnor]

I saw this, brand new, video from Tom @ Lawrence Systems.

It appears that all he did was use the "replace" option. It seems so plug and play.

Is this something that is good to go? Or are there downsides? Are there missing steps?

UPDATE: This step isn't going to work for my use case. The reason is because I do not have any "hot spares" in my 4 drive dedicated NAS box.

 

[okynnor]

Not sure why I missed this notification.

I was implying that you can do what I did. Take the opportunity of wanting to expand the pool to do a "shuffle" into a new pool with native ZFS encryption. As long as you can keep the current pool's drives connected, and add two (or however many) new drives.

Then you replicate everything to the new pool using native encryption (might be better to use an SSH + tmux session). After completed, you can unplug the old drives, and then you're good to go. Or re-purpose the old drives to expand the new pool, so long as you have a backup of all your data.

Sadly, my 4 bay NAS box doesn't have any space.

If it can be confirmed that it's safe to "offline" a single drive in an encrypted pool, It's a good step forward.

If it can't be done, I'll build a new TrueNAS box and migrate all my data there.

 

[okynnor]

I'm still trying.

Since I haven't see anymore replies, I thought I'd document my journey.

Because my Pool is encrypted with the GLI key, I think that's why I"m getting the warning about off lining one of the drives that are in a video mirror.
To protect my data, I will use the replication task function to copy all my data to another TrueNAS sever that will have enough space. It will be in a stipe configuration because those older HDDs are of smaller capacity. I just hope that they don't fail. Then I will be really screwed.
I will download my TrueNAS config and the encryption keys.
Next, I will for one of the mirrored drives to offline, replace, and resilver. If that works, I will move on to the next one.
If that fails, then my business will have to try to run on the "holding server" while I rebuild the server with the new 14TB drives.
Rinse and repeat by moving the data.

wish me luck!