SOLVED Help with VDEV's and Check my Understanding

[kranzel]

Hello all,

Looking for some clarifications and possibly to double check my plan. Thanks in advance for any help.

Item #1. I have been reading the storage section of the FreeNAS 11.2-U3 manual. Specifically, I am locking at expanding a pool by adding a second VDEV. Currently, I have one pool with a single VDEV. The VDEV is 6 x8TB WD RED in a RaidZ2 configuration. I am at the 90% mark and plan to add a new 6x8TB WD RED RaidZ2 VDEV to that pool.

Question 1: The instructions on adding a VDEV are clear, the thing I cannot seem to find is how the storage will be managed after I add the VDEV. Will FreeNAS automatically shift data from my first VDEV to the new VDEV to ensure I am below the 80% threshold? Or do I do this manually somehow?

Question 2: Going forward, can I assume FreeNAS will handle balancing the storage across the VDEV’s? Or is this something I must watch for?

Item#2 Before I commit to the adding the second VDEV, I am considering moving to an encrypted state. To do this, I believe the actions would be as follows,

Create a new pool, let’s call it NewPool, with my new VDEV and encrypt it.
Copy data from existing or OldPool to NewPool. I have seen a few guides out there on how to copy the data, so I think I have that down.
Once data is copied. Destroy OldPool and move the VDEV to NewPool where encryption will occur once added.
Backup Key’s etc.

I realize there are details in there such as ensuring to properly transfer the data etc. But form a high level, is that correct?

 

[sretalla]

Question 1: The instructions on adding a VDEV are clear, the thing I cannot seem to find is how the storage will be managed after I add the VDEV. Will FreeNAS automatically shift data from my first VDEV to the new VDEV to ensure I am below the 80% threshold? Or do I do this manually somehow?

ZFS manages this for you (but won't balance the content if you already have existing data, which means most writes will go only to the new VDEV until they are about even).

Question 2: Going forward, can I assume FreeNAS will handle balancing the storage across the VDEV’s? Or is this something I must watch for?

See answer 1

You need to forget about managing content on VDEVs and think about pools and datasets. ZFS does the rest for you.

If you really want/need to see a balanced pool, you need to start a fresh one with the correct count of VDEVs and put the content on there from the beginning (backup, destroy, recreate, restore)

 

[sretalla]

Create a new pool, let’s call it NewPool, with my new VDEV and encrypt it.

VDEVs don't exist outside pools.

You can't transfer a VDEV between pools. (since it stops existing as soon as it leaves the first one... which by the way destroys the first pool anyway even if there are more VDEVs in it).

 

[kranzel]

VDEVs don't exist outside pools.

You can't transfer a VDEV between pools. (since it stops existing as soon as it leaves the first one... which by the way destroys the first pool anyway even if there are more VDEVs in it).

Thanks for all the info. I didn't realize VDEV's didn't exist outside the pool. Up until now, I have only ever had a single pool with a single VDEV. I assumed the two were "separable". good to know.

So, just to make sure I understand this part correctly.

My existing pool, which I will call OldPool contains a single VDEV and the pool is not encrypted.

If I want to encrypt my stuff, I would perform the following.

1. Create a new Pool, called NewPool and add my newly acquired 6x8TB drives to the pool in a single RaidZ2 VDEV and encrypt it.
2. Migrate all data from OldPool to NewPool.
3. Detroy OldPool
4. Add the 6x8TB drives that have been made available to by destroying OldPool to NewPool as a new RaidZ2 VDEV. Because it is being added to an encrypted pool, it will also be enxrypted.
5. Back up keys.

Sound about right?

 

[sretalla]

This sounds good, but I'm not an expert on encrypted pools. I recall seeing something about needing to re-key when replacing a disk, not sure if re-keying is needed with a new VDEV addition.

 

[Chris Moore]

Sound about right?

When you add or replace disks in an encrypted pool, you need to re-key the pool to ensure that all disks are keyed the same. It is a precaution against being locked out of your own data. Be very cautious about encrypting. People have lost access to their data this way.

I didn't realize VDEV's didn't exist outside the pool. Up until now, I have only ever had a single pool with a single VDEV. I assumed the two were "separable". good to know.

A pool is made up of vdevs and the pool could be thought of as a logical container for vdevs, but the vdev can't be separated from the pool. A vdev is a 'virtual device' made up of containers, could be disks, partitions on disks, or even files in a partition on a disk. ZFS is super flexible. In FreeNAS, the system middle-ware creates a ZFS partition on the disk and the partitions are used to make the vdev, referenced by the gptid of the partition. The middle-ware also creates a swap partition on each of your data drives.
If you encrypt your pool, a .eli extension will be added to the gptid in your zpool list output which would look kind of like this:

Code:

root@freenas:~ # zpool import
   pool: VolumeName
     id: 5057694033267979932
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
    devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
config:

    VolumeName                                     UNAVAIL  missing device
      mirror-1                                          ONLINE
        gptid/6b64892f-1140-11e6-980e-d0509913e8a9.eli  ONLINE
        gptid/6c49e955-1140-11e6-980e-d0509913e8a9.eli  ONLINE

    Additional devices are known to be part of this pool, though their
    exact configuration cannot be determined.

PS. This listing is from a pool that can't be accessed because the keys were lost.

 

[Chris Moore]

Also useful to know. You can use the command glabel status to get a list of the gptid and associated partition like this:

Code:

root@freenas:~ # glabel status
                                      Name  Status  Components
gptid/8f3569c9-7489-11e5-8705-d0509913e8a9     N/A  da0p1
gptid/6a45ab6d-1140-11e6-980e-d0509913e8a9     N/A  ada0p2
gptid/695ab06c-1140-11e6-980e-d0509913e8a9     N/A  ada1p2
gptid/6c49e955-1140-11e6-980e-d0509913e8a9     N/A  ada2p2
gptid/6b64892f-1140-11e6-980e-d0509913e8a9     N/A  ada3p2

 

[kranzel]

Sorry for the delay. Everything is working. Thanks again for all your help.