Ok, I've replaced another disk and followed this guide to fix it.
- These steps are for FreeNAS 8.3.1-p2 only. I didn't verify these steps work on any other version of FreeNAS. You are welcome to try but you are on your own if things go ugly(aka... keep backups!).
- You must be able to use the key+passphrase to mount the encrypted zpool since the recovery key will not be fully functional during this procedure. You should always be using the key+passphrase for this guide except at the very end when I specifically say to use the recovery key.
- As always.. you should have backups! Yes, I had to say it twice.
- I am not responsible if you mess this up so badly you lose your data. This worked for me and I make no guarantees that it will work for you.
- You will get a lot of "An Error Occured" messages in the GUI while we are doing all of this. Based on the other posts in this thread they can be safely ignored. If you are concerned you can look at the footer messages to see if it is anything to worry about.
- The following steps won't be performed all at once since the resilvering process can be quite long. I recommend you print out these instructions so you can check them off as you do them. Skipping steps can cause very undesirable effects.
Here's step by step for those that aren't familiar with this stuff and confused:
Here's the steps I did that are basically from the manual, but in my own words with a little emphasis on dealing with the encrypted zpool(This is similar to section 6.3.11 of the FreeNAS manual).
1. Backup your FreeNAS config file. ( System -> Settings -> "Save Config" )
2. Mount the encrypted zpool if it is not mounted already.
3. Identify the disk you wish to replace. Using any combination of device designation(for example: /dev/ada1), the gptid (for example: gptid/6fbb91d5-4a95-11e2-bca4-0015171496ae), the disk serial number(depends on your hard drive model), and the label printed on the hard drive you can determine for certainty which disk is the one to replace. For verification purposes you should write down the serial number of the disk to be replaced to ensure you aren't offlining one disk from the array and then physically removing a different disk.
Your data could depend on you verifying the correct disk is removed. All of the information above can be matched to other data using the commands:
zpool status (lists gptids used by the zpool)
smartctl -a /dev/adaXX (matches device name to disk serial number) This line may have different parameters if you are using any kind of RAID controller. Consult the
manpage for smartctl for assistance.
gpart show -r (matches gptid to /dev/adaXX)
4. In the FreeNAS GUI click on Storage -> Volumes -> View Volumes. Click on the "Volume Status" button for the applicable zpool that will have a disk replaced. Click "Offline" next to the partition that is part of the disk to be replaced. For example, if you are removing da1 you should look for something like da1p2(this stands for da1 partition 2).
5. If you can hot-swap your drives now is the time swap disks. I don't trust hot-swap in FreeBSD so I always shutdown my server. By shutting down the server this also lets me ensure I'm not physically removing a different disk thereby removing another disk from my zpool. In some cases removing the wrong disk with the system online can result in permanent dataloss. Verify that the serial number of the disk you removed matches the serial number you wrote down in step 2.
6. Install your new disk. Power on the system.
7. From the FreeNAS GUI mount your encrypted zpool. ( Storage -> Volumes -> View Volumes )
8. In the FreeNAS GUI click on Storage -> Volumes -> View Volumes. Click on the "Volume Status" button for the applicable zpool. Click the "Replace" button next to the disk that has been removed. Note that it will not give a device and partition. It will be easy to spot because it should be the only disk that doesn't have an "Edit", "Replace" and "Offline" button on its row. In the window that pops up choose the new disk you wish to use and click "Replace Disk".
Resilvering will automatically begin. Resilvering can take anywhere from a few minutes to many days depending on the speed and amount of data in your zpool. You can monitor the resilvering process with the command "zpool status | grep scan". When the resilvering process is complete continue with this guide.
9. In the FreeNAS GUI click on Storage -> Volumes -> View Volumes. Click on the "Volume Status" button for the applicable zpool. Click the "Detach" button for the old disk to complete the disk replacement procedure.
NOTE: All steps after this note are not part of the manual but should repair your encrypted zpool's FreeNAS database information to allow for proper rekeying. This will delete the extra gptid from the FreeNAS database preventing further errors and allowing you to successfully rekey the zpool.
10. Save a list of your gptids for your zpool by using the command
zpool status (yourzpoolname) > /mnt/yourzpoolname/somewherethatisshared/gpt.txt. Then copy the text file to your desktop or print it out. Alternately you can choose to write down the gptids on a sheet of paper by typing the command
zpool status. Make sure that if you have a large number of disks that you scroll up to see them all.
*** Remember that you are directly editing the FreeNAS configuration file and can cause serious damage if you have any typos. ***
11. I rebooted my server at this point and left the encrypted zpool unmounted in case there is any inconsistencies within FreeNAS while we make our changes. You can choose not to reboot at your own risk. SSH into your box and run the commands as shown. The bold is what I typed:
Code:
# sqlite3 /data/freenas-v1.db
SQLite version 3.7.13 2012-06-11 02:05:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select * from storage_encrypteddisk;
2|2|24|gptid/c1560e68-b86a-11e2-9cfb-0015171496ae
3|2|30|gptid/c1b125e6-b86a-11e2-9cfb-0015171496ae
4|2|31|gptid/c20a0232-b86a-11e2-9cfb-0015171496ae
5|2|33|gptid/c25ef2da-b86a-11e2-9cfb-0015171496ae
6|2|32|gptid/c2b79c49-b86a-11e2-9cfb-0015171496ae
8|2|24|gptid/991f4916-ba6f-11e2-9ed2-0015171496ae
9|2|24|gptid/7b7cf9d9-cda0-11e2-aac8-0015171496ae
sqlite>.quit
# zpool status
pool: encryptedzpool
state: ONLINE
scan: resilvered 0.45GB in 1m with 0 errors on Wed Jun 5 13:39:04 2013
config:
NAME STATE READ WRITE CKSUM
encryptedzpool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/7b7cf9d9-cda0-11e2-aac8-0015171496ae.eli ONLINE 0 0 0
gptid/c1560e68-b86a-11e2-9cfb-0015171496ae.eli ONLINE 0 0 0
gptid/c1b125e6-b86a-11e2-9cfb-0015171496ae.eli ONLINE 0 0 0
gptid/c20a0232-b86a-11e2-9cfb-0015171496ae.eli ONLINE 0 0 0
gptid/c25ef2da-b86a-11e2-9cfb-0015171496ae.eli ONLINE 0 0 0
gptid/c2b79c49-b86a-11e2-9cfb-0015171496ae.eli ONLINE 0 0 0
errors: No known data errors
12. Now to determine what entry in the FreeNAS databse shouldn't be there. If you look at the sql database there are 7 entries while the zpool status lists only 6. The extra entry in the database shouldn't be there. In my case, the extra entry is this line:
Code:
8|2|24|gptid/991f4916-ba6f-11e2-9ed2-0015171496ae
Notice that the line id(the first number) is 8. That's the line we are going to delete.
13. Verify that your encrypted zpool is not mounted. I don't think there is any risk to doing this with the zpool mounted, but its better to be safe than sorry. If it is mounted reboot the server.
14. Run the commands as I demonstrate below, but use your line number:
Code:
# sqlite3 /data/freenas-v1.db
SQLite version 3.7.13 2012-06-11 02:05:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select * from storage_encrypteddisk;
2|2|24|gptid/c1560e68-b86a-11e2-9cfb-0015171496ae
3|2|30|gptid/c1b125e6-b86a-11e2-9cfb-0015171496ae
4|2|31|gptid/c20a0232-b86a-11e2-9cfb-0015171496ae
5|2|33|gptid/c25ef2da-b86a-11e2-9cfb-0015171496ae
6|2|32|gptid/c2b79c49-b86a-11e2-9cfb-0015171496ae
8|2|24|gptid/991f4916-ba6f-11e2-9ed2-0015171496ae
9|2|24|gptid/7b7cf9d9-cda0-11e2-aac8-0015171496ae
sqlite> delete from storage_encrypteddisk where id=8;
sqlite> select * from storage_encrypteddisk;
2|2|24|gptid/c1560e68-b86a-11e2-9cfb-0015171496ae
3|2|30|gptid/c1b125e6-b86a-11e2-9cfb-0015171496ae
4|2|31|gptid/c20a0232-b86a-11e2-9cfb-0015171496ae
5|2|33|gptid/c25ef2da-b86a-11e2-9cfb-0015171496ae
6|2|32|gptid/c2b79c49-b86a-11e2-9cfb-0015171496ae
9|2|24|gptid/7b7cf9d9-cda0-11e2-aac8-0015171496ae
sqlite> .quit
15. Reboot the server again from the GUI or from the command line with
shutdown -r now.
16. When the server has booted back up mount your encrypted zpool again. Verify that your zpool is showing as HEALTHY. If it is then all of the drives were properly decrypted. If the zpool is not showing as healthy stop and do not continue.
17. After the zpool is mounted click the "Add Recovery Key" button. A file named "geli_recovery.key" should be available to download.
At this point both your recovery key and the key+passphrase method
should again allow your zpool to mount. I recommend you verify that both function and then store them in a safe place. Remember that the old recovery key will no longer work and if you keep it in a secure second location then you will need to update that location also.
18. To verify that the recovery key works you simply need to reboot and mount the encrypted zpool again using the recovery key.
19. To verify the key+passphrase work simply reboot again and mount the encrypted zpool with your key+passphrase. If the zpool status is HEALTHY then everything is working.
