Encryption Debugging Tips

PhiloEpisteme

Guru

Mar 1, 2020

PhiloEpisteme submitted a new resource:

Encryption Debugging Tips - Basic commands and suggestions for debugging encryption issues

I'm not making any argument for or against encryption in this post. Discussion of the merits of encryption in general or FreeNAS's specific use/implementation is beyond the scope of this guide.

To by knowledge the commands below are good on all versions of 11.2 and possibly even 11.3. I have not tested them on older versions and have only limited testing on 11.3. Despite this, the commands blow are non-destructive and reversible.

Encryption problems typically fall into 3 categories...

Read more about this resource...

Don't forget to like quality questions and replies! :)

Case: ~~Fractal Design Define R6 (Build Report)~~ Supermicro CSE-846
Backplane: ~~SAS846EL2~~ SAS2-846-EL1
FreeNAS Release: FreeNAS-11.3-RELEASE-U4.1
Board: SuperMicro X11SSM-F, LGA 1151/Socket, IPMI, 2x GbE Intel i210-AT
Processor: Intel Core i3-7100 @ 3.90GHz, 2 Cores, 4 Threads
Memory: 64 GB of 4x16GB Crucial CT16G4WFD8266 DDR4-2666 ECC UDIMM
NIC: Chelsio T520-CR
HBA: Supermicro AOC-SLG3-2M2 PCIe3.0x8 -> 2x M.2 @ PCIe3.0x4
HBA: LSI/Broadcom SAS9207-8i, 6Gbps SAS PCI-E 3.0 HBA, Firmware 20.00.07.00
Storage Pool 1:

vdev-0 mirror: 2x 5900rpm 3TB Seagate IronWolf ST3000VN007

vdev-1 mirror: 2x 5900rpm 3TB Seagate IronWolf ST3000VN007

vdev-2 mirror: 2x 5900rpm 3TB Seagate IronWolf ST3000VN007

Storage Pool 2 (encrypted):

vdev-0 raidz2:

3x 7200rpm 2TB mixed desktop drives

3x 5900rpm 2TB Seagate IronWolf ST2000VN007

Boot pool: 1 mirror vdev, 2 x 64GB SuperMicro SATA DOM SSD-DM064-SMCMVN1

Storage Pool 1 SLOG

2 x 480GB Samsung 883 DCT

Storage Pool 2 SLOG

2 x 100|200GB Intel Optane SSD DC P4801X

Chassis: Supermicro CSE-836TQ 16 x hot-swap bays
FreeNAS Release: FreeNAS-11.3-U4.1
Board: SuperMicro X9SCM-F, LGA 1155), IPMI, 2x GbE Intel 82579LM/82574L
Processor: Intel Xeon E2-1220 @ 3.10 GHz, 4 Cores, 4 Threads
Memory: 8 GB of 1x8GB Crucial CT102472BD160B DDR3 SDRAM-1600 ECC UDIMM DDR3L
HBA: LSI/Broadcom SAS9210-8i, 6Gbps SAS PCI-E 2.0 HBA, Firmware 20.00.07.00
Expander: IBM 46M0997 16-Port SAS2, Firmware 634A
Backplane: Supermicro BPN-SAS-836TQ Direct Attached SAS2 Backplane
NIC: Chelsio T520-CR
Storage Pool 1: rotated off-site with Pool 2

vdev-0 mirror: 2x 5400rpm 8TB Seagate BaraCuda ST8000DM004

Storage Pool 2: rotated off-site with Pool 1

vdev-0 mirror: 2x 5400rpm 8TB WD Red WD80EFAX

Storage Pool 3: rotated off-site with Pool 4

vdev-0 mirror: 2x 5400rpm 8TB WD Red WD80EFAX|WD80EFZX

Storage Pool 4: rotated off-site with Pool 3

vdev-0 mirror: 2x 5400rpm 8TB WD Red WD80EFAX

Boot pool: mirror vdev of 2x 64GB TOSHIBA-TR200 SSD via

MCP-220-81506-0N

MCP-220-81504-0N

We've all heard the 3-2-1 rule, right? Well, I don't follow it exactly, but I do well enough (I break the "2" rule). Here is my backup strategy at the time of writing.

I run two servers, Dorothy and Baret. Dorothy is my production server; Baret is my backup server.
Dorothy is configured to take regular snapshots and send them to Baret over a dedicated 10Gbit/s connection via zfs replication.
I never upgrade their FreeNAS version simultaneously. Baret always goes first, then Dorothy once I see that Baret is stable.
Baret has 2 pools that receive backups, both encrypted because . . .
I keep spare copies of both of Baret's pools off-site and rotate them with Baret on a regular basis. Worst-case my house burns down and I still have off-site copies, though admittedly a little out of date.
Any time I add irreplaceable data to Dorothy I rotate Baret's disks once replication completes so my off-sites always contain my most precious data.

In the Spring of 2020 Baret had an unscheduled reboot, locking both pools until I logged on. I planned on doing some experimenting anyway with some spare disks I have. My experimenting involved exporting and deleting pools. When I was all set I went to unlock Baret's pools and log off. The pools would not unlock. I tried my backup keys. No luck. I went through all of the encryption issue troubleshooting steps; no luck. I checked the partitions; they were gone. Then I discovered the NAS-105782 bug report. At that point I was sure that 20TB of data had been erased from Baret's backup pools.

But, I didn't lose any data. I wasn't even worried about losing data.

I knew I had full copies of all of my data on Dorothy and Dorothy was completely unaffected; a huge benefit of having a separate, dedicated backup server.
Even if Dorothy went down in flames before I brought Baret back up I had my off-site backups. Worst-case they are a little out of date.
Even if I had done my experiments on Dorothy as well, I would not have lost data there because Dorothy was on an unaffected version.

I know not everyone can implement a backup strategy like the above; but if you can, you should consider it. FreeNAS and ZFS do all they can to keep your data safe. But, bugs happen. And sometimes really insidious bugs happen. My backup strategy saved my data.

Important Announcement for The TrueNAS Community.

Encryption Debugging Tips

PhiloEpisteme

Guru

Similar threads