invalid start byte decrypting pool into clean install (solved)

[Dylan Tully]

Solved! winnielinnie pointed out that I had been using a geli.key to try and unlock an encrypted zfs dataset when I should have been using a .json file. Found the .json and stuff seems to be working.

Error short text: 'utf-8' codec can't decode byte 0xb6 in position 1: invalid start byte
TrueNAS version: TrueNAS 12.0-U8
CPU: AMD Athlon(tm) 5350 APU with Radeon(tm) R3
RAM: 7.4gb
Mobo: AM1B-ITX
PCI/STA interface: https://www.amazon.com/gp/product/B09DYVX5VJ
Drives involved: 2 PNY SSDs (mirrored boot), 2 Western Digital Red (mirrored pool), 3 Hitachi Drives (striped pool)

Show : dmesg result

root@truenas[~]# dmesg
---<<BOOT>>---
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-RELEASE-p12 ec84e0c52a1(HEAD) TRUENAS amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
VT(efifb): resolution 1280x1024
CPU: AMD Athlon(tm) 5350 APU with Radeon(tm) R3 (2050.04-MHz K8-class CPU)
Origin="AuthenticAMD" Id=0x700f01 Family=0x16 Model=0x0 Stepping=1
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C>
AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
AMD Features2=0x154037ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,Topology,PNXC,DBE,PL2I>
Structured Extended Features=0x8<BMI1>
XSAVE Features=0x1<XSAVEOPT>
SVM: NP,NRIP,AFlush,DAssist,NAsids=8
TSC: P-state invariant, performance statistics
real memory = 8589934592 (8192 MB)
avail memory = 7570104320 (7219 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
Firmware Warning (ACPI): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20200430/tbfadt-796)
ioapic0 <Version 2.1> irqs 0-23 on motherboard
ioapic1 <Version 2.1> irqs 24-55 on motherboard
Launching APs: 1 3 2
Timecounter "TSC" frequency 2050040284 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
nexus0
efirtc0: <EFI Realtime Clock> on motherboard
efirtc0: registered as a time-of-day clock, resolution 1.000000s
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard
padlock0: No ACE support.
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf0ff mem 0xc0000000-0xcfffffff,0xd0000000-0xd07fffff,0xffb00000-0xffb3ffff irq 44 at device 1.0 on pci0
vgapci0: Boot video device
pci0: <multimedia, HDA> at device 1.1 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 24 at device 2.1 on pci0
pci1: <ACPI PCI bus> on pcib1
ahci0: <Marvell 88SE9215 AHCI SATA controller> port 0xe050-0xe057,0xe040-0xe043,0xe030-0xe037,0xe020-0xe023,0xe000-0xe01f mem 0xffa40000-0xffa407ff irq 24 at device 0.0 on pci1
ahci0: AHCI v1.00 with 4 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
pcib2: <ACPI PCI-PCI bridge> irq 25 at device 2.2 on pci0
pci2: <ACPI PCI bus> on pcib2
ahci1: <ASMedia ASM1062 AHCI SATA controller> port 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f mem 0xff900000-0xff9001ff irq 28 at device 0.0 on pci2
ahci1: AHCI v1.20 with 2 6Gbps ports, Port Multiplier supported
ahci1: quirks=0xc00000<NOCCS,NOAUX>
ahcich4: <AHCI channel> at channel 0 on ahci1
ahcich5: <AHCI channel> at channel 1 on ahci1
pcib3: <ACPI PCI-PCI bridge> irq 26 at device 2.3 on pci0
pci3: <ACPI PCI bus> on pcib3
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xc000-0xc0ff mem 0xff800000-0xff800fff,0xd0800000-0xd0803fff irq 32 at device 0.0 on pci3
re0: Using 1 MSI-X message
re0: ASPM disabled
re0: Chip rev. 0x4c000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0
rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: d0:50:99:9e:ae:56
pcib4: <ACPI PCI-PCI bridge> irq 27 at device 2.4 on pci0
pci4: <ACPI PCI bus> on pcib4
xhci0: <ASMedia ASM1042A USB 3.0 controller> mem 0xff700000-0xff707fff irq 36 at device 0.0 on pci4
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Unable to map MSI-X table
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
xhci1: <AMD FCH USB 3.0 controller> mem 0xffb68000-0xffb69fff irq 18 at device 16.0 on pci0
xhci1: 32 bytes context size, 64-bit DMA
xhci1: Unable to map MSI-X table
usbus1 on xhci1
usbus1: 5.0Gbps Super Speed USB v3.0
ahci2: <AMD Hudson-2 AHCI SATA controller> port 0xf140-0xf147,0xf130-0xf133,0xf120-0xf127,0xf110-0xf113,0xf100-0xf10f mem 0xffb6e000-0xffb6e3ff irq 19 at device 17.0 on pci0
ahci2: AHCI v1.30 with 2 6Gbps ports, Port Multiplier supported
ahcich6: <AHCI channel> at channel 0 on ahci2
ahcich7: <AHCI channel> at channel 1 on ahci2
ohci0: <AMD FCH USB Controller> mem 0xffb6d000-0xffb6dfff irq 18 at device 18.0on pci0
usbus2 on ohci0
usbus2: 12Mbps Full Speed USB v1.0
ehci0: <AMD FCH USB 2.0 controller> mem 0xffb6c000-0xffb6c0ff irq 17 at device 18.2 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
usbus3: 480Mbps High Speed USB v2.0
ohci1: <AMD FCH USB Controller> mem 0xffb6b000-0xffb6bfff irq 18 at device 19.0on pci0
usbus4 on ohci1
usbus4: 12Mbps Full Speed USB v1.0
ehci1: <AMD FCH USB 2.0 controller> mem 0xffb6a000-0xffb6a0ff irq 17 at device 19.2 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci1
usbus5: 480Mbps High Speed USB v2.0
pci0: <multimedia, HDA> at device 20.2 (no driver attached)
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb5
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
orm0: <ISA Option ROMs> at iomem 0xcf000-0xd1fff,0xd2000-0xd9fff pnpid ORM0000 on isa0
amdsbwd0: <AMD SB8xx/SB9xx/Axx Watchdog Timer> at iomem 0xfec000f0-0xfec000f3,0xfec000f4-0xfec000f7 on isa0
superio0: <Nuvoton NCT6776> at port 0x2e-0x2f on isa0
wbwd0: <Nuvoton NCT6776 (0xc3/0x33) Watchdog Timer> at WDT ldn 0x08 on superio0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen3.1: <AMD EHCI root HUB> at usbus3
ugen5.1: <AMD EHCI root HUB> at usbus5
uhub0: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen0.1: <0x1b21 XHCI root HUB> at usbus0
uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ugen4.1: <AMD OHCI root HUB> at usbus4
ugen1.1: <0x1022 XHCI root HUB> at usbus1
uhub2: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub3: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
ugen2.1: <AMD OHCI root HUB> at usbus2
uhub4: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub5: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
Trying to mount root from zfs:boot-pool/ROOT/default []...
Root mount waiting for: CAM usbus0 usbus1 usbus2 usbus3 usbus4 usbus5
pmp0 at ahcich3 bus 0 scbus3 target 15 lun 0
pmp0: <Port Multiplier 5755197b 000e> ATA device
pmp0: 600.000MB/s transfers (SATA 3.x, NONE, PIO 8192bytes)
pmp0: 5 fan-out ports
uhub4: 4 ports with 4 removable, self powered
uhub5: 4 ports with 4 removable, self powered
uhub3: 4 ports with 4 removable, self powered
uhub2: 4 ports with 4 removable, self powered
ugen1.2: <Logitech USB Receiver> at usbus1
ukbd0 on uhub3
ukbd0: <Logitech USB Receiver, class 0/0, rev 2.00/12.03, addr 1> on usbus1
kbd2 at ukbd0
ada0 at ahcich3 bus 0 scbus3 target 0 lun 0
ada0: <Hitachi HUA722020ALA331 JKAOA3NH> ATA8-ACS SATA 2.x device
ada0: Serial Number BEG71AHW
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors)
ada1 at ahcich3 bus 0 scbus3 target 1 lun 0
ada1: <Hitachi HUA722020ALA331 JKAOA3NH> ATA8-ACS SATA 2.x device
ada1: Serial Number BFHAHSDT
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors)
ada2 at ahcich3 bus 0 scbus3 target 2 lun 0
ada2: <Hitachi HUA722020ALA331 JKAOA3NH> ATA8-ACS SATA 2.x device
ada2: Serial Number B8G3AW0V
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors)
ada3 at ahcich3 bus 0 scbus3 target 3 lun 0
ada3: <WDC WD60EFRX-68L0BN1 82.00A82> ACS-2 ATA SATA 3.x device
ada3: Serial Number WD-WX72D507JU6E
ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 5723166MB (11721045168 512 byte sectors)
ada3: quirks=0x1<4K>
ada4 at ahcich3 bus 0 scbus3 target 4 lun 0
ada4: <WDC WD60EFRX-68L0BN1 82.00A82> ACS-2 ATA SATA 3.x device
ada4: Serial Number WD-WX12D50LHCN9
ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 5723166MB (11721045168 512 byte sectors)
ada4: quirks=0x1<4K>
ada5 at ahcich4 bus 0 scbus4 target 0 lun 0
ada5: <PNY CS900 120GB SSD CS900615> ACS-4 ATA SATA 3.x device
ada5: Serial Number PNY21442111050103899
ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 114473MB (234441648 512 byte sectors)
ada6 at ahcich6 bus 0 scbus6 target 0 lun 0
ada6: <PNY CS900 120GB SSD CS900615> ACS-4 ATA SATA 3.x device
ada6: Serial Number PNY214421110501038A1
ada6: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 114473MB (234441648 512 byte sectors)
ums0 on uhub3
ums0: <Logitech USB Receiver, class 0/0, rev 2.00/12.03, addr 1> on usbus1
ums0: 16 buttons and [XYZT] coordinates ID=2
uhid0 on uhub3
uhid0: <Logitech USB Receiver, class 0/0, rev 2.00/12.03, addr 1> on usbus1
Root mount waiting for: usbus1 usbus3 usbus5
uhub1: 4 ports with 4 removable, self powered
uhub0: 4 ports with 4 removable, self powered
ugen1.3: <CPS CP1500PFCLCD> at usbus1
intsmb0: <AMD FCH SMBus Controller> at device 20.0 on pci0
smbus0: <System Management Bus> on intsmb0
interface uhid.1 already present in the KLD 'kernel'!
linker_load_file: /boot/kernel/uhid.ko - unsupported file type
interface wmt.1 already present in the KLD 'kernel'!
linker_load_file: /boot/kernel/wmt.ko - unsupported file type
lo0: link state changed to UP
re0: link state changed to UP
GEOM_MIRROR: Device mirror/swap0 launched (2/2).
GEOM_MIRROR: Device mirror/swap1 launched (2/2).
GEOM_ELI: Device mirror/swap0.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
GEOM_ELI: Device mirror/swap1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
GEOM_MIRROR: Device mirror/swap2 launched (2/2).
GEOM_ELI: Device mirror/swap2.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> K8/16/48/0x1ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA>
re0: link state changed to DOWN
re0: link state changed to UP
CPU: AMD Athlon(tm) 5350 APU with Radeon(tm) R3 (2050.04-MHz K8-class CPU)
Origin="AuthenticAMD" Id=0x700f01 Family=0x16 Model=0x0 Stepping=1
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x3ed8220b<SSE3,PCLMULQDQ,MON,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C>
AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
AMD Features2=0x154037ff<LAHF,CMP,SVM,ExtAP

I have a 3 disk RAIDZ1 array with a single unencrypted pool on it and a 2 disk mirrored array with a single unencrypted pool. It's fairly old so the pools each use the older style .geli encryption from FreeNAS 8.x days have encrypted datasets using native zfs encryption from 9.x. I had a USB thumb drive that was my boot drive and then (predictably) that crapped out on me after a little while and I was no longer able to boot the server. I have recently reinstalled TrueNAS onto 2 mirrored 120gb SSDs per the current recommendations to hopefully avoid this happening again. The new install is able to see my pools (Storage/Pools/Add/Import Existing Pool/No, Continue with the import) but when I try and unlock the datasets using the previously downloaded .geli key geli.key files (no I don't know why I named it something so confusing), I get the "'utf-8' codec can't decode byte 0xb6 in position 1: invalid start byte" error.

Show : full error text

Error: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 367, in run
await self.future
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 405, in __run_body
rv = await self.middleware.run_in_thread(self.method, *([self] + args))
File "/usr/local/lib/python3.9/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 2509, in encryption_summary
keys_supplied = self._retrieve_keys_from_file(job)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 2726, in _retrieve_keys_from_file
data = json.loads(job.pipes.input.r.read(10240))
File "/usr/local/lib/python3.9/json/__init__.py", line 341, in loads
s = s.decode(detect_encoding(s), 'surrogatepass')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb6 in position 1: invalid start byte

A series of screengrabs of the import/decrypt attempt are in the post referenced below.

From start to finish of trying to add the pool and decrypt the dataset.

I think after this I am going to move to unencrypted datasets. I think I have previously had difficulty decrypting, it was solved by using a different TrueNAS version. I know people occasionally have problems with PCI/SATA interfaces but I have had this one working in the system before and found it on these forums as a card known to work. The mirrored boot drives are plugged into the mobo, the 5 drives between the two pools are plugged into the PCI/SATA card. I have additionally tried to decrypt the drives with 1 boot drive and the 3 drive pool plugged directly into the mobo (only 4 SATA ports on mobo itself) and got the same problem. Anyone have any ideas what to try next?

Note: made some edits to clarify when I said 'pool' when I should have said 'dataset'
I had originally said this was geli encryption from 8.x, winnielinnie has helped determine that this is zfs encryption from 9.x.

 

[winnielinnie]

The new install is able to see my pools (Storage/Pools/Add/Import Existing Pool/No, Continue with the import)

Doesn't that mean you're skipping the step to decrypt the underlying GELI devices before importing the pool?

 

[Dylan Tully]

Doesn't that mean you're skipping the step to decrypt the underlying GELI devices before importing the pool?

Hopefully I didn't say it incorrectly, but my understanding of the situation is that the zfs pools are encrypted, not the disk/disks. So I can now see them under Storage/Pools, but I get the error when I click the three dots and click unlock.

 

[winnielinnie]

It's fairly old so it is using the older style geli encryption from FreeNAS 8.x days.

I'm a bit confused now. I thought you were trying to re-import an old pool from when you used GELI encryption for the underlying drives (which means there is no native ZFS encryption involved.)

 

[Dylan Tully]

Maybe I am mistaken about the current state of things so please bear with me and I'll clarify in the original post. I disconnected the two pools, went back to Add Pool and clicked Yes, decrypt the disks, no disks or pools pop up in the drop down menu. When I click No, don't decrypt the disks, I see both pools and can add them. Maybe the word I am using pool and dataset interchangeably in a way that is incorrect. After I add the pool, I get a message that the pool includes encrypted datasets, I try to give it the .geli key and it fails with the given error. The datasets are then in my storage tab, still encrypted.

 

[winnielinnie]

This is a bit all over the place, and unnerving.

With TrueNAS Core 12+ it's "either-or" in regards to GELI vs native ZFS encryption.

With FreeNAS 11 and earlier, you could only use GELI (not native encryption). Once you create the underlying GELI devices, they can then be supplied into a vdev during a new pool creation. This is a done deal. From now on, you must unlock the GELI devices first, which will then make them available (as vdev) for the pool to be imported.

With TrueNAS Core 12+, you can only use native ZFS encryption when creating new pools. No underlying devices are encrypted. It's a per-dataset native ZFS encryption. However, you can still (for now) import old pools in which the vdev(s) is unavailable until you unlock the underlying GELI devices. (This is the optional prompt it asks you during the import process.)

There is no way around this. I'm not even sure how you supposedly imported your old (GELI) pool without first unlocking the GELI devices during the pre-import stage.

Maybe the word I am using pool and dataset interchangeably in a way that is incorrect. After I add the pool, I get a message that the pool includes encrypted datasets, I try to give it the .geli key and it fails with the given error. The datasets are then in my storage tab, still encrypted.

What you're describing is importing a pool that is using native ZFS encryption. It has nothing to do with GELI, and hence whatever .geli key you have is meaningless. Which is worrying because what pools are we talking about anymore? Did you ever back up the keyfile for your natively encrypted ZFS dataset(s)? To have such native ZFS encryption means you've used TrueNAS Core 12 to create the pool: not FreeNAS 11 or earlier.

 

[Dylan Tully]

... I did a dumb with my file naming and called them geli.key and geli(1).key and then I got mixed up. I am sorry, I have been saying .geli keys but they are in fact .key keys. So I have those. And I have been regularly using those for years to unlock my datasets and perform export/imports for years now, so I know they are the right keys.

Ok. So it seems like we are narrowing this down to a problem of decrypting the ZFS data sets, not the pools. Now what? I could grab and screen shots privately if that would help better narrow down what's going on.

 

[winnielinnie]

So I have those. And I have been regularly using those for years to unlock my datasets and perform export/imports for years now, so I know they are the right keys.

Ok. So it seems like we are narrowing this down to a problem of decrypting the ZFS data sets, not the pools. Now what? I could grab and screen shots privately if that would help better narrow down what's going on.

I still think you're dealing with GELI. I don't understand how you could have created native ZFS encrypted datasets from when you built these pools under FreeNAS 8.

Screenshots and less ambiguous descriptions would help immensely. I'm not trying to sound rude. Just makes it more clear for yourself and others if you don't conflate things (even if by accident.) I get it. ZFS and TrueNAS and FreeBSD have some overlapping and confusing terminologies.

 

[Dylan Tully]

I still think you're dealing with GELI. I don't understand how you could have created native ZFS encrypted datasets from when you built these pools under FreeNAS 8.

Just went back and looked at when FreeNAS versions happened and I think these pools were made after FreeNAS 9.x was available, so we could have native ZFS encrypted datasets. Sorry for the confusion/thank you for the patience.

From start to finish of trying to add the pool and decrypt the dataset.

Nothing at the start

Import existing pool

No, continue with import

Pools show up here and they don't if I click Yes, decrypt the disks in the above image

great, import pool:

Get a notification about encrypted dataset:

Try to give it the appropriate .key file

and error

But everything shows up (still locked) in my Storage/pools tab:

 

[winnielinnie]

Now I'm even more confused.

Native ZFS encryption wasn't available with FreeNAS 11 or earlier; only TrueNAS Core 12 and newer.

With FreeNAS 11 and earlier, you could only use GELI (not native encryption).

Just went back and looked at when FreeNAS versions happened and I think these pools were made after FreeNAS 9.x was available, so we could have native ZFS encrypted datasets.

What do the following show?

To view currently imported pools:
zpool list -v

To view ZFS dataset native encryption information:
zfs list -o name,encryption,keylocation,keyformat

To see current GELI devices:
geli list -a

To view device and partition information:
gpart list -a

You can censor out private stuff. The last two commands will output a lot of text.

Your screenshot supposedly shows a pool with native encryption enabled. But you're only encrypting the top-level root dataset, while most child datasets are non-encrypted. (Some children are inheriting the root dataset's encryption.)

Have you been "upgrading" the pools as you went along through FreeNAS/TrueNAS versions?

 

[Dylan Tully]

Thank you for asking for the exact CLI inputs you need to see. With our comparative levels of expertise here, that's probably going to get you the answers quicker than me trying to figure out what I have wrought.

Have you been "upgrading" the pools as you went along through FreeNAS/TrueNAS versions?

I remember taking this dataset, transferring it all to another pool and then transferring it back at some point since the FreeNAS to TrueNAS switch. Which was probably to update the pool from it's older format to a newer format.

Your screenshot supposedly shows a pool with native encryption enabled. But you're only encrypting the top-level root dataset, while most child datasets are non-encrypted. (Some children are inheriting the root dataset's encryption.)

I only intentionally encrypted the root level data set, some of the children inherited that, some of the children (looks like the ones created after transitioning from FreeNAS to TrueNAS) did not inherit the encryption.

Show : zpool list -v

root@truenas[~]# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
********* 5.44T 3.35T 2.08T - - 0% 61% 1.00x ONLINE /mnt
raidz1 5.44T 3.35T 2.08T - - 0% 61.7% - ONLINE
gptid/58044ac3-b8eb-11eb-82a4-d050999eae56 - - - - - - - - ONLINE
gptid/58647bd1-b8eb-11eb-82a4-d050999eae56 - - - - - - - - ONLINE
gptid/5875924c-b8eb-11eb-82a4-d050999eae56 - - - - - - - - ONLINE
boot-pool 95.5G 1.21G 94.3G - - 0% 1% 1.00x ONLINE

Show : zfs list -o name,encryprion,keylocation,keyformat

root@truenas[~]# zfs list -o name,encryption,keylocation,keyformat
NAME ENCRYPTION KEYLOCATION KEYFORMAT
********* aes-256-gcm prompt hex
*********/********* off none none
*********/********* off none none
*********/********* off none none
*********/********* aes-256-gcm none hex
*********/********* off none none
*********/********* off none none
*********/********* aes-256-gcm none hex
boot-pool off none none
boot-pool/.system off none none
boot-pool/.system/configs-342111fac902458e8468ba3ccee40d16 off none none
boot-pool/.system/cores off none none
boot-pool/.system/rrd-342111fac902458e8468ba3ccee40d16 off none none
boot-pool/.system/samba4 off none none
boot-pool/.system/services off none none
boot-pool/.system/syslog-342111fac902458e8468ba3ccee40d16 off none none
boot-pool/.system/webui off none none

Show : geli list -a

none
root@truenas[~]#
root@truenas[~]#
root@truenas[~]# geli list -a
Geom name: mirror/swap0.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: hardware
Version: 7
Flags: ONETIME
KeysAllocated: 32
KeysTotal: 32
Providers:
1. Name: mirror/swap0.eli
Mediasize: 17179869184 (16G)
Sectorsize: 512
Mode: r1w1e0
Consumers:
1. Name: mirror/swap0
Mediasize: 17179869184 (16G)
Sectorsize: 512
Mode: r1w1e1

Geom name: mirror/swap1.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: hardware
Version: 7
Flags: ONETIME
KeysAllocated: 4
KeysTotal: 4
Providers:
1. Name: mirror/swap1.eli
Mediasize: 2147483648 (2.0G)
Sectorsize: 51

Show : gpart list -a

root@truenas[~]# gpart list -a
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 65536
Mode: r0w0e0
efimedia: HD(1,GPT,58483b70-b8eb-11eb-82a4-d050999eae56,0x80,0x400000)
rawuuid: 58483b70-b8eb-11eb-82a4-d050999eae56
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada0p2
Mediasize: 1998251364352 (1.8T)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 2147549184
Mode: r1w1e2
efimedia: HD(2,GPT,5875924c-b8eb-11eb-82a4-d050999eae56,0x400080,0xe8a08808)
rawuuid: 5875924c-b8eb-11eb-82a4-d050999eae56
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 1998251364352
offset: 2147549184
type: freebsd-zfs
index: 2
end: 3907029127
start: 4194432
Consumers:
1. Name: ada0
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e3

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 65536
Mode: r1w1e1
efimedia: HD(1,GPT,57ccc645-b8eb-11eb-82a4-d050999eae56,0x80,0x400000)
rawuuid: 57ccc645-b8eb-11eb-82a4-d050999eae56
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada1p2
Mediasize: 1998251364352 (1.8T)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 2147549184
Mode: r1w1e2
efimedia: HD(2,GPT,58044ac3-b8eb-11eb-82a4-d050999eae56,0x400080,0xe8a08808)
rawuuid: 58044ac3-b8eb-11eb-82a4-d050999eae56
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 1998251364352
offset: 2147549184
type: freebsd-zfs
index: 2
end: 3907029127
start: 4194432
Consumers:
1. Name: ada1
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r2w2e5

Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 65536
Mode: r1w1e1
efimedia: HD(1,GPT,5832005b-b8eb-11eb-82a4-d050999eae56,0x80,0x400000)
rawuuid: 5832005b-b8eb-11eb-82a4-d050999eae56
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada2p2
Mediasize: 1998251364352 (1.8T)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 2147549184
Mode: r1w1e2
efimedia: HD(2,GPT,58647bd1-b8eb-11eb-82a4-d050999eae56,0x400080,0xe8a08808)
rawuuid: 58647bd1-b8eb-11eb-82a4-d050999eae56
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 1998251364352
offset: 2147549184
type: freebsd-zfs
index: 2
end: 3907029127
start: 4194432
Consumers:
1. Name: ada2
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r2w2e5

Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 11721045127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
efimedia: HD(1,GPT,cb9e6963-84ff-11eb-8909-d050999eae56,0x80,0x400000)
rawuuid: cb9e6963-84ff-11eb-8909-d050999eae56
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada3p2
Mediasize: 5999027556352 (5.5T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
efimedia: HD(2,GPT,cbbbe054-84ff-11eb-8909-d050999eae56,0x400080,0x2ba60f408)
rawuuid: cbbbe054-84ff-11eb-8909-d050999eae56
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 5999027556352
offset: 2147549184
type: freebsd-zfs
index: 2
end: 11721045127
start: 4194432
Consumers:
1. Name: ada3
Mediasize: 6001175126016 (5.5T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0

Geom name: ada4
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 11721045127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada4p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
efimedia: HD(1,GPT,cb5d2af5-84ff-11eb-8909-d050999eae56,0x80,0x400000)
rawuuid: cb5d2af5-84ff-11eb-8909-d050999eae56
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 2147483648
offset: 65536
type: freebsd-swap
index: 1
end: 4194431
start: 128
2. Name: ada4p2
Mediasize: 5999027556352 (5.5T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0
efimedia: HD(2,GPT,cb88002b-84ff-11eb-8909-d050999eae56,0x400080,0x2ba60f408)
rawuuid: cb88002b-84ff-11eb-8909-d050999eae56
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 5999027556352
offset: 2147549184
type: freebsd-zfs
index: 2
end: 11721045127
start: 4194432
Consumers:
1. Name: ada4
Mediasize: 6001175126016 (5.5T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r0w0e0

Geom name: ada5
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 234441607
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada5p1
Mediasize: 272629760 (260M)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 20480
Mode: r0w0e0
efimedia: HD(1,GPT,37e2d695-8cd3-11ec-8727-d050999eae56,0x28,0x82000)
rawuuid: 37e2d695-8cd3-11ec-8727-d050999eae56
rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
label: (null)
length: 272629760
offset: 20480
type: efi
index: 1
end: 532519
start: 40
2. Name: ada5p2
Mediasize: 102575898624 (96G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 272650240
Mode: r1w1e1
efimedia: HD(2,GPT,37ed5de6-8cd3-11ec-8727-d050999eae56,0x2082028,0xbf10000)
rawuuid: 37ed5de6-8cd3-11ec-8727-d050999eae56
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 102575898624
offset: 17452519424
type: freebsd-zfs
index: 2
end: 234430503
start: 34086952
3. Name: ada5p3
Mediasize: 17179869184 (16G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 272650240
Mode: r1w1e1
efimedia: HD(3,GPT,37e8b305-8cd3-11ec-8727-d050999eae56,0x82028,0x2000000)
rawuuid: 37e8b305-8cd3-11ec-8727-d050999eae56
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 17179869184
offset: 272650240
type: freebsd-swap
index: 3
end: 34086951
start: 532520
Consumers:
1. Name: ada5
Mediasize: 120034123776 (112G)
Sectorsize: 512
Mode: r2w2e4

Geom name: ada6
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 234441607
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada6p1
Mediasize: 272629760 (260M)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 20480
Mode: r0w0e0
efimedia: HD(1,GPT,37f401e5-8cd3-11ec-8727-d050999eae56,0x28,0x82000)
rawuuid: 37f401e5-8cd3-11ec-8727-d050999eae56
rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
label: (null)
length: 272629760
offset: 20480
type: efi
index: 1
end: 532519
start: 40
2. Name: ada6p2
Mediasize: 102575898624 (96G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 272650240
Mode: r1w1e1
efimedia: HD(2,GPT,37fefc0f-8cd3-11ec-8727-d050999eae56,0x2082028,0xbf10000)
rawuuid: 37fefc0f-8cd3-11ec-8727-d050999eae56
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 102575898624
offset: 17452519424
type: freebsd-zfs
index: 2
end: 234430503
start: 34086952
3. Name: ada6p3
Mediasize: 17179869184 (16G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 272650240
Mode: r1w1e1
efimedia: HD(3,GPT,37fa14bf-8cd3-11ec-8727-d050999eae56,0x82028,0x2000000)
rawuuid: 37fa14bf-8cd3-11ec-8727-d050999eae56
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: (null)
length: 17179869184
offset: 272650240
type: freebsd-swap
index: 3
end: 34086951

 

[winnielinnie]

I remember taking this dataset, transferring it all to another pool and then transferring it back at some point since the FreeNAS to TrueNAS switch. Which was probably to update the pool from it's older format to a newer format.

Can you describe this in more detail? Was a new pool created in the process, in which you replicated/transferred datasets to it? From what you're saying, the original pool (from the FreeNAS 9 days) no longer exists?

Your output shows no use of GELI (other than swap, which is always encrypted with GELI devices in FreeNAS/TrueNAS.)

Some different possibilities that come to mind:

• Something happened with the original-to-new migration, and you've mistaken one keyfile for the other? The fact that your keyfile is named "geli" hints that maybe you mixed them up at one point , and you're literally trying to use an old and incompatible GELI keyfile, when in fact you need to be using the new keyfile (normally exported as a .json file) to unlock the root dataset. Do you remember ever exporting a .json keyfile after creating the new pool with native ZFS encryption?
• You can try again with an older version of TrueNAS (-U7?, -U5.1?)
• There's a bug in the middleware in which it cannot read your keyfile for some reason.
• You can try to access the pool by using a live Linux ISO (i.e, Manjaro), and installing the latest ZFS package. (This still requires the relevant keyfile.)

To rule something out, you can open up the "keyfile" in a text editor. You'll immediately know if it's meant for GELI or for native ZFS encryption by its format. The type for ZFS native encryption will have a "table-like" form with the name of relevant datasets (even only the top-level root dataset as the sole entry), visually similar to a simple XML file. GELI keyfiles make no such reference to anything ZFS-related, let alone dataset names.

 

[Dylan Tully]

Do you remember ever exporting a .json keyfile after creating the new pool with native ZFS encryption?

WELL GAG ME WITH A SPOON MRS. HENDERSON, IT WORKED!

I went and found the .json keys and then that unlocked the root data set. I need to mess with my network shares before I can confirm I have everything.

This particular situation when trying to use a geli.key instead of a .json key when trying to unlock a zfs encrypted dataset should probably throw a different error than "invalid start byte" which doesn't really tell me anything about what is going wrong.

 

[winnielinnie]

Glad you're back in business! Make sure to safeguard and make multiple copies of this .json keyfile and/or copy the 64-character HEX string somewhere. Without it, you lose your data for good.

I never understood why TrueNAS / FreeNAS has always had these cryptic python errors that are barfed at the end-user, when a simple explanation would save so much time and spare confusion. It should be: simple explanation, with the option to expand into more details.



Imagine trying to login to a website or your email, only to be hit with something like:

"FAILURE! Hash mismatch fault due to XYZ algorithm blabitty blah 0x03842! Here are some random library files that mean noting to you, unless you happen to be a software developer..."


I think the end-user would prefer it to read: "Invalid password. Try again."



I had considered filing a "feature ticket" in the Jira, and this might be a good reason to do so. Cryptic error message dumps can lead a user astray on a wild goosechase, when in fact a simpler message would allow them to figure out the error immediately.

I'm sure if it read "Wrong keyfile" or "Invalid keyfile", it would have saved a lot of time.

The cryptic message makes you think "Is something wrong with my pool? With my system? With TrueNAS? Because of an upgrade? A migration?"

 

[Dylan Tully]

Even some prompting on the Uploading key page with "should be a .json file"
Or a table of "Geli encryption: .key, ZFS encryption: .json"
My first thought was that the problem had to do with the change from having the pool plugged into the SATA controller instead of the board and that some part of the directory was off or something.