After replace a drive [Unable to set key: geli: Cannot open ...]

Status
Not open for further replies.
wtfuar

wtfuar

Dabbler
Joined
Jun 25, 2013
Messages
36
Following all done by GUI - FreeBSD 8.3-RELEASE-p7 #1 r249203M: Sat Apr 6 09:28:27 PDT 2013

A drive had errors and after scrubbing - with no dataloss and successfull repair - I decided to replace the drive in question.
I followed the manual: 6.3.11

- zfs RaidZ1 was mounted using the recovery-key and in unknown state
- [Offline] the drive - state changed to degraded
- hot swaped it with the new one (same type and place)
- [Replace] button hit adn resilvering toke 12h and was without errors finished - state degraded
- [Detache] button hit on old drive - status healthy

Now as the drive has changed I needed to recreate the keys but:

Code:
Aug 24 08:44:37 freenas notifier: 1+0 records in
Aug 24 08:44:37 freenas notifier: 1+0 records out
Aug 24 08:44:37 freenas notifier: 64 bytes transferred in 0.000024 secs (2657777 bytes/sec)
Aug 24 08:44:44 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ]
Aug 24 08:45:07 freenas notifier: 1+0 records in
Aug 24 08:45:07 freenas notifier: 1+0 records out
Aug 24 08:45:07 freenas notifier: 64 bytes transferred in 0.000024 secs (2657777 bytes/sec)
Aug 24 08:45:07 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set recovery key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ]
Aug 24 08:47:18 freenas notifier: 1+0 records in
Aug 24 08:47:18 freenas notifier: 1+0 records out
Aug 24 08:47:18 freenas notifier: 64 bytes transferred in 0.000025 secs (2556528 bytes/sec)
Aug 24 08:47:18 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set recovery key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ]


I wont switch off the box until this problem is solved as I think I would not able to mount my zfspool again.

I read the thread cyberjock started and I'd like to here a more official work around or better a solution as the manual take this problem not in account.

It seems the [Detache] button does NOT remove the old drive from the config.

It seems there have to be an Information when starting the procedure of replacing drives on an encrypted zfs pool how to proceed - better in manual.

When replacing drives on an encrypted zfs pool it is necessary to add further more steps that require

- change of passphrase
- download new key
- encryption re-key

Following actual pool status:

Code:
[root@freenas] ~# zpool status
  pool: 000
state: ONLINE
  scan: resilvered 1.25T in 12h32m with 0 errors on Sat Aug 24 07:37:39 2013
config:

        NAME                                                STATE    READ WRITE CKSUM
        zfs1                                                ONLINE      0    0    0
          raidz1-0                                          ONLINE      0    0    0
            gptid/27c029db-0c16-11e3-b6f5-002215ab012f.eli  ONLINE      0    0    0
            gptid/622428ec-9d54-11e2-9312-000c29cec408.eli  ONLINE      0    0    0
            gptid/634a7ede-9d54-11e2-9312-000c29cec408.eli  ONLINE      0    0    0
            gptid/64712091-9d54-11e2-9312-000c29cec408.eli  ONLINE      0    0    0
            gptid/65977639-9d54-11e2-9312-000c29cec408.eli  ONLINE      0    0    0
            gptid/66bc5d54-9d54-11e2-9312-000c29cec408.eli  ONLINE      0    0    0

errors: No known data errors



Following the complete syslog:

Code:
Aug 23 11:06:22 freenas newsyslog[1402]: logfile first created
Aug 23 11:06:22 freenas syslogd: kernel boot file is /boot/kernel/kernel
Aug 23 11:06:22 freenas kernel: Copyright (c) 1992-2012 The FreeBSD Project.
Aug 23 11:06:22 freenas kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Aug 23 11:06:22 freenas kernel: The Regents of the University of California. All rights reserved.
Aug 23 11:06:22 freenas kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Aug 23 11:06:22 freenas kernel: FreeBSD 8.3-RELEASE-p7 #1 r249203M: Sat Apr  6 09:28:27 PDT 2013
Aug 23 11:06:22 freenas kernel: root@build.ixsystems.com:/tank/home/jpaetzel/fn8.3/freenas/os-base/amd64/tank/home/jpaetzel/fn8.3/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64
Aug 23 11:06:22 freenas kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Aug 23 11:06:22 freenas kernel: CPU: Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz (3292.54-MHz K8-class CPU)
Aug 23 11:06:22 freenas kernel: Origin = "GenuineIntel"  Id = 0x00000  Family = 6  Model = 2a  Stepping = 7
Aug 23 11:06:22 freenas kernel: Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Aug 23 11:06:22 freenas kernel: Features2=0x179ae3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT,TSCDLT,AESNI,XSAVE,AVX>
Aug 23 11:06:22 freenas kernel: AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
Aug 23 11:06:22 freenas kernel: AMD Features2=0x1<LAHF>
Aug 23 11:06:22 freenas kernel: TSC: P-state invariant
Aug 23 11:06:22 freenas kernel: real memory  = 17179869184 (16384 MB)
Aug 23 11:06:22 freenas kernel: avail memory = 16452546560 (15690 MB)
Aug 23 11:06:22 freenas kernel: ACPI APIC Table: <INTEL  DH67BL  >
Aug 23 11:06:22 freenas kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Aug 23 11:06:22 freenas kernel: FreeBSD/SMP: 1 package(s) x 4 core(s)
Aug 23 11:06:22 freenas kernel: cpu0 (BSP): APIC ID:  0
Aug 23 11:06:22 freenas kernel: cpu1 (AP): APIC ID:  2
Aug 23 11:06:22 freenas kernel: cpu2 (AP): APIC ID:  4
Aug 23 11:06:22 freenas kernel: cpu3 (AP): APIC ID:  6
Aug 23 11:06:22 freenas kernel: WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
Aug 23 11:06:22 freenas kernel: ioapic0 <Version 2.0> irqs 0-23 on motherboard
Aug 23 11:06:22 freenas kernel: kbd1 at kbdmux0
Aug 23 11:06:22 freenas kernel: hpt27xx: RocketRAID 27xx controller driver v1.0 (Apr  6 2013 09:28:15)
Aug 23 11:06:22 freenas kernel: cryptosoft0: <software crypto> on motherboard
Aug 23 11:06:22 freenas kernel: aesni0: <AES-CBC,AES-XTS> on motherboard
Aug 23 11:06:22 freenas kernel: acpi0: <INTEL DH67BL> on motherboard
Aug 23 11:06:22 freenas kernel: acpi0: [ITHREAD]
Aug 23 11:06:22 freenas kernel: acpi0: Power Button (fixed)
Aug 23 11:06:22 freenas kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
Aug 23 11:06:22 freenas kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
Aug 23 11:06:22 freenas kernel: cpu0: <ACPI CPU> on acpi0
Aug 23 11:06:22 freenas kernel: cpu1: <ACPI CPU> on acpi0
Aug 23 11:06:22 freenas kernel: cpu2: <ACPI CPU> on acpi0
Aug 23 11:06:22 freenas kernel: cpu3: <ACPI CPU> on acpi0
Aug 23 11:06:22 freenas kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Aug 23 11:06:22 freenas kernel: pci0: <ACPI PCI bus> on pcib0
Aug 23 11:06:22 freenas kernel: pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
Aug 23 11:06:22 freenas kernel: pci1: <ACPI PCI bus> on pcib1
Aug 23 11:06:22 freenas kernel: mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xfe5c0000-0xfe5c3fff,0xfe580000-0xfe5bffff irq 16 at device 0.0 on pci1
Aug 23 11:06:22 freenas kernel: mps0: Firmware: 16.00.00.00, Driver: 13.00.00.00-fbsd
Aug 23 11:06:22 freenas kernel: mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
Aug 23 11:06:22 freenas kernel: mps0: [ITHREAD]
Aug 23 11:06:22 freenas kernel: vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xfe000000-0xfe3fffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
Aug 23 11:06:22 freenas kernel: pci0: <simple comms> at device 22.0 (no driver attached)
Aug 23 11:06:22 freenas kernel: em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0x0000-0x0000 mem 0xfe600000-0x0000000,0x00000000-0x00000000 irq 20 at device 25.0 on pci0
Aug 23 11:06:22 freenas kernel: em0: Using an MSI interrupt
Aug 23 11:06:22 freenas kernel: em0: [FILTER]
Aug 23 11:06:22 freenas kernel: em0: Ethernet address: 00:00:00:00:00:00
Aug 23 11:06:22 freenas kernel: ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe623000-0xfe6233ff irq 16 at device 26.0 on pci0
Aug 23 11:06:22 freenas kernel: ehci0: [ITHREAD]
Aug 23 11:06:22 freenas kernel: usbus0: EHCI version 1.0
Aug 23 11:06:22 freenas kernel: usbus0: <EHCI (generic) USB 2.0 controller> on ehci0
Aug 23 11:06:22 freenas kernel: pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
Aug 23 11:06:22 freenas kernel: pci2: <ACPI PCI bus> on pcib2
Aug 23 11:06:22 freenas kernel: pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
Aug 23 11:06:22 freenas kernel: pci3: <ACPI PCI bus> on pcib3
Aug 23 11:06:22 freenas kernel: pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
Aug 23 11:06:22 freenas kernel: pci4: <ACPI PCI bus> on pcib4
Aug 23 11:06:22 freenas kernel: pci4: <serial bus, USB> at device 0.0 (no driver attached)
Aug 23 11:06:22 freenas kernel: ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe622000-0xfe6223ff irq 23 at device 29.0 on pci0
Aug 23 11:06:22 freenas kernel: ehci1: [ITHREAD]
Aug 23 11:06:22 freenas kernel: usbus1: EHCI version 1.0
Aug 23 11:06:22 freenas kernel: usbus1: <EHCI (generic) USB 2.0 controller> on ehci1
Aug 23 11:06:22 freenas kernel: isab0: <PCI-ISA bridge> at device 31.0 on pci0
Aug 23 11:06:22 freenas kernel: isa0: <ISA bus> on isab0
Aug 23 11:06:22 freenas kernel: ahci0: <Intel Cougar Point AHCI SATA controller> port 0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf060-0xf07f mem 0xfe621000-0xfe6217ff irq 19 at device 31.2 on pci0
Aug 23 11:06:22 freenas kernel: ahci0: [ITHREAD]
Aug 23 11:06:22 freenas kernel: ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
Aug 23 11:06:22 freenas kernel: ahcich0: <AHCI channel> at channel 0 on ahci0
Aug 23 11:06:22 freenas kernel: ahcich0: [ITHREAD]
Aug 23 11:06:22 freenas kernel: ahcich1: <AHCI channel> at channel 1 on ahci0
Aug 23 11:06:22 freenas kernel: ahcich1: [ITHREAD]
Aug 23 11:06:22 freenas kernel: ahcich2: <AHCI channel> at channel 2 on ahci0
Aug 23 11:06:22 freenas kernel: ahcich2: [ITHREAD]
Aug 23 11:06:22 freenas kernel: ahcich3: <AHCI channel> at channel 3 on ahci0
Aug 23 11:06:22 freenas kernel: ahcich3: [ITHREAD]
Aug 23 11:06:22 freenas kernel: pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
Aug 23 11:06:22 freenas kernel: acpi_button0: <Power Button> on acpi0
Aug 23 11:06:22 freenas kernel: acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Aug 23 11:06:22 freenas kernel: Timecounter "HPET" frequency 14318180 Hz quality 900
Aug 23 11:06:22 freenas kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Aug 23 11:06:22 freenas kernel: orm0: <ISA Option ROM> at iomem 0xce800-0xcefff on isa0
Aug 23 11:06:22 freenas kernel: sc0: <System console> at flags 0x100 on isa0
Aug 23 11:06:22 freenas kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Aug 23 11:06:22 freenas kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Aug 23 11:06:22 freenas kernel: atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
Aug 23 11:06:22 freenas kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Aug 23 11:06:22 freenas kernel: kbd0 at atkbd0
Aug 23 11:06:22 freenas kernel: atkbd0: [GIANT-LOCKED]
Aug 23 11:06:22 freenas kernel: atkbd0: [ITHREAD]
Aug 23 11:06:22 freenas kernel: ppc0: cannot reserve I/O port range
Aug 23 11:06:22 freenas kernel: coretemp0: <CPU On-Die Thermal Sensors> on cpu0
Aug 23 11:06:22 freenas kernel: est0: <Enhanced SpeedStep Frequency Control> on cpu0
Aug 23 11:06:22 freenas kernel: p4tcc0: <CPU Frequency Thermal Control> on cpu0
Aug 23 11:06:22 freenas kernel: coretemp1: <CPU On-Die Thermal Sensors> on cpu1
Aug 23 11:06:22 freenas kernel: est1: <Enhanced SpeedStep Frequency Control> on cpu1
Aug 23 11:06:22 freenas kernel: p4tcc1: <CPU Frequency Thermal Control> on cpu1
Aug 23 11:06:22 freenas kernel: coretemp2: <CPU On-Die Thermal Sensors> on cpu2
Aug 23 11:06:22 freenas kernel: est2: <Enhanced SpeedStep Frequency Control> on cpu2
Aug 23 11:06:22 freenas kernel: p4tcc2: <CPU Frequency Thermal Control> on cpu2
Aug 23 11:06:22 freenas kernel: coretemp3: <CPU On-Die Thermal Sensors> on cpu3
Aug 23 11:06:22 freenas kernel: est3: <Enhanced SpeedStep Frequency Control> on cpu3
Aug 23 11:06:22 freenas kernel: p4tcc3: <CPU Frequency Thermal Control> on cpu3
Aug 23 11:06:22 freenas kernel: fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8
Aug 23 11:06:22 freenas kernel: Timecounters tick every 1.000 msec
Aug 23 11:06:22 freenas kernel: hpt27xx: no controller detected.
Aug 23 11:06:22 freenas kernel: GEOM_RAID5: Module loaded, version 1.1.20110927.40 (rev 00ce00e5abb4)
Aug 23 11:06:22 freenas kernel: usbus0: 480Mbps High Speed USB v2.0
Aug 23 11:06:22 freenas kernel: usbus1: 480Mbps High Speed USB v2.0
Aug 23 11:06:22 freenas kernel: ugen0.1: <Intel> at usbus0
Aug 23 11:06:22 freenas kernel: uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
Aug 23 11:06:22 freenas kernel: ugen1.1: <Intel> at usbus1
Aug 23 11:06:22 freenas kernel: uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
Aug 23 11:06:22 freenas kernel: uhub0: 2 ports with 2 removable, self powered
Aug 23 11:06:22 freenas kernel: uhub1: 2 ports with 2 removable, self powered
Aug 23 11:06:22 freenas kernel: ugen1.2: <vendor 0x8087> at usbus1
Aug 23 11:06:22 freenas kernel: uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
Aug 23 11:06:22 freenas kernel: ugen0.2: <vendor 0x8087> at usbus0
Aug 23 11:06:22 freenas kernel: uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
Aug 23 11:06:22 freenas kernel: (probe14:mps0:0:14:0): TEST UNIT READY. CDB: 0 0 0 0 0 0
Aug 23 11:06:22 freenas kernel: (probe14:mps0:0:14:0): CAM status: SCSI Status Error
Aug 23 11:06:22 freenas kernel: (probe14:mps0:0:14:0): SCSI status: Check Condition
Aug 23 11:06:22 freenas kernel: (probe14:mps0:0:14:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Aug 23 11:06:22 freenas kernel: uhub3: 6 ports with 6 removable, self powered
Aug 23 11:06:22 freenas kernel: uhub2: 8 ports with 8 removable, self powered
Aug 23 11:06:22 freenas kernel: ugen0.3: <vendor 0x13fe> at usbus0
Aug 23 11:06:22 freenas kernel: umass0: <vendor 0x13fe product 0x3600, class 0/0, rev 2.00/1.00, addr 3> on usbus0
Aug 23 11:06:22 freenas kernel: da0 at mps0 bus 0 scbus0 target 21 lun 0
Aug 23 11:06:22 freenas kernel: da0: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device
Aug 23 11:06:22 freenas kernel: da0: 300.000MB/s transfers
Aug 23 11:06:22 freenas kernel: da0: Command Queueing enabled
Aug 23 11:06:22 freenas kernel: da0: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Aug 23 11:06:22 freenas kernel: da1 at mps0 bus 0 scbus0 target 22 lun 0
Aug 23 11:06:22 freenas kernel: da1: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device
Aug 23 11:06:22 freenas kernel: da1: 300.000MB/s transfers
Aug 23 11:06:22 freenas kernel: da1: Command Queueing enabled
Aug 23 11:06:22 freenas kernel: da1: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Aug 23 11:06:22 freenas kernel: da2 at mps0 bus 0 scbus0 target 23 lun 0
Aug 23 11:06:22 freenas kernel: da2: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device
Aug 23 11:06:22 freenas kernel: da2: 300.000MB/s transfers
Aug 23 11:06:22 freenas kernel: da2: Command Queueing enabled
Aug 23 11:06:22 freenas kernel: da2: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Aug 23 11:06:22 freenas kernel: da3 at mps0 bus 0 scbus0 target 24 lun 0
Aug 23 11:06:22 freenas kernel: da3: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device
Aug 23 11:06:22 freenas kernel: da3: 300.000MB/s transfers
Aug 23 11:06:22 freenas kernel: da3: Command Queueing enabled
Aug 23 11:06:22 freenas kernel: da3: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Aug 23 11:06:22 freenas kernel: da4 at mps0 bus 0 scbus0 target 25 lun 0
Aug 23 11:06:22 freenas kernel: da4: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device
Aug 23 11:06:22 freenas kernel: da4: 300.000MB/s transfers
Aug 23 11:06:22 freenas kernel: da4: Command Queueing enabled
Aug 23 11:06:22 freenas kernel: da4: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Aug 23 11:06:22 freenas kernel: da5 at mps0 bus 0 scbus0 target 26 lun 0
Aug 23 11:06:22 freenas kernel: da5: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device
Aug 23 11:06:22 freenas kernel: da5: 300.000MB/s transfers
Aug 23 11:06:22 freenas kernel: da5: Command Queueing enabled
Aug 23 11:06:22 freenas kernel: da5: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Aug 23 11:06:22 freenas kernel: ses0 at mps0 bus 0 scbus0 target 14 lun 0
Aug 23 11:06:22 freenas kernel: ses0: <ESG-SHV. SCA HSBP M1..... 2.14> Fixed Enclosure Services SCSI-3 device da6 at umass-sim0 bus 0 scbus5 target 0 lun 0
Aug 23 11:06:22 freenas kernel: da6: <  DL03> Removable Direct Access SCSI-0 device
Aug 23 11:06:22 freenas kernel: da6: 40.000MB/s transfers
Aug 23 11:06:22 freenas kernel: da6: 3825MB (7833600 512 byte sectors: 255H 63S/T 487C)
Aug 23 11:06:22 freenas kernel:
Aug 23 11:06:22 freenas kernel: ses0: 300.000MB/s transfers
Aug 23 11:06:22 freenas kernel: ses0: SCSI-3 SES Device
Aug 23 11:06:22 freenas kernel: SMP: AP CPU #1 Launched!
Aug 23 11:06:22 freenas kernel: SMP: AP CPU #2 Launched!
Aug 23 11:06:22 freenas kernel: SMP: AP CPU #3 Launched!
Aug 23 11:06:22 freenas kernel: GEOM: da6s1: geometry does not match label (16h,63s != 255h,63s).
Aug 23 11:06:22 freenas kernel: Trying to mount root from ufs:/dev/ufs/FreeNASs1a
Aug 23 11:06:22 freenas kernel: ZFS filesystem version 5
Aug 23 11:06:22 freenas kernel: ZFS storage pool version 28
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da0p1.eli created.
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256
Aug 23 11:06:22 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da1p1.eli created.
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256
Aug 23 11:06:22 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da2p1.eli created.
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256
Aug 23 11:06:22 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da3p1.eli created.
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256
Aug 23 11:06:22 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da4p1.eli created.
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256
Aug 23 11:06:22 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da5p1.eli created.
Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256
Aug 23 11:06:22 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:06:22 freenas root: /etc/rc: WARNING: Dump device does not exist.  Savecore not run.
Aug 23 11:06:22 freenas root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
Aug 23 11:06:23 freenas ntpd[1713]: ntpd 4.2.4p5-a (1)
Aug 23 11:06:32 freenas avahi-daemon[2117]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Aug 23 11:06:31 freenas ntpd[1714]: time reset -0.918430 s
Aug 23 11:07:01 freenas autosnap.py: [tools.autosnap:56] Popen()ing: /sbin/zfs snapshot 000/000@auto-20130823.1107-1w
Aug 23 11:07:01 freenas autosnap.py: [tools.autosnap:247] Failed to create snapshot '000/000@auto-20130823.1107-1w': cannot open '000/000': dataset does not exist
Aug 23 11:07:17 freenas manage.py: [storage.forms:1794] <InMemoryUploadedFile: geli_recovery.key (application/octet-stream)>
Aug 23 11:07:17 freenas kernel: GEOM_ELI: Device gptid/65977639-9d54-11e2-9312-000c29cec408.eli created.
Aug 23 11:07:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128
Aug 23 11:07:17 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:07:17 freenas kernel: GEOM_ELI: Device gptid/66bc5d54-9d54-11e2-9312-000c29cec408.eli created.
Aug 23 11:07:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128
Aug 23 11:07:17 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:07:17 freenas kernel: GEOM_ELI: Device gptid/6125e604-9d54-11e2-9312-000c29cec408.eli created.
Aug 23 11:07:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128
Aug 23 11:07:17 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:07:17 freenas kernel: GEOM_ELI: Device gptid/622428ec-9d54-11e2-9312-000c29cec408.eli created.
Aug 23 11:07:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128
Aug 23 11:07:17 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:07:17 freenas kernel: GEOM_ELI: Device gptid/634a7ede-9d54-11e2-9312-000c29cec408.eli created.
Aug 23 11:07:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128
Aug 23 11:07:17 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:07:18 freenas kernel: GEOM_ELI: Device gptid/64712091-9d54-11e2-9312-000c29cec408.eli created.
Aug 23 11:07:18 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128
Aug 23 11:07:18 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 11:07:40 freenas notifier: Stopping dbus.
Aug 23 11:07:40 freenas avahi-daemon[2117]: Disconnected from D-Bus, exiting.
Aug 23 11:07:40 freenas notifier: dbus not running? (check /var/run/dbus/dbus.pid).
Aug 23 11:07:40 freenas notifier: Starting dbus.
Aug 23 11:07:40 freenas notifier: Stopping avahi-daemon.
Aug 23 11:07:40 freenas notifier: Failed to kill daemon: No such file or directory
Aug 23 11:07:40 freenas notifier: Stopping avahi-daemon.
Aug 23 11:07:40 freenas notifier: Failed to kill daemon: No such file or directory
Aug 23 11:07:40 freenas notifier: Starting avahi-daemon.
Aug 23 11:07:40 freenas avahi-daemon[3029]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Aug 23 11:07:40 freenas notifier: winbindd not running? (check /var/run/samba/winbindd.pid).
Aug 23 11:07:40 freenas notifier: Stopping smbd.
Aug 23 11:07:40 freenas notifier: Waiting for PIDS: 1976.
Aug 23 11:07:40 freenas notifier: Stopping nmbd.
Aug 23 11:07:40 freenas notifier: Waiting for PIDS: 1973.
Aug 23 11:07:40 freenas notifier: Performing sanity check on Samba configuration: OK
Aug 23 11:07:40 freenas notifier: smbd not running? (check /var/run/samba/smbd.pid).
Aug 23 11:07:40 freenas notifier: nmbd not running? (check /var/run/samba/nmbd.pid).
Aug 23 11:07:40 freenas notifier: Removing stale Samba tdb files: ....... done
Aug 23 11:07:40 freenas notifier: Starting nmbd.
Aug 23 11:07:40 freenas notifier: Starting smbd.
Aug 23 12:54:53 freenas kernel: em0: link state changed to DOWN
Aug 23 12:56:07 freenas kernel: em0: link state changed to UP
Aug 23 18:53:08 freenas kernel: GEOM_ELI: Detached da0p1.eli on last close.
Aug 23 18:53:08 freenas notifier: geli: No such device: /dev/da0p1.
Aug 23 18:53:57 freenas kernel: mps0: mpssas_alloc_tm freezing simq
Aug 23 18:53:58 freenas kernel: mps0: mpssas_remove_complete on handle 0x000a, IOCStatus= 0x0
Aug 23 18:53:58 freenas kernel: mps0: mpssas_free_tm releasing simq
Aug 23 18:53:58 freenas kernel: (da0:mps0:0:21:0): lost device - 0 outstanding, 1 refs
Aug 23 18:53:58 freenas kernel: GEOM_ELI: Device gptid/6125e604-9d54-11e2-9312-000c29cec408.eli destroyed.
Aug 23 18:53:58 freenas kernel: (da0:mps0:0:21:0): removing device entry
Aug 23 19:03:19 freenas kernel: da0 at mps0 bus 0 scbus0 target 27 lun 0
Aug 23 19:03:19 freenas kernel: da0: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device
Aug 23 19:03:19 freenas kernel: da0: 150.000MB/s transfers
Aug 23 19:03:19 freenas kernel: da0: Command Queueing enabled
Aug 23 19:03:19 freenas kernel: da0: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Aug 23 19:05:04 freenas notifier: 1+0 records in
Aug 23 19:05:04 freenas notifier: 1+0 records out
Aug 23 19:05:04 freenas notifier: 1048576 bytes transferred in 0.009222 secs (113703374 bytes/sec)
Aug 23 19:05:04 freenas notifier: dd: /dev/da0: short write on character device
Aug 23 19:05:04 freenas notifier: dd: /dev/da0: end of device
Aug 23 19:05:04 freenas notifier: 5+0 records in
Aug 23 19:05:04 freenas notifier: 4+1 records out
Aug 23 19:05:04 freenas notifier: 4284416 bytes transferred in 0.080944 secs (52930579 bytes/sec)
Aug 23 19:05:16 freenas kernel: GEOM_ELI: Device gptid/27c029db-0c16-11e3-b6f5-002215ab012f.eli created.
Aug 23 19:05:16 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128
Aug 23 19:05:16 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 23 19:05:21 freenas kernel: GEOM_ELI: Device da0p1.eli created.
Aug 23 19:05:21 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128
Aug 23 19:05:21 freenas kernel: GEOM_ELI:    Crypto: hardware
Aug 24 08:44:37 freenas notifier: 1+0 records in
Aug 24 08:44:37 freenas notifier: 1+0 records out
Aug 24 08:44:37 freenas notifier: 64 bytes transferred in 0.000024 secs (2657777 bytes/sec)
Aug 24 08:44:44 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ]
Aug 24 08:45:07 freenas notifier: 1+0 records in
Aug 24 08:45:07 freenas notifier: 1+0 records out
Aug 24 08:45:07 freenas notifier: 64 bytes transferred in 0.000024 secs (2657777 bytes/sec)
Aug 24 08:45:07 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set recovery key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ]
Aug 24 08:47:18 freenas notifier: 1+0 records in
Aug 24 08:47:18 freenas notifier: 1+0 records out
Aug 24 08:47:18 freenas notifier: 64 bytes transferred in 0.000025 secs (2556528 bytes/sec)
Aug 24 08:47:18 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set recovery key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ]


Edit:

If I take a look into the freenas-v1.db:


Code:
[root@freenas] /data# sqlite3 /data/freenas-v1.db "select * from storage_disk;"

1|Maximum|30|WD-WCAZAF9XXXXX||{serial}WD-WCAZAF9XXXXX|1|128|Auto||||1|da0

1|Maximum|30|WD-WCAZAK9XXXXX||{serial}WD-WCAZAK9XXXXX|1|128|Auto||||2|da1

1|Maximum|30|WD-WCAZAK9XXXXX||{serial}WD-WCAZAK9XXXXX|1|128|Auto||||3|da2

1|Maximum|30|WD-WCAZAK9XXXXX||{serial}WD-WCAZAK9XXXXX|1|128|Auto||||4|da3

1|Maximum|30|WD-WCAZAK9XXXXX||{serial}WD-WCAZAK9XXXXX|1|128|Auto||||5|da4

1|Maximum|30|WD-WCAZAK8XXXXX||{serial}WD-WCAZAK8XXXXX|1|128|Auto||||6|da5

sqlite3 /data/freenas-v1.db "select * from storage_encrypteddisk;"
1|1|5|gptid/65977639-9d54-11e2-9312-000c29cec408
2|1|6|gptid/66bc5d54-9d54-11e2-9312-000c29cec408
3|1|1|gptid/6125e604-9d54-11e2-9312-000c29cec408
4|1|2|gptid/622428ec-9d54-11e2-9312-000c29cec408
5|1|3|gptid/634a7ede-9d54-11e2-9312-000c29cec408
6|1|4|gptid/64712091-9d54-11e2-9312-000c29cec408
7|1|1|gptid/27c029db-0c16-11e3-b6f5-002215ab012f

sqlite3 /data/freenas-v1.db "delete from storage_encrypteddisk where id=3;"
sqlite3 /data/freenas-v1.db "select * from storage_encrypteddisk;"
1|1|5|gptid/65977639-9d54-11e2-9312-000c29cec408
2|1|6|gptid/66bc5d54-9d54-11e2-9312-000c29cec408
4|1|2|gptid/622428ec-9d54-11e2-9312-000c29cec408
5|1|3|gptid/634a7ede-9d54-11e2-9312-000c29cec408
6|1|4|gptid/64712091-9d54-11e2-9312-000c29cec408
7|1|1|gptid/27c029db-0c16-11e3-b6f5-002215ab012f


And deleted ID 3 of the storage_encrypteddisk and could create new key and recovery key.

Code:
Aug 24 11:38:53 freenas notifier: 1+0 records in
Aug 24 11:38:53 freenas notifier: 1+0 records out
Aug 24 11:38:53 freenas notifier: 64 bytes transferred in 0.000024 secs (2657777 bytes/sec)
Aug 24 11:39:33 freenas notifier: 1+0 records in
Aug 24 11:39:33 freenas notifier: 1+0 records out
Aug 24 11:39:33 freenas notifier: 64 bytes transferred in 0.000023 secs (2767376 bytes/sec)


I did not changed the passphrase as I think it is not needed.

How can I test if this solves the problem without dismounting the zfs pool and try to mount it again? Could some official please confirm or describe how to validate this solution?
 
Last edited:
C

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
It looks like you've done everything correctly. The onyl way I know of to verify that your keys are valid is to reboot and try to use them.
 
David E

David E

Contributor
Joined
Nov 1, 2013
Messages
119
FYI this is still an issue as of 9.3, today I removed my SLOG to upgrade to a new one, when I added the new one via the volume manager it failed while trying to key it because it also tried to key the unplugged disk. The fix for me was to remove the new SLOG from the pool, delete both the old SLOG, and new SLOG's entries in the storage_encrypteddisk table, then add the new SLOG again via the volume manager. This really needs to be fixed.
 
SweetAndLow

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
is there a bug for this?
 
F

FreeNAS_DIY

Cadet
Joined
Jul 20, 2014
Messages
6
I can also confirm it.

One disk in my RAIDZ2 Pool had an issue with the power supply -> Degraded state. After fixing that problem the only button was "Replace", I had no chance to put the drive offline or to detach it. After pressing that "replace" button the resilvering starts. Approx 34 hrs. later the resilvering was complete, so I tried to re-key the pool as stated in the manual. Instead of a success message I got the same error message, stating "unable to set key, unable to open gptid ...." and so on.

So many thanks to wtfuar for showing the necessary steps, I located and deleted the wrong drive with the sqlite commands and voila: Re-keying was successful.

Hard to believe that this bug persisted over two years. Hopefully it will be fixed very soon, without this thread the result would have been disastrous.
 
Last edited:
Status
Not open for further replies.

Similar threads

V
  • Locked
Problem on drives detached/destroyed FreeNAS 11
Replies
10
Views
5K
joeschmuck
joeschmuck
Quadgnim
  • Locked
lost device
Replies
9
Views
2K
np141592
N
NadoNate
  • Locked
SOLVED Device detaches randomly [Help]
Replies
6
Views
1K
NadoNate
NadoNate
K
  • Locked
Volume Encryption always on?
Replies
2
Views
1K
survive
survive
wtfuar
SOLVED Wrong key for gptid/... : Failed to geli attach
Replies
5
Views
5K
EsTaF
EsTaF
Top