Raidz1 pool unavailable due to two disks being "UNAVAIL"

Status
Not open for further replies.
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
Hello,
I have been searching for a while now, and I haven't found anything matching my case 100%, so I'll try my luck posting my problems to you guys. Sorry if my problem is obvious or already answered! Also, as you may notice, I'm quite a noobie, so please forgive any stupid questions.

One thread I found that was pretty similar, but I believe not completely the same was these:
https://forums.freenas.org/index.php?threads/lost-zpool-please-help.13878/
https://forums.freenas.org/index.ph...-one-disk-offline-cannot-be-put-online.14104/

So, my problem is this:
I build a FreeNAS v8 server some years ago, but then suddenly the motherboard died. I took out the 6 disks and almost forgot about them for a long while. I then set up a virtual Debian machine on my Windows 10 computer, and made copies of the disks to virtual harddisks of all of the disks, except for one of the disks which experienced a hardware error rendering it useless (it connected/disconnected repeatedly) and therefore I have no backup if this one.

Now I have set up the remaining 5 disks in a virtual FreeNAS install, and I've tried 8.3.2 and 9.10.2-U1 with same results: When I run zpool import I see that 2 of the 6 disks is UNAVAIL, and not just the one I haven't connected to the virtual machine.

Code:
[root@freenas] /# zpool import
   pool: RaidZ
	 id: 13844435100348741427
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
		devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

		RaidZ										   UNAVAIL  insufficient replicas
		  raidz1-0									  UNAVAIL  insufficient replicas
			gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8  ONLINE
			gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8  ONLINE
			5596984570285208485						 UNAVAIL  cannot open
			6519031140053721195						 UNAVAIL  cannot open
			gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8  ONLINE
			gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8  ONLINE

glabel status | grep p2 gives the following output (ada5p2 is missing):
Code:
[root@freenas] /# glabel status | grep p2
gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8	 N/A  ada1p2
gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8	 N/A  ada2p2
gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8	 N/A  ada3p2
gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8	 N/A  ada4p2

I also ran gpart show -l and then the output said the GPT was corrupt on all of the disks. If I remember correctly, I ran gpart recover /dev/ada1 on all of the disks (ada[1-5]), and then the "corrupt" label disappeared when I ran gpart show -l again.

I'm not sure how to proceed, so I'll just post the output of some of the commands I've stumbled across, in case they can prove helpful.

gpart status
Code:
[root@freenas] /# gpart status
  Name  Status  Components
ada0p1	  OK  ada0
ada0p2	  OK  ada0
ada1p1	  OK  ada1
ada1p2	  OK  ada1
ada2p1	  OK  ada2
ada2p2	  OK  ada2
ada3p1	  OK  ada3
ada3p2	  OK  ada3
ada4p1	  OK  ada4
ada4p2	  OK  ada4
ada5p1	  OK  ada5
ada5p2	  OK  ada5

gpart list
Code:
[root@freenas] /# gpart list
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 16777182
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   rawuuid: e288176c-dd09-11e6-8b42-08002710025b
   rawtype: 21686148-6449-6e6f-744e-656564454649
   label: (null)
   length: 524288
   offset: 17408
   type: bios-boot
   index: 1
   end: 1057
   start: 34
2. Name: ada0p2
   Mediasize: 8589369344 (8.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r1w1e1
   rawuuid: e28a74ff-dd09-11e6-8b42-08002710025b
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 8589369344
   offset: 544768
   type: freebsd-zfs
   index: 2
   end: 16777175
   start: 1064
Consumers:
1. Name: ada0
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Mode: r1w1e2

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 4294965214
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   rawuuid: c793d511-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada1p2
   Mediasize: 1998251367936 (1.8T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: c7ac0491-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251367936
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029134
   start: 4194432
Consumers:
1. Name: ada1
   Mediasize: 2199022206976 (2.0T)
   Sectorsize: 512
   Mode: r0w0e0

Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 4294965214
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   rawuuid: c991b271-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada2p2
   Mediasize: 1998250286080 (1.8T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: c9a6908b-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998250286080
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907027021
   start: 4194432
Consumers:
1. Name: ada2
   Mediasize: 2199022206976 (2.0T)
   Sectorsize: 512
   Mode: r0w0e0

Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 4294965214
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   rawuuid: c7123a39-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada3p2
   Mediasize: 1998251367936 (1.8T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: c7259a85-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251367936
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029134
   start: 4194432
Consumers:
1. Name: ada3
   Mediasize: 2199022206976 (2.0T)
   Sectorsize: 512
   Mode: r0w0e0

Geom name: ada4
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 4294965214
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada4p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   rawuuid: c914ad72-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada4p2
   Mediasize: 1998251367936 (1.8T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: c929c4be-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251367936
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029134
   start: 4194432
Consumers:
1. Name: ada4
   Mediasize: 2199022206976 (2.0T)
   Sectorsize: 512
   Mode: r0w0e0

Geom name: ada5
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 4294965214
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada5p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   rawuuid: c7123a39-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada5p2
   Mediasize: 1998251367936 (1.8T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: c7259a85-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251367936
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029134
   start: 4194432
Consumers:
1. Name: ada5
   Mediasize: 2199022206976 (2.0T)
   Sectorsize: 512
   Mode: r0w0e0

camcontrol devlist
Code:
[root@freenas] /# camcontrol devlist
<VBOX HARDDISK 1.0>				at scbus0 target 0 lun 0 (pass0,ada0)
<VBOX CD-ROM 1.0>				  at scbus0 target 1 lun 0 (pass1,cd0)
<VBOX HARDDISK 1.0>				at scbus2 target 0 lun 0 (pass2,ada1)
<VBOX HARDDISK 1.0>				at scbus3 target 0 lun 0 (pass3,ada2)
<VBOX HARDDISK 1.0>				at scbus4 target 0 lun 0 (pass4,ada3)
<VBOX HARDDISK 1.0>				at scbus5 target 0 lun 0 (pass5,ada4)
<VBOX HARDDISK 1.0>				at scbus6 target 0 lun 0 (pass6,ada5)

dmesg
Code:
[root@freenas] /# dmesg
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
		The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-STABLE #0 r295946+1805185(9.10.2-STABLE): Wed Jan 11 17:12:42 UTC 2017
	root@gauntlet:/freenas-9.10-releng/_BE/objs/freenas-9.10-releng/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Core(TM) i7 CPU		 960  @ 3.20GHz (3197.81-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  Features2=0x180209<SSE3,MON,SSSE3,SSE4.1,SSE4.2>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 4831838208 (4608 MB)
avail memory = 4110860288 (3920 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <VBOX   VBOXAPIC>
random: <Software, Yarrow> initialized
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
aesni0: No AESNI support.
padlock0: No ACE support.
cryptosoft0: <software crypto> on motherboard
acpi0: <VBOX VBOXXSDT> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd000-0xd00f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
vgapci0: <VGA-compatible display> mem 0xe0000000-0xe07fffff irq 18 at device 2.0 on pci0
vgapci0: Boot video device
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0xd010-0xd017 mem 0xf0000000-0xf001ffff irq 19 at device 3.0 on pci0
em0: Ethernet address: 08:00:27:10:02:5b
pci0: <multimedia, audio> at device 5.0 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xf0804000-0xf0804fff irq 22 at device 6.0 on pci0
usbus0 on ohci0
pci0: <bridge> at device 7.0 (no driver attached)
em1: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0xd240-0xd247 mem 0xf0820000-0xf083ffff irq 16 at device 8.0 on pci0
em1: Ethernet address: 08:00:27:50:fa:84
ehci0: <Intel 82801FB (ICH6) USB 2.0 controller> mem 0xf0840000-0xf0840fff irq 19 at device 11.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
ahci0: <Intel ICH8M AHCI SATA controller> port 0xd248-0xd24f,0xd258-0xd25f,0xd270-0xd27f mem 0xf0842000-0xf0843fff irq 21 at device 13.0 on pci0
ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
acpi_acad0: <AC Adapter> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
qpi0: <QPI system bus> on motherboard
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xe2000-0xe2fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
Event timer "RTC" frequency 32768 Hz quality 0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
			to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
em0: link state changed to UP
em1: link state changed to UP
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Apple> at usbus0
uhub0: <Apple OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub0: 12 ports with 12 removable, self powered
uhub1: 12 ports with 12 removable, self powered
random: unblocking device.
ada0 at ata0 bus 0 scbus0 target 0 lun 0
ada0: <VBOX HARDDISK 1.0> ATA-6 device
ada0: Serial Number VB1b86c12f-ff9f85ed
ada0: 33.300MB/s transfers (UDMA2, PIO 65536bytes)
ada0: 8192MB (16777216 512 byte sectors)
ada0: Previously was known as ad0
ada1 at ahcich0 bus 0 scbus2 target 0 lun 0
ada1: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada1: Serial Number VB1329485b-40a7f2a8
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2097151MB (4294965248 512 byte sectors)
ada1: Previously was known as ad4
ada2 at ahcich1 bus 0 scbus3 target 0 lun 0
ada2: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada2: Serial Number VBe1186804-10b17d7a
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 2097151MB (4294965248 512 byte sectors)
ada2: Previously was known as ad6
ada3 at ahcich2 bus 0 scbus4 target 0 lun 0
ada3: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada3: Serial Number VBfd2afe42-8d9623c3
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 2097151MB (4294965248 512 byte sectors)
ada3: Previously was known as ad8
ada4 at ahcich3 bus 0 scbus5 target 0 lun 0
ada4: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada4: Serial Number VBa292eddb-372ecccf
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 2097151MB (4294965248 512 byte sectors)
ada4: Previously was known as ad10
ada5 at ahcich4 bus 0 scbus6 target 0 lun 0
ada5: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada5: Serial Number VBb0677f0e-eaf0d780
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 2097151MB (4294965248 512 byte sectors)
ada5: Previously was known as ad12
cd0 at ata0 bus 0 scbus0 target 1 lun 0
cd0: <VBOX CD-ROM 1.0> Removable CD-ROM SCSI device
cd0: Serial Number VB1-01f003f6
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
Timecounter "TSC-low" frequency 1598904247 Hz quality 1000
Trying to mount root from zfs:freenas-boot/ROOT/default []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
hwpc_core: unknown PMC architecture: 0
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI>

zdb -l /dev/ada5p2
Code:
[root@freenas] /# zdb -l /dev/ada5p2
--------------------------------------------
LABEL 0
--------------------------------------------
	version: 15
	name: 'RaidZ'
	state: 0
	txg: 13018627
	pool_guid: 13844435100348741427
	hostid: 3271116453
	hostname: 'freenas.local'
	top_guid: 6075442249147471512
	guid: 15374796970963134483
	vdev_tree:
		type: 'raidz'
		id: 0
		guid: 6075442249147471512
		nparity: 1
		metaslab_array: 23
		metaslab_shift: 36
		ashift: 12
		asize: 11989471985664
		is_log: 0
		children[0]:
			type: 'disk'
			id: 0
			guid: 15374796970963134483
			path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 158
		children[1]:
			type: 'disk'
			id: 1
			guid: 6420862054695758756
			path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 157
		children[2]:
			type: 'disk'
			id: 2
			guid: 5596984570285208485
			path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 156
		children[3]:
			type: 'disk'
			id: 3
			guid: 6519031140053721195
			path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			not_present: 1
			DTL: 155
		children[4]:
			type: 'disk'
			id: 4
			guid: 5085730408897018382
			path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 154
		children[5]:
			type: 'disk'
			id: 5
			guid: 2261504104292084216
			path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 99
--------------------------------------------
LABEL 1
--------------------------------------------
	version: 15
	name: 'RaidZ'
	state: 0
	txg: 13018627
	pool_guid: 13844435100348741427
	hostid: 3271116453
	hostname: 'freenas.local'
	top_guid: 6075442249147471512
	guid: 15374796970963134483
	vdev_tree:
		type: 'raidz'
		id: 0
		guid: 6075442249147471512
		nparity: 1
		metaslab_array: 23
		metaslab_shift: 36
		ashift: 12
		asize: 11989471985664
		is_log: 0
		children[0]:
			type: 'disk'
			id: 0
			guid: 15374796970963134483
			path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 158
		children[1]:
			type: 'disk'
			id: 1
			guid: 6420862054695758756
			path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 157
		children[2]:
			type: 'disk'
			id: 2
			guid: 5596984570285208485
			path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 156
		children[3]:
			type: 'disk'
			id: 3
			guid: 6519031140053721195
			path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			not_present: 1
			DTL: 155
		children[4]:
			type: 'disk'
			id: 4
			guid: 5085730408897018382
			path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 154
		children[5]:
			type: 'disk'
			id: 5
			guid: 2261504104292084216
			path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 99
--------------------------------------------
LABEL 2
--------------------------------------------
	version: 15
	name: 'RaidZ'
	state: 0
	txg: 13018627
	pool_guid: 13844435100348741427
	hostid: 3271116453
	hostname: 'freenas.local'
	top_guid: 6075442249147471512
	guid: 15374796970963134483
	vdev_tree:
		type: 'raidz'
		id: 0
		guid: 6075442249147471512
		nparity: 1
		metaslab_array: 23
		metaslab_shift: 36
		ashift: 12
		asize: 11989471985664
		is_log: 0
		children[0]:
			type: 'disk'
			id: 0
			guid: 15374796970963134483
			path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 158
		children[1]:
			type: 'disk'
			id: 1
			guid: 6420862054695758756
			path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 157
		children[2]:
			type: 'disk'
			id: 2
			guid: 5596984570285208485
			path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 156
		children[3]:
			type: 'disk'
			id: 3
			guid: 6519031140053721195
			path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			not_present: 1
			DTL: 155
		children[4]:
			type: 'disk'
			id: 4
			guid: 5085730408897018382
			path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 154
		children[5]:
			type: 'disk'
			id: 5
			guid: 2261504104292084216
			path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 99
--------------------------------------------
LABEL 3
--------------------------------------------
	version: 15
	name: 'RaidZ'
	state: 0
	txg: 13018627
	pool_guid: 13844435100348741427
	hostid: 3271116453
	hostname: 'freenas.local'
	top_guid: 6075442249147471512
	guid: 15374796970963134483
	vdev_tree:
		type: 'raidz'
		id: 0
		guid: 6075442249147471512
		nparity: 1
		metaslab_array: 23
		metaslab_shift: 36
		ashift: 12
		asize: 11989471985664
		is_log: 0
		children[0]:
			type: 'disk'
			id: 0
			guid: 15374796970963134483
			path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 158
		children[1]:
			type: 'disk'
			id: 1
			guid: 6420862054695758756
			path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 157
		children[2]:
			type: 'disk'
			id: 2
			guid: 5596984570285208485
			path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 156
		children[3]:
			type: 'disk'
			id: 3
			guid: 6519031140053721195
			path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			not_present: 1
			DTL: 155
		children[4]:
			type: 'disk'
			id: 4
			guid: 5085730408897018382
			path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 154
		children[5]:
			type: 'disk'
			id: 5
			guid: 2261504104292084216
			path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 99

Some observations I make is:
  • In the last output, I see that the fourth disk (id: 3) is "not_present: 1", which corresponds with my previous observation that the fourth disk was the one with the hardware error.
  • The third disk (id: 2) is connected, but doesn't show in glabel status
Thank you so much for any suggestions! I'm starting to get tired of searching, and I really don't want to lose this data. :(
 
SweetAndLow

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Have you tried with the real disks? The fact that you have lost 1 disks already does not put the odds in your favor. The chance of a read error on one of the other disks is really high.
 
R

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
Your only hope of recovering any data is to get at least 5 of the 6 disks connected at the same time. Do you have the original physical disks? If so, you could try cloning the one that was failing using something like GNU ddrescue. If not, you have to somehow persuade your virtual FreeNAS to recognize ada0 or ada5 (if those are indeed the missing devices). I've seen threads in these forums where people were able to recover corrupted partitions, but it's not something I can guide you on.
 
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
SweetAndLow said:
Have you tried with the real disks? The fact that you have lost 1 disks already does not put the odds in your favor. The chance of a read error on one of the other disks is really high.
Click to expand...
I have tried, but haven't tried too much recovery on them. I have some output from some commands below. Please let me know if I need to run anything else.
It seems to be a problem with the GPT info on ada4 and ada5. Is there any chance of doing anything wrong with gpart recover?
Robert Trevellyan said:
Your only hope of recovering any data is to get at least 5 of the 6 disks connected at the same time. Do you have the original physical disks? If so, you could try cloning the one that was failing using something like GNU ddrescue. If not, you have to somehow persuade your virtual FreeNAS to recognize ada0 or ada5 (if those are indeed the missing devices). I've seen threads in these forums where people were able to recover corrupted partitions, but it's not something I can guide you on.
Click to expand...
I have 5 of the disks. The 6th have I returned to the store, as I thought I couldn't use it for anything, and that I could rebuild it instead.
Somehow I have got this idea that the data itself is not corrupt, but only the partition information, but of course, I may be wrong.
I also had the same idea of "tricking" FreeNAS to recognize the disks, but I didn't know if it was possible.

So, I tried connecting the physical disks to the virtual machine (using
VBoxManage internalcommands createrawvmdk
), and ran some of the same commands as before:
zpool import
Code:
[root@freenas] ~# zpool import
   pool: RaidZ
	 id: 13844435100348741427
  state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
   see: http://illumos.org/msg/ZFS-8000-EY
config:

		RaidZ					UNAVAIL  insufficient replicas
		  raidz1-0			   UNAVAIL  insufficient replicas
			ada3p2			   ONLINE
			ada2p2			   ONLINE
			ada4p2			   ONLINE
			6519031140053721195  UNAVAIL  cannot open
			5085730408897018382  UNAVAIL  cannot open
			ada1p2			   ONLINE

ls /dev/ | grep ada
Code:
[root@freenas] ~# ls /dev/ | grep ada
ada0
ada0p1
ada0p2
ada1
ada1p1
ada1p2
ada2
ada2p1
ada2p2
ada3
ada3p1
ada3p2
ada4
ada4p1
ada4p2
ada5

gpart show -l
Code:
[root@freenas] ~# gpart show -l
=>	  34  16777149  ada0  GPT  (8.0G)
		34	  1024	 1  (null)  (512K)
	  1058		 6		- free -  (3.0K)
	  1064  16776112	 2  (null)  (8.0G)
  16777176		 7		- free -  (3.5K)

=>		34  3907026988  ada1  GPT  (1.8T)
		  34		  94		- free -  (47K)
		 128	 4194304	 1  (null)  (2.0G)
	 4194432  3902832590	 2  (null)  (1.8T)

=>		34  3907029101  ada2  GPT  (1.8T)
		  34		  94		- free -  (47K)
		 128	 4194304	 1  (null)  (2.0G)
	 4194432  3902834703	 2  (null)  (1.8T)

=>		34  3907029101  ada3  GPT  (1.8T)
		  34		  94		- free -  (47K)
		 128	 4194304	 1  (null)  (2.0G)
	 4194432  3902834703	 2  (null)  (1.8T)

=>		34  3907026988  ada4  GPT  (1.8T) [CORRUPT]
		  34		  94		- free -  (47K)
		 128	 4194304	 1  (null)  (2.0G)
	 4194432  3902832590	 2  (null)  (1.8T)

gpart list
Code:
[root@freenas] ~# gpart list
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 16777182
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   rawuuid: e288176c-dd09-11e6-8b42-08002710025b
   rawtype: 21686148-6449-6e6f-744e-656564454649
   label: (null)
   length: 524288
   offset: 17408
   type: bios-boot
   index: 1
   end: 1057
   start: 34
2. Name: ada0p2
   Mediasize: 8589369344 (8.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r1w1e1
   rawuuid: e28a74ff-dd09-11e6-8b42-08002710025b
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 8589369344
   offset: 544768
   type: freebsd-zfs
   index: 2
   end: 16777175
   start: 1064
Consumers:
1. Name: ada0
   Mediasize: 8589934592 (8.0G)
   Sectorsize: 512
   Mode: r1w1e2

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907027021
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   rawuuid: c991b271-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada1p2
   Mediasize: 1998250286080 (1.8T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: c9a6908b-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998250286080
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907027021
   start: 4194432
Consumers:
1. Name: ada1
   Mediasize: 2000397852160 (1.8T)
   Sectorsize: 512
   Mode: r0w0e0

Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   rawuuid: c793d511-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada2p2
   Mediasize: 1998251367936 (1.8T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: c7ac0491-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251367936
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029134
   start: 4194432
Consumers:
1. Name: ada2
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Mode: r0w0e0

Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   rawuuid: c7123a39-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada3p2
   Mediasize: 1998251367936 (1.8T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: c7259a85-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251367936
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029134
   start: 4194432
Consumers:
1. Name: ada3
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Mode: r0w0e0

Geom name: ada4
modified: false
state: CORRUPT
fwheads: 16
fwsectors: 63
last: 3907027021
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada4p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   rawuuid: c81162da-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada4p2
   Mediasize: 1998250286080 (1.8T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: c826480a-0b0e-11e2-baee-001fd0af0df8
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998250286080
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907027021
   start: 4194432
Consumers:
1. Name: ada4
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Mode: r0w0e0

camcontrol devlist
Code:
[root@freenas] ~# camcontrol devlist
<VBOX HARDDISK 1.0>				at scbus0 target 0 lun 0 (pass0,ada0)
<VBOX CD-ROM 1.0>				  at scbus0 target 1 lun 0 (pass1,cd0)
<VBOX HARDDISK 1.0>				at scbus2 target 0 lun 0 (pass2,ada1)
<VBOX HARDDISK 1.0>				at scbus3 target 0 lun 0 (pass3,ada2)
<VBOX HARDDISK 1.0>				at scbus4 target 0 lun 0 (pass4,ada3)
<VBOX HARDDISK 1.0>				at scbus5 target 0 lun 0 (pass5,ada4)
<VBOX HARDDISK 1.0>				at scbus6 target 0 lun 0 (pass6,ada5)

dmesg
Code:
[root@freenas] ~# dmesg
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
		The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-STABLE #0 r295946+1805185(9.10.2-STABLE): Wed Jan 11 17:12:42 UTC 2017
	root@gauntlet:/freenas-9.10-releng/_BE/objs/freenas-9.10-releng/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Core(TM) i7 CPU		 960  @ 3.20GHz (3197.81-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  Features2=0x180209<SSE3,MON,SSSE3,SSE4.1,SSE4.2>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 4831838208 (4608 MB)
avail memory = 4110860288 (3920 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <VBOX   VBOXAPIC>
random: <Software, Yarrow> initialized
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
aesni0: No AESNI support.
padlock0: No ACE support.
cryptosoft0: <software crypto> on motherboard
acpi0: <VBOX VBOXXSDT> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd000-0xd00f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
vgapci0: <VGA-compatible display> mem 0xe0000000-0xe07fffff irq 18 at device 2.0 on pci0
vgapci0: Boot video device
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0xd010-0xd017 mem 0xf0000000-0xf001ffff irq 19 at device 3.0 on pci0
em0: Ethernet address: 08:00:27:10:02:5b
pci0: <multimedia, audio> at device 5.0 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xf0804000-0xf0804fff irq 22 at device 6.0 on pci0
usbus0 on ohci0
pci0: <bridge> at device 7.0 (no driver attached)
em1: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0xd240-0xd247 mem 0xf0820000-0xf083ffff irq 16 at device 8.0 on pci0
em1: Ethernet address: 08:00:27:50:fa:84
ehci0: <Intel 82801FB (ICH6) USB 2.0 controller> mem 0xf0840000-0xf0840fff irq 19 at device 11.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
ahci0: <Intel ICH8M AHCI SATA controller> port 0xd248-0xd24f,0xd258-0xd25f,0xd270-0xd27f mem 0xf0842000-0xf0843fff irq 21 at device 13.0 on pci0
ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
acpi_acad0: <AC Adapter> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
qpi0: <QPI system bus> on motherboard
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xe2000-0xe2fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
Event timer "RTC" frequency 32768 Hz quality 0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
			to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
em0: link state changed to UP
em1: link state changed to UP
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Apple> at usbus0
uhub0: <Apple OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub0: 12 ports with 12 removable, self powered
uhub1: 12 ports with 12 removable, self powered
random: unblocking device.
ada0 at ata0 bus 0 scbus0 target 0 lun 0
ada0: <VBOX HARDDISK 1.0> ATA-6 device
ada0: Serial Number VB1b86c12f-ff9f85ed
ada0: 33.300MB/s transfers (UDMA2, PIO 65536bytes)
ada0: 8192MB (16777216 512 byte sectors)
ada0: Previously was known as ad0
ada1 at ahcich0 bus 0 scbus2 target 0 lun 0
ada1: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada1: Serial Number VBa005213e-2623e475
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907728MB (3907027055 512 byte sectors)
ada1: Previously was known as ad4
ada2 at ahcich1 bus 0 scbus3 target 0 lun 0
ada2: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada2: Serial Number VBc54e5701-a34fe005
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors)
ada2: Previously was known as ad6
ada3 at ahcich2 bus 0 scbus4 target 0 lun 0
ada3: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada3: Serial Number VB4543cf3a-a508a340
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors)
ada3: Previously was known as ad8
ada4 at ahcich3 bus 0 scbus5 target 0 lun 0
ada4: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada4: Serial Number VB7c192c14-e8249c7d
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors)
ada4: Previously was known as ad10
ada5 at ahcich4 bus 0 scbus6 target 0 lun 0
ada5: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada5: Serial Number VBe219f1c2-b58de228
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 1907728MB (3907027055 512 byte sectors)
ada5: Previously was known as ad12
cd0 at ata0 bus 0 scbus0 target 1 lun 0
cd0: <VBOX CD-ROM 1.0> Removable CD-ROM SCSI device
cd0: Serial Number VB1-01f003f6
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
Timecounter "TSC-low" frequency 1598903515 Hz quality 1000
GEOM: ada4: the secondary GPT header is not in the last LBA.
GEOM: ada5: corrupt or invalid GPT detected.
GEOM: ada5: GPT rejected -- may not be recoverable.
Trying to mount root from zfs:freenas-boot/ROOT/default []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
hwpc_core: unknown PMC architecture: 0
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI>


Now I'm off to work for some hours, but I'm glad for all the help I can get. Thank you!
 
R

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
It looks like the OS sees 5 data disks, but gpart only sees partition tables on 4, which is 2 short of the original vdev's 6 disks, so even if you fix this,
morra said:
Geom name: ada4 modified: false state: CORRUPT
Click to expand...
you're still short one disk.

I don't know much about gpart, but it looks to me like you have one disk with a corrupt partition table and one with no partition table. Perhaps you can clone the partition table from one of the good disks to the two problem disks. But you might want to wait for confirmation from someone more knowledgable.
 
Last edited:
R

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
morra said:
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
Click to expand...
Don't attempt recovery with a recent version of FreeNAS unless you can give the VM the required minimum of 8GB.
 
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
Robert Trevellyan said:
It looks like the OS sees 5 data disks, but gpart only sees partition tables on 4, which is 2 short of the original vdev's 6 disks, so even if you fix this,

you're still short one disk.
Click to expand...
I started gpart recover /dev/ada4, but it seems to take time. I really hope this wasn't a bad idea...
Robert Trevellyan said:
I don't know much about gpart, but it looks to me like you have one disk with a corrupt partition table and one with no partition table. Perhaps you can clone the partition table from one of the good disks to the two problem disks. But you might want to wait for confirmation from someone more knowledgable.
Click to expand...
That might actually work, as it seems that all partitions are of equal sizes and positions, but I want to make sure I don't overwrite any data when doing so.
Robert Trevellyan said:
Don't attempt recovery with a recent version of FreeNAS unless you can give the VM the required minimum of 8GB.
Click to expand...
Thanks! I set it to 10GB before starting the GPT recovery of ada4.
 
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
I've spent 7-8 hours on this today, and I don't think I'm any closer to a solution, probably the opposite. I tried copying the GPT information from one disk to another, and I believe I made it worse as another one of my disks have got GPT errors. So, that was stupid of me. Also I notice that the sector size has been reduced on one of the disks, which is why the GPT info is now wrong (it says the partition is outside of the disk).

I still somehow has a hope for the virtual backups I made though, as all GPT information seems to be valid. I just can't understand why I have a disk that is unavailable by ZFS, but seems to be working in every other way. If it's possible to "trick" ZFS to see the disk as online, I would very much like some help in finding out how.

Can it be some ZFS info stored in the start/end of one of the partitions, that could be different in the failing disk, than in the others?

Another thing that gives me a little hope is the output of zdb -l /dev/ada5p2 which says that one of the disks is not present (ergo the other one is detected the same way as the working ones):
Code:
		children[2]:
			type: 'disk'
			id: 2
			guid: 5596984570285208485
			path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 156
		children[3]:
			type: 'disk'
			id: 3
			guid: 6519031140053721195
			path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			not_present: 1
			DTL: 155

I'll go to bed now, but I will answer as soon as I can, in case any helpful souls out there have a clue. Thanks!
 
R

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
I wish I could help, but I'm really guessing at this point.

Can you transfer the apparently valid partition information from the backups to the physical disks?

I assume you need to preserve all the unique identifiers of each disk.
 
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
Robert Trevellyan said:
Can you transfer the apparently valid partition information from the backups to the physical disks?

I assume you need to preserve all the unique identifiers of each disk.
Click to expand...
I can try, but I'm not sure how that's different from copying a partition table from a working disk. The problem I met when I tried was that the partition table I tried to write was invalid as the end sector of the hard disk had suddenly been moved, so that the partition I try to copy doesn't fit on the disk.

Also, when I try to write to the physical disks now, VirtualBox hangs with an error message about an I/O error (when I think of it now, it may be because VBox can't write t a sector outside of the disk :confused:). I can unpause the machine, but only to get the same error message again and again.

You may have a point regarding the identifier, but not sure how to copy the partition info when the OS won't let me. I can try rebooting my host computer into a Ubuntu live CD in order to copy the partition information there (and not have VBox in my way).
Code:
# sgdisk --replicate=/dev/DISK_DESTINATION /dev/DISK_SOURCE
# sgdisk -G /dev/DISK_DESTINATION


Thanks for helping! I'll let you know how it goes. But I won't be able to try before Monday, so I hope you'll have a good weekend! :)
 
R

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
morra said:
I'm not sure how that's different from copying a partition table from a working disk
Click to expand...
I imagined a 1:1 correspondence between backups and physical disks, perhaps avoiding issues with sector counts and unique identifiers.

Good luck!
 
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
Robert Trevellyan said:
I imagined a 1:1 correspondence between backups and physical disks, perhaps avoiding issues with sector counts and unique identifiers.
Click to expand...
Ah, smart! The only downside is that the physical disks seem to have encountered some errors after the backups were made, so the sector counts does not match anymore :(

However, I might have found something interesting on the backups. By comparing the output from zdb -l /dev/ada3p2 and zdb -l /dev/ada5p2, they look too much alike in the beginning (everything is the same before Label 2), but there are errors in the output from ada3p2: failed to unpack label 2 and failed to unpack label 3.

Complete output from zdb -l /dev/ada3p2:
Code:
[root@freenas] ~# zdb -l /dev/ada3p2
--------------------------------------------
LABEL 0
--------------------------------------------
	version: 15
	name: 'RaidZ'
	state: 0
	txg: 13018627
	pool_guid: 13844435100348741427
	hostid: 3271116453
	hostname: 'freenas.local'
	top_guid: 6075442249147471512
	guid: 15374796970963134483
	vdev_tree:
		type: 'raidz'
		id: 0
		guid: 6075442249147471512
		nparity: 1
		metaslab_array: 23
		metaslab_shift: 36
		ashift: 12
		asize: 11989471985664
		is_log: 0
		children[0]:
			type: 'disk'
			id: 0
			guid: 15374796970963134483
			path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 158
		children[1]:
			type: 'disk'
			id: 1
			guid: 6420862054695758756
			path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 157
		children[2]:
			type: 'disk'
			id: 2
			guid: 5596984570285208485
			path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 156
		children[3]:
			type: 'disk'
			id: 3
			guid: 6519031140053721195
			path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			not_present: 1
			DTL: 155
		children[4]:
			type: 'disk'
			id: 4
			guid: 5085730408897018382
			path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 154
		children[5]:
			type: 'disk'
			id: 5
			guid: 2261504104292084216
			path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 99
--------------------------------------------
LABEL 1
--------------------------------------------
	version: 15
	name: 'RaidZ'
	state: 0
	txg: 13018627
	pool_guid: 13844435100348741427
	hostid: 3271116453
	hostname: 'freenas.local'
	top_guid: 6075442249147471512
	guid: 15374796970963134483
	vdev_tree:
		type: 'raidz'
		id: 0
		guid: 6075442249147471512
		nparity: 1
		metaslab_array: 23
		metaslab_shift: 36
		ashift: 12
		asize: 11989471985664
		is_log: 0
		children[0]:
			type: 'disk'
			id: 0
			guid: 15374796970963134483
			path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 158
		children[1]:
			type: 'disk'
			id: 1
			guid: 6420862054695758756
			path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 157
		children[2]:
			type: 'disk'
			id: 2
			guid: 5596984570285208485
			path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 156
		children[3]:
			type: 'disk'
			id: 3
			guid: 6519031140053721195
			path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			not_present: 1
			DTL: 155
		children[4]:
			type: 'disk'
			id: 4
			guid: 5085730408897018382
			path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 154
		children[5]:
			type: 'disk'
			id: 5
			guid: 2261504104292084216
			path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 99
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3

And the output from zdb -l /dev/ada5p2:
Code:
[root@freenas] ~# zdb -l /dev/ada5p2
--------------------------------------------
LABEL 0
--------------------------------------------
	version: 15
	name: 'RaidZ'
	state: 0
	txg: 13018627
	pool_guid: 13844435100348741427
	hostid: 3271116453
	hostname: 'freenas.local'
	top_guid: 6075442249147471512
	guid: 15374796970963134483
	vdev_tree:
		type: 'raidz'
		id: 0
		guid: 6075442249147471512
		nparity: 1
		metaslab_array: 23
		metaslab_shift: 36
		ashift: 12
		asize: 11989471985664
		is_log: 0
		children[0]:
			type: 'disk'
			id: 0
			guid: 15374796970963134483
			path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 158
		children[1]:
			type: 'disk'
			id: 1
			guid: 6420862054695758756
			path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 157
		children[2]:
			type: 'disk'
			id: 2
			guid: 5596984570285208485
			path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 156
		children[3]:
			type: 'disk'
			id: 3
			guid: 6519031140053721195
			path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			not_present: 1
			DTL: 155
		children[4]:
			type: 'disk'
			id: 4
			guid: 5085730408897018382
			path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 154
		children[5]:
			type: 'disk'
			id: 5
			guid: 2261504104292084216
			path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 99
--------------------------------------------
LABEL 1
--------------------------------------------
	version: 15
	name: 'RaidZ'
	state: 0
	txg: 13018627
	pool_guid: 13844435100348741427
	hostid: 3271116453
	hostname: 'freenas.local'
	top_guid: 6075442249147471512
	guid: 15374796970963134483
	vdev_tree:
		type: 'raidz'
		id: 0
		guid: 6075442249147471512
		nparity: 1
		metaslab_array: 23
		metaslab_shift: 36
		ashift: 12
		asize: 11989471985664
		is_log: 0
		children[0]:
			type: 'disk'
			id: 0
			guid: 15374796970963134483
			path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 158
		children[1]:
			type: 'disk'
			id: 1
			guid: 6420862054695758756
			path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 157
		children[2]:
			type: 'disk'
			id: 2
			guid: 5596984570285208485
			path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 156
		children[3]:
			type: 'disk'
			id: 3
			guid: 6519031140053721195
			path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			not_present: 1
			DTL: 155
		children[4]:
			type: 'disk'
			id: 4
			guid: 5085730408897018382
			path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 154
		children[5]:
			type: 'disk'
			id: 5
			guid: 2261504104292084216
			path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 99
--------------------------------------------
LABEL 2
--------------------------------------------
	version: 15
	name: 'RaidZ'
	state: 0
	txg: 13018627
	pool_guid: 13844435100348741427
	hostid: 3271116453
	hostname: 'freenas.local'
	top_guid: 6075442249147471512
	guid: 15374796970963134483
	vdev_tree:
		type: 'raidz'
		id: 0
		guid: 6075442249147471512
		nparity: 1
		metaslab_array: 23
		metaslab_shift: 36
		ashift: 12
		asize: 11989471985664
		is_log: 0
		children[0]:
			type: 'disk'
			id: 0
			guid: 15374796970963134483
			path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 158
		children[1]:
			type: 'disk'
			id: 1
			guid: 6420862054695758756
			path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 157
		children[2]:
			type: 'disk'
			id: 2
			guid: 5596984570285208485
			path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 156
		children[3]:
			type: 'disk'
			id: 3
			guid: 6519031140053721195
			path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			not_present: 1
			DTL: 155
		children[4]:
			type: 'disk'
			id: 4
			guid: 5085730408897018382
			path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 154
		children[5]:
			type: 'disk'
			id: 5
			guid: 2261504104292084216
			path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 99
--------------------------------------------
LABEL 3
--------------------------------------------
	version: 15
	name: 'RaidZ'
	state: 0
	txg: 13018627
	pool_guid: 13844435100348741427
	hostid: 3271116453
	hostname: 'freenas.local'
	top_guid: 6075442249147471512
	guid: 15374796970963134483
	vdev_tree:
		type: 'raidz'
		id: 0
		guid: 6075442249147471512
		nparity: 1
		metaslab_array: 23
		metaslab_shift: 36
		ashift: 12
		asize: 11989471985664
		is_log: 0
		children[0]:
			type: 'disk'
			id: 0
			guid: 15374796970963134483
			path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7259a85-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 158
		children[1]:
			type: 'disk'
			id: 1
			guid: 6420862054695758756
			path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c7ac0491-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 157
		children[2]:
			type: 'disk'
			id: 2
			guid: 5596984570285208485
			path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c826480a-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 156
		children[3]:
			type: 'disk'
			id: 3
			guid: 6519031140053721195
			path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c8ac96f2-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			not_present: 1
			DTL: 155
		children[4]:
			type: 'disk'
			id: 4
			guid: 5085730408897018382
			path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c929c4be-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 154
		children[5]:
			type: 'disk'
			id: 5
			guid: 2261504104292084216
			path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			phys_path: '/dev/gptid/c9a6908b-0b0e-11e2-baee-001fd0af0df8'
			whole_disk: 0
			DTL: 99

I would expect to see guid: 5596984570285208485 in every label header in the output from ada3, instead of guid: 15374796970963134483.

Is it possible to fix this? For example by writing the expected values directly to the disk?
 
R

rs225

Guru
Joined
Jun 28, 2014
Messages
878
OK, I see your last post is working on the physical disks. So be very careful.

What do you get on zpool import with the physical disks?
Do you realize that you have 2 disks of one size, and 3 disks of another size?

Sector counts do not change by themselves. Your virtual copy disks are larger than the original physical disks, probably because virtualbox is rounding the size up.(I think it is possible to specify an exact byte size, which would get around that problem.) This would mean the secondary GPT is potentially lost on the copy, since it is stored in the last sector of the disk.

On your virtual disks (based on the partition table you posted, but one could obviously be wrong), you have 4 disks of one size, and 1 disk of another size. I think you might have a confusion between which disk is which on the physical vs. virtual.
 
Last edited:
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
rs225 said:
OK, I see your last post is working on the physical disks. So be very careful.

What do you get on zpool import with the physical disks?
Do you realize that you have 2 disks of one size, and 3 disks of another size?

Sector counts do not change by themselves. Your virtual copy disks are larger than the original physical disks, probably because virtualbox is rounding the size up.(I think it is possible to specify an exact byte size, which would get around that problem.) This would mean the secondary GPT is potentially lost on the copy, since it is stored in the last sector of the disk.

On your virtual disks (based on the partition table you posted, but one could obviously be wrong), you have 4 disks of one size, and 1 disk of another size. I think you might have a confusion between which disk is which on the physical vs. virtual.
Click to expand...
I have been working on both physical and virtual disks, and I try to be careful. On the upside, I have uploaded the virtual disks to the cloud, so in essence I have unlimited backups of these (but it would take a couple of weeks to download a copy of all the drives).

I just realized that you are correct regarding the disk sizes. I did a compare of the first 500mb of each second partition on the virtual disks, so it seems I have been confused at some point when taking backups. If I am correct, I have two copies of one physical disk and none of another. Which means I am not doing anything before I have found which disk I am missing a backup of, and taken a backup of this disk.

Thanks for helping! I will take baby steps from now on, and even though I am stupid, I will try to stop the obvious stupidity. :)
 
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
Hi again,
now I finally seem to have some luck. I am now sure I have virtual backups of all disks, so I started up my virtual FreeNAS and ran zpool import:
Code:
[root@freenas] ~# zpool import
   pool: RaidZ
     id: 13844435100348741427
  state: FAULTED
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
config:

        RaidZ                    FAULTED  corrupted data
          raidz1-0               DEGRADED
            ada5p2               ONLINE
            ada1p2               ONLINE
            ada4p2               ONLINE
            6519031140053721195  UNAVAIL  cannot open
            ada3p2               ONLINE
            ada2p2               ONLINE

Then I run zpool import RaidZ:
Code:
[root@freenas] ~# zpool import RaidZ
cannot import 'RaidZ': pool may be in use from other system, it was last accessed by debian (hostid: 0x7f0101) on Tue Dec  6 12:22:08 2016
use '-f' to import anyway

Leading to zpool import -f RaidZ:
Code:
[root@freenas] ~# zpool import -f RaidZ
cannot import 'RaidZ': I/O error
        Recovery is possible, but will result in some data loss.
        Returning the pool to its state as of Sun Aug 21 12:35:03 2016
        should correct the problem.  Approximately 154127 minutes of data
        must be discarded, irreversibly.  After rewind, several
        persistent user-data errors will remain.  Recovery can be attempted
        by executing 'zpool import -F RaidZ'.  A scrub of the pool
        is strongly recommended after recovery.

Then, when I run zpool import -fF RaidZ, the screen is spammed with a lot of text before the FreeNAS reboots.

Do you have any tips on how to proceed from here?
 
R

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
Robert Trevellyan said:
That's over 3 months! It seems unlikely to be successful.
Click to expand...
I know, but the disks haven't been in use for over a year, so it doesn't matter if I lose the last three months. :)
Robert Trevellyan said:
For some users, adding RAM has helped, but in this case it looks like a lost cause.
:(
Click to expand...
Hmm, okay :/ I'll try expanding a bit more, but I will try to prepare myself for the worst :(
 
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
Ok, so I have expanded to 18GB RAM on my virtual FreeNAS installation before retrying, and the result is almost the same. zpool import on my physical disks gives:
Code:
[root@freenas] ~# zpool import
   pool: RaidZ
     id: 13844435100348741427
  state: DEGRADED
 status: The pool was last accessed by another system.
 action: The pool can be imported despite missing or damaged devices.  The
        fault tolerance of the pool may be compromised if imported.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

        RaidZ                    DEGRADED
          raidz1-0               DEGRADED
            ada3p2               ONLINE
            ada2p2               ONLINE
            ada5p2               ONLINE
            6519031140053721195  UNAVAIL  cannot open
            ada4p2               ONLINE
            ada1p2               ONLINE

While the same command on my virtual disks returns:
Code:
[root@freenas] ~# zpool import
   pool: RaidZ
     id: 13844435100348741427
  state: FAULTED
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
config:

        RaidZ                    FAULTED  corrupted data
          raidz1-0               DEGRADED
            ada5p2               ONLINE
            ada1p2               ONLINE
            ada4p2               ONLINE
            6519031140053721195  UNAVAIL  cannot open
            ada3p2               ONLINE
            ada2p2               ONLINE

Ergo: The virtual disks have corrupted data, while the physical disks doesn't seem to have.
I can run zpool import -f RaidZ to try and import the degraded pool on my physical disks and zpool import -fF RaidZ on my virtual disks, and both times it returns a wall of text before a reboot. The same happens when I try to import using GUI (as my pool is now detected in the GUI).

Starting to realize my changes are gone...

Thanks for helping!
 
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
One small update before I go to bed:
I successfully got to see the content of the pool by running zpool import -f -o readonly=on RaidZ! I haven't yet tried to copy the files, but I ran du -hs to see file sizes, and then it reported lots of file errors before spamming the output and restarting like before.

I don't know if this means I should get my hopes up...
 
M

morra

Dabbler
Joined
Jan 17, 2017
Messages
12
zpool import -f -o readonly=on RaidZ did the trick! I'm able to copy a lot of files, but not all of them. I have noticed three different errors.
  • I have some backups of old Windows installations, and these are especially causing problems. If I try to copy, or even ls some folders, the kernel panics and the virtual FreeNAS reboots like when I tried to mount the pool with read/write access.
  • Another problem is during copy, I get some error like "unexpected error 122".
  • Also I notice that lots of files are missing (3TB+ or so). They just seem to be gone, as there is no record of them at all.
But except for these errors, I am able to extract some of the data from the disks.
 
Status
Not open for further replies.

Similar threads

D
  • Locked
unable to remove old faulted pool
Replies
7
Views
5K
xMx
X
R
  • Locked
Zpool offline drive wont online/replace
Replies
9
Views
8K
ripkurrle
R
H
  • Locked
Probleme nach Ausfall Festplatte
Replies
14
Views
3K
hormy
H
stangri
  • Locked
Recovering pool which crashes OS
Replies
19
Views
3K
stangri
stangri
A
  • Locked
Samsung HD204UI (2TB) disks causing headache. (Write speed > read speed)
Replies
19
Views
10K
joeschmuck
joeschmuck
Top