Drive Crashed - thoughts on seeing what survived pool

[smith000]

Hello,

Long time lurker, but I just had a disaster. I had a drive die without warning. It was part of a 2 disk pool. One of the drives is still functioning, but the other one is a paperweight now.

1. How do I tell if I set it up that 2 disk volume as a mirror? I shuffled data around and ended up with a single disk volume of 1TB and another volume of what I thought was 3TB - leading me to believe it was mirrored since it was on 2 disks of 3TB each. I don't recall expanding the volume to 6TB. In fact the whole reason I bought the new singe 1TB drive was to get the 3TB volume mirrored.

2. If it is a mirror, how do I turn it into a single disk volume to access the data?

3. If it is a mirror, what do I do when I receive the new 3TB drive from amazon on Friday to bring it online and restore the volume to functioning?

4. If it's not a mirror, can I recover anything at all? The data was just media that I can replace. I had a ton of jail config, VPN, privoxy, etc nicely configured and that's just gonna be a nightmare to recreate.

So far all I have done is detach the volume and remove the cratered drive in an effort to see if it was repairable. And when I say cratered, I mean it makes 2 clunks and stops spinning. I have tried all sorts of recovery mechanisms on it (the command line SeaChest tools) and it just laughs at me. Angrily laughs.

The "missing" volume is called "media". I could still see it when I ran a zpool command and it said that there were missing devices but the other drive was still online. I can't force it to reattach itself and the GUI "Import Volume" wizard doesn't see it hanging out there.

Yes, I know I should have paid more attention to how it was setup. I've kicked myself far more effectively than anyone here could, but feel free to call me dumb along with your suggestions.

Thanks,

Josh

 

[smith000]

I didn't post hardware info, mainly because this is more software config, but reading the rules:

Build FreeNAS-11.1-U6
Platform Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz
Memory 24424MBS
Network - <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet>
Drives (working)
umass0: <Kingston DataTraveler 3.0, class 0/0, rev 2.10/1.00, addr 3> on usbus0
umass0: SCSI over Bulk-Only; quirks = 0xc100
umass0:5:0: Attached to scbus5
ada0 at ata2 bus 0 scbus0 target 0 lun 0
ada0: <ST1000DM010-2EP102 CC43> ATA8-ACS SATA 3.x device
da0 at umass-sim0 bus 0 scbus5 target 0 lun 0
ada0: Serial Number Z9AMG6DS
ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada0: 953869MB (1953525168 512 byte sectors)
ada0: quirks=0x1<4K>
ada1 at ata2 bus 0 scbus0 target 1 lun 0
da0: <Kingston DataTraveler 3.0 PMAP> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 00190F0C02B2BE5099657B6F
da0: 40.000MB/s transfersada1: <ST3000DM001-1ER166 CC25> ACS-2 ATA SATA 3.x devi
ce
ada1: Serial Number Z5017FK3
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: 2861588MB (5860533168 512 byte sectors)
ada1: quirks=0x1<4K>

Mainboard -
Base Board Information
Manufacturer: Gigabyte Technology Co., Ltd.
Product Name: Z68MA-D2H-B3

 

[smith000]

If this helps:

Code:

root@freenas:~ # zpool import
   pool: media
	 id: 7476341166463039865
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
		devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
 config:

		media										 UNAVAIL  missing device
		  gptid/5f561821-f4fd-11e4-98c4-50e549e445e4  ONLINE

		Additional devices are known to be part of this pool, though their
		exact configuration cannot be determined.
root@freenas:~ # zdb -e media

Configuration for import:
		vdev_children: 2
		version: 5000
		pool_guid: 7476341166463039865
		name: 'media'
		state: 0
		hostid: 4082203137
		hostname: ''
		vdev_tree:
			type: 'root'
			id: 0
			guid: 7476341166463039865
			children[0]:
				type: 'disk'
				id: 0
				guid: 2987597981562062239
				whole_disk: 1
				metaslab_array: 35
				metaslab_shift: 34
				ashift: 12
				asize: 2998440558592
				is_log: 0
				create_txg: 4
				path: '/dev/gptid/5f561821-f4fd-11e4-98c4-50e549e445e4'
			children[1]:
				type: 'missing'
				id: 1
				guid: 0
zdb: can't open 'media': No such file or directory

ZFS_DBGMSG(zdb):
root@freenas:~ #

 

[joeschmuck]

So your hard drive has failed in a physical maner, the two cluncks you heard are the heads trying to load and they fail. So the drive is gone at this point. If the data were valuable then you might be able to have a repair service fix it but there is a huge cost for work like this. Were you running routine SMART Short and Long test? These tests typically catch these problems before they occur and let you knw so you can take action before full failure.

Odds are your pool was not a mirror, had it been a mirror then you would have access to your data because the other hard drive is working. Post the output of zpool status and I suspect it will show you that you are running a RAIDZ Stripe configuration not mirror.

 

[smith000]

joeschmuck:

zpool status doesn't show the pool - it's already been detached is the reason I assume why.

 

[joeschmuck]

joeschmuck:

zpool status doesn't show the pool - it's already been detached is the reason I assume why.

Ah, I guess I skipped the part that you removed the pool, I thought you said you detached the drive which is not the same thing.

 

[smith000]

That disk obviously knows it's part of a pool with more disks than are available.

I know the pool is toast. Is there no way to mount the data partition as part of a new pool so I can see what data is still there? Like I said, most of the data is replaceable except for all the jail setup and configuration. It looks like the drive I still have operational is the one I filled first.

Seems to me there has to be a way to tell that drive that it's partitions are now part of a new pool or mount them somewhere and copy off what i need.

 

[droeders]

If this helps:

Code:

root@freenas:~ # zpool import
   pool: media
	 id: 7476341166463039865
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
		devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
 config:

		media										 UNAVAIL  missing device
		  gptid/5f561821-f4fd-11e4-98c4-50e549e445e4  ONLINE

		Additional devices are known to be part of this pool, though their
		exact configuration cannot be determined.

It looks to me like you created a striped VDEV for the pool.

I hate to say it, but if you can't get that drive back, all your data is gone from that pool.

 

[Arwen]

@smith000, from what I can tell, I agree with @droeders, your pool was 2 disks striped. This is not recommended as either disk having a total failure will cause all data to be un-available. And that is what seems to have happened, one disk is totally un-available, (or dead). If you can't get that un-available disk back to functioning, time to restore from backups.

We highly recommend reading the ZFS guide for noobs guide, (I don't have a link handy).

Basically, if you care enough about your data to use ZFS, (and FreeNAS), then;

• Use some form of reliability, (Mirroring, RAID-Zx, or the soon to be released RAID-Dx)
• Monitor your disks and server
• Run scrubs
• Perform backups as needed

There are MANY use cases that don't need redundancy. Even your media is an example. Some people don't bother with Mirroring or RAID-Zx with media because they have excellent backups. Or a redundant server with 100% copy. Or the ability to re-create the media.

 

[joeschmuck]

Yup, I agree too that you likely had a stripe in order to gain more storage over two drives mirrored.

 

[Ericloewe]

Additional devices are known to be part of this pool, though their exact configuration cannot be determined.

That message indicated that vdevs are missing. In 11.2, you can actually import such pools and possibly get to some data - but only if you had copies=2 or =3 set, otherwise you only get metadata back.

 

[joeschmuck]

That message indicated that vdevs are missing. In 11.2, you can actually import such pools and possibly get to some data - but only if you had copies=2 or =3 set, otherwise you only get metadata back.

While your statement may be true, I don't see this being a typical setup for 99.9% of the folks using FreeNAS. I would expect the pool would have needed to be created via CLI to create a pool of this nature. But as it does provide completeness of an answer I guess it's okay. Had this OP manually created this pool like this, I would have expected them to know about it.