ZFS Volume state is unknown

[BRIT]

• FreeNAS-9.3-STABLE-201509160044 on a Dell 580 with 32Gb RAM.
  • FreeNAS is installed on a 16Gb USB flash drive.
• The 580 has a USB3 card in it, to which an 8 bay ProBox is attached.
  • 7 of the bays have Seagate 3Tb drives installed in them

Things have been working just fine for well over a year. This morning I come in and notice that my CIFS share isn't present on the network and a couple of my VMs aren't running. Upon inspection of the FreeNAS system I see that the alert is telling me that my two ZFS volumes are in an "unknown" state. I was out of the office yesterday and have no idea if there was any power outages etc.

On further inspection it appears that all of the drives that are present in the ProBox are no longer there. Upon rebooting FreeNAS, I catch a glimpse of a log stating that the two pools that I had created (ZFS) couldn't be imported and that I should delete and recreate the pool from a backup source.

I'm hoping that the USB3 card has blown, perhaps the PCI slot on the motherboard that it's plugged into perhaps? The fact that all 7 drives have miraculously disappeared leads me to think that this isn't a catastrophic event, but more of a annoyance of a failed piece of hardware. However, I'm unsure how one should proceed in this vain. I can mirror the hardware setup (as near as possible) with a new USB card in a secondary system, but how does one go ahead and use an existing ZFS array of disks in a completely new FreeNAS installation.

Should there be files that I can use from the existing (failed) system?

 

[danb35]

how does one go ahead and use an existing ZFS array of disks in a completely new FreeNAS installation.

That's easy, just use the Import Volume button in the web GUI. But your choice of hardware configuration is disastrously bad. USB drives for a ZFS pool are not remotely a safe setup.

 

[m0nkey_]

The 580 has a USB3 card in it, to which an 8 bay ProBox is attached.

Ugh, you were using USB. Also, you don't mention how the pool was configured. Hate to tell you this, but you've probably lost your pool.

If you weren't using any of the fancy RAID or JBOD features on that external box, you might be able to connect the drives to the system using SATA (for the love of god, not eSATA) and mount the pool.

 

[BRIT]

Ugh, you were using USB. Also, you don't mention how the pool was configured. Hate to tell you this, but you've probably lost your pool.

If you weren't using any of the fancy RAID or JBOD features on that external box, you might be able to connect the drives to the system using SATA (for the love of god, not eSATA) and mount the pool.

I'd read that one shouldn't use external RAID devices, but let ZFS handle everything itself in a native environment, so that's what I went with. Also, the pools were set up with ZFS2-0, if that's of any consequence?

Do most people then find a hardware shell that 8 drives can plug directly into the motherboard then, and not use any USB3.0 external devices?

 

[joeschmuck]

Do most people then find a hardware shell that 8 drives can plug directly into the motherboard then, and not use any USB3.0 external devices?

Yes, or to a controller card (HBA).

 

[danb35]

Nobody here is going to recommend USB-connected drives for anything more than temporary storage (and, in most cases, not even that). SATA or SAS connections are what's usually recommended, though I suppose SCSI would be OK as well.

 

[danb35]

8 drives can plug directly into the motherboard

It's not a matter of the drives plugging directly into the motherboard--HBAs and other drive controllers are (or can be) just fine. But USB is an absolute no-go.

 

[BRIT]

Understood, understood. Thank you both for your input.

I'm really looking for a small form factor system that can run as a FreeNAS system. I see a lot of systems that are great if I had a rack or location where "noisy" servers would reside without bothering anyone, but in a single office environment I'm really looking for something that is quiet and able to handle about 5 - 6Tb of data. I'm ok with building something up (mobo, chassis, RAM etc) and I understand that a lot of people go the SuperMicro route, for obvious reasons. TBH, I've not looked around for a year or so as my existing set up (albeit flawed) has been working just fine. There doesn't appear to be much choice (however) in small form factor systems that can handle a solid 6Tb of data, unless I'm seriously mistaken?

 

[pirateghost]

Understood, understood. Thank you both for your input.

I'm really looking for a small form factor system that can run as a FreeNAS system. I see a lot of systems that are great if I had a rack or location where "noisy" servers would reside without bothering anyone, but in a single office environment I'm really looking for something that is quiet and able to handle about 5 - 6Tb of data. I'm ok with building something up (mobo, chassis, RAM etc) and I understand that a lot of people go the SuperMicro route, for obvious reasons. TBH, I've not looked around for a year or so as my existing set up (albeit flawed) has been working just fine. There doesn't appear to be much choice (however) in small form factor systems that can handle a solid 6Tb of data, unless I'm seriously mistaken?

https://www.ixsystems.com/freenas-mini/

 

[danb35]

https://www.ixsystems.com/freenas-mini/

Or a Dell PowerEdge T20 with an extra 8 GB stick of RAM, and 4 x 4 TB drives (either in striped mirrors, or RAIDZ2). Might be bigger than the mini, but probably cheaper.

 

[SweetAndLow]

Node 304 with 6TB drives in raidz2 gives you 24TB storage. There are several quite small form factor solutions that can handle 5-6TB, heck almost any itx case could handle that small amount of storage.

 

[Ericloewe]

Node 304 with 6TB drives in raidz2 gives you 24TB storage. There are several quite small form factor solutions that can handle 5-6TB, heck almost any itx case could handle that small amount of storage.

Well, 24"TB", minus overhead, minus 20% free, ...

OP, here's a handy calculator to give you ballpark figures: https://forums.freenas.org/index.php?threads/zfs-raid-size-and-reliability-calculator.28191/

 

[BRIT]

To everyone, many many thanks - certainly a lot here to think about...

One final thought on my failed pools : would it be common for all 7 drives to fail immediately and all at the same time? In the hope that perhaps this is a hardware issue (USB3 card, blown port on either the mobo or perhaps the ProBox itself), m0nkey had mentioned that if I plug the drives in directly using SATA connections to a system loaded with a fresh install of FreeNAS, there's a chance that the data might still be available. Is there any chance that that could work? If I were to try it, can I take it that the drives can be plugged into any SATA port on the motherboard in the hope that they'd spin up with identifiers that FreeNAS would be able to read and rebuild the pool from?

 

[SweetAndLow]

Worth a try, also try replacing your usb stuff to see if that works. You are in scary waters though.

 

[BRIT]

Node 304 with 6TB drives in raidz2 gives you 24TB storage. There are several quite small form factor solutions that can handle 5-6TB, heck almost any itx case could handle that small amount of storage.

I like the size of that Node 304 - the idea there would be to laden it up with 3½" drives, place FreeNAS on a good USB drive and then put a couple of NIC cards in the chassis in order to allow the network to connect to it, correct?

 

[pirateghost]

I like the size of that Node 304 - the idea there would be to laden it up with 3½" drives, place FreeNAS on a good USB drive and then put a couple of NIC cards in the chassis in order to allow the network to connect to it, correct?

The 304 doesn't give you much room to expand if you wanted and you only need one NIC.

 

[BRIT]

By the way, this was the outcome of a "zpool import" I ran a few hours ago... to me, it appears that my first pool should still be available?

[root@freenas ~]# zpool import
pool: CIFS_A
id: 8121838758098070984
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

CIFS_A ONLINE
raidz2-0 ONLINE
gptid/e1a62252-f5e0-11e4-a479-00151795dfaa ONLINE
gptid/e26d0b5d-f5e0-11e4-a479-00151795dfaa ONLINE
gptid/e3103c23-f5e0-11e4-a479-00151795dfaa ONLINE
gptid/e3d40fec-f5e0-11e4-a479-00151795dfaa ONLINE

pool: VMware_vol1
id: 6121196609444076099
state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://illumos.org/msg/ZFS-8000-2Q
config:

VMware_vol1 DEGRADED
raidz1-0 DEGRADED
gptid/2360c7e7-f5cd-11e4-a479-00151795dfaa ONLINE
gptid/240db837-f5cd-11e4-a479-00151795dfaa ONLINE
5254766704247237793 UNAVAIL cannot open
[root@freenas ~]#

 

[SweetAndLow]

I like the size of that Node 304 - the idea there would be to laden it up with 3½" drives, place FreeNAS on a good USB drive and then put a couple of NIC cards in the chassis in order to allow the network to connect to it, correct?

Your motherboard has a nic so you don't need to add any.

 

[Robert Trevellyan]

Seagate 3Tb

Let me guess - ST3000DM001?

it appears that my first pool should still be available?

It appears that both your pools should be importable. Looks like you've lost one drive from the RAIDZ1 pool, so that pool now has no redundancy. Top priority would be to get that pool mounted and replace the faulty drive.

Best bet is probably to put the drives in a system that can accommodate them without using the ProBox. That thing has terrible reviews.

 

[BRIT]

Would the best way of ascertaining which drives are associated to which pool, by using glabel -status? I've tried using smartctl and the only drive it sees is the USB boot drive, that being the one where FreeNAS is installed...
And using the GUI is a no go - it doesn't see any disks