Cannot mount ZFS pool due to "insufficient replicas"

[russellseymour]

My FreeNAS machine rebooted last night, I am not sure why but I am guessing it was a brief power cut due to storms.

The problem I have is that one of the ZFS pools will not mount, this is the output from "zpool import":

Code:

   pool: storage02
     id: 2588794960955536546
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

        storage02                                       UNAVAIL  insufficient replicas
          mirror-0                                      ONLINE
            gptid/49ff4c71-74f0-11e6-81ea-94de80678aa1  ONLINE
            gptid/4b1cf3b4-74f0-11e6-81ea-94de80678aa1  ONLINE
          16534644111877360144                          UNAVAIL  cannot open

As can be seen the disks are ONLINE, but it cannot find the replicas it needs.

My other two pools are working without issue:

Code:

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:03:59 with 0 errors on Thu Jun 18 03:48:59 2020
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2     ONLINE       0     0     0

errors: No known data errors

  pool: storage01
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 05:29:52 with 0 errors on Sun May 17 05:29:54 2020
config:

        NAME                                            STATE     READ WRITE CKSUM
        storage01                                       ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/74cd5025-8c1e-11e2-9e97-001d7de623a5  ONLINE       0     0     0
            gptid/77e72d51-8c1e-11e2-9e97-001d7de623a5  ONLINE       0     0     0
            gptid/7affddfc-8c1e-11e2-9e97-001d7de623a5  ONLINE       0     0     0
            gptid/7e1ba968-8c1e-11e2-9e97-001d7de623a5  ONLINE       0     0     0

errors: No known data errors

Is there anyway that I can make FreeNAS finds its replicas so I can mount the pool?
If this cannot be done can I mount one of the disks so I can at least take an up to date backup of the data on there?

I am more than happy to provide the 'Information' about my FreeNAS setup but I cannot find in the GUI where this info is :(.

The machine I am running on has 16Gb RAM and the CPU is a 'Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz'

Happy to provide more information if needed. Thanks for any help.

 

[russellseymour]

My apologies I forgot to state that the version of FreeNAS I am running is 11.2-U7

 

[Samuel Tai]

I'm sorry, but it appears you may have made a mistake in constructing the storage02 pool. You may have intended to create a mirror of 3 disks, but it's actually a stripe of a 2-way mirror and a single disk. As the single disk is unavailable, the stripe can't be reconstituted.

Check if the unavailable disk's power and data connections are still OK. If the disk no longer spins up, then your pool is lost.

 

[russellseymour]

Thanks for the reply, however I am now more confused.

The pool storage02 was set across 2 disks in a mirror, indeed the `zpool import` output shows to disks with different Partition IDs. (Unless I am misunderstanding the `gptid` being different disks)

Code:

        storage02                                       UNAVAIL  insufficient replicas
          mirror-0                                      ONLINE
            gptid/49ff4c71-74f0-11e6-81ea-94de80678aa1  ONLINE
            gptid/4b1cf3b4-74f0-11e6-81ea-94de80678aa1  ONLINE
          16534644111877360144                          UNAVAIL  cannot open

According to the FreeNAS UI i have two disks that are not in any pool, which used to be in storage02.

Ad I am getting size the disk registering in FreeNAS I am confident that they are being seen.

So I created a mirror with 2 disks and the import states that the disks are online. Do I need 3 disks in a mirror for FreeNAS? If so, I do not understand that because a mirror is 2 disks with one disk mirroring the other one.

 

[Samuel Tai]

Hmm. Something funny's going on. You've already provided the zpool status storage02 output. Can you provide the gpart list adax output for the relevant disks? You'll want to match up the rawuuids from the gpart output to see what's attached to the storage02 pool.

 

[russellseymour]

So using the gpart list adax command for the disks I can see that the pool is pointing to p2 on disks ada0 and ada1.

So for ada0 I get the following:

Code:

2. Name: ada0p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   efimedia: HD(2,GPT,4b1cf3b4-74f0-11e6-81ea-94de80678aa1,0x400080,0xe8a08808)
   rawuuid: 4b1cf3b4-74f0-11e6-81ea-94de80678aa1
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432

And for ada1 I get:

Code:

2. Name: ada1p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   efimedia: HD(2,GPT,49ff4c71-74f0-11e6-81ea-94de80678aa1,0x400080,0xe8a08808)
   rawuuid: 49ff4c71-74f0-11e6-81ea-94de80678aa1
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432

As can be seen the rawuuid for the partition in each disk is the one that is seen in the storage02 pool information.

I am not opposed to recreating this if there is a way I can get the data off it. I do have backups of my media but not of the Minecraft world that my children have been building so it would be great if I can get that in someway - if this cannot be fixed.

 

[HoneyBadger]

What does the output of zpool history storage02 show?

Did you at some point attempt to attach an SSD for "cache" to this pool perhaps?

 

[russellseymour]

When I run zpool history storage02 I get the response:

cannot open 'storage02': no such pool

The only way I have been able to see the pool information for storage02 is by trying to import the pool with zpool import

Nope I have not tried to attach anything to the pool. It has been sitting there for months (without issue or any modification on my part) handling some of my media and a share for the Minecraft worlds. The problems only showed up when the box restarted last night.

 

[HoneyBadger]

Right, exported pool.

Try zdb -U /data/zfs/zpool.cache -eCC storage02 and paste the results in a CODE block here - I want to see if it thinks you have another top-level vdev in either the cachefile or MOS.

 

[russellseymour]

As requested, here is the output of zdb -U /data/zfs/zpool.cache -eCC storage02

Code:

Configuration for import:
        vdev_children: 2
        version: 5000
        pool_guid: 2588794960955536546
        name: 'storage02'
        state: 0
        hostid: 3362829740
        hostname: 'turtle-store-01.home.turtlesystems.co.uk'
        vdev_tree:
            type: 'root'
            id: 0
            guid: 2588794960955536546
            children[0]:
                type: 'mirror'
                id: 0
                guid: 14117786082381185466
                metaslab_array: 35
                metaslab_shift: 34
                ashift: 12
                asize: 1998246641664
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 4779388346123298445
                    whole_disk: 1
                    DTL: 208
                    create_txg: 4
                    path: '/dev/gptid/49ff4c71-74f0-11e6-81ea-94de80678aa1'
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 11247153312154708343
                    whole_disk: 1
                    DTL: 207
                    create_txg: 4
                    path: '/dev/gptid/4b1cf3b4-74f0-11e6-81ea-94de80678aa1'
            children[1]:
                type: 'missing'
                id: 1
                guid: 0
zdb: can't open 'storage02': No such file or directory

In this context what is an exported pool?

 

[Samuel Tai]

what is an exported pool?

Did you run zfs export storage02? You mentioned zfs import earlier. The zdb output indicates the pool somehow thinks it's no longer just a mirror of 2 drives, but a stripe of a 2-way mirror, and a 3rd disk.

 

[HoneyBadger]

An exported pool is simply one that isn't imported; this happened as a result of the reboot.

Code:

vdev_children: 2

The pool thinks it has a second child vdev, in this case the single device identified by GUID 16534644111877360144 - I'm assuming you've never had any more than six disks in this machine, so I'm at a loss as to why it would be thinking there's another drive missing here.

 

[russellseymour]

@Samuel Tai No I have not run any export at all. This is the result of an unplanned reboot on Wednesday evening.

@HoneyBadger you are correct in your assumption that there has never been more than 6 disks in the machine. The only indication I had that there was a problem was the morning after the reboot when my children said they could not connect to their Minecraft world. When I looked it was that this pool was not mounted.

I am assuming from what you are saying that this is a dead pool and the chances of recovering anything from it are small?

 

[HoneyBadger]

@russellseymour - Unfortunately it's looking that way. ZFS has somehow become convinced that your pool has a third device, and it was attached as a "stripe" at the top level; so it was holding some manner of data that wasn't replicated elsewhere.

If you run zdb -l /dev/adaX against each of the devices and try to grep out the GUID, you might be able to find a match for the 16534644111877360144 ID - perhaps it's on another partition, a loopback file on another device, something like that; I'm just at a loss to explain this. ZFS isn't known for grenading itself under a reboot (in fact, quite the opposite)

Can you run zdb -U /data/zfs/zpool.cache -eh storage02 - this should pull the history off your devices.

If you want the verbose output, zdb -U /data/zfs/zpool.cache -ehh storage02 will give you transaction groups, but you'll want to redirect the result to a text file as it'll be pretty massive to copy otherwise.

If there's no history of attaching/adding other devices, you can try zpool import -N -o readonly=on -f -R storage02 - this will try checking to see if you can open your pool in read-only mode by forcibly rolling back a few transactions. (It won't actually mount it though, the -N switch is the "test it out")

Beyond this we're starting to get into "picking out txg id's and trying hidden import options" which is arcane voodoo but might be your only hope.

 

[russellseymour]

@HoneyBadger Thanks for all your help. I think I am resigned to the fact I have lost the data. My children are a bit annoyed at having to redo everything, but they will recreate their worlds :).

I am surprised it has done what it has done, as I too thought that zfs was meant to be able to cope with the sort of thing.

I am not able to get any history for storage02 and I get the following output from readonly mount:

Code:

   pool: storage02
     id: 2588794960955536546
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
config:

        storage02                                       UNAVAIL  insufficient replicas
          mirror-0                                      ONLINE
            gptid/49ff4c71-74f0-11e6-81ea-94de80678aa1  ONLINE
            gptid/4b1cf3b4-74f0-11e6-81ea-94de80678aa1  ONLINE
          16534644111877360144                          UNAVAIL  cannot open

 

[HoneyBadger]

I'm really puzzled, honestly. If the drives themselves never changed, then I don't know why the cachefile would think it needs a third device.

You could try booting up a clean FreeNAS install with only the two storage02 members present, and try to import the volume again; that would guarantee there's no cachefile trying to muddy the waters.

 

[Samuel Tai]

You should also invest in a good UPS.

 

[russellseymour]

@HoneyBadger and @Samuel Tai I owe you both a massive apology.

I could not sleep last night so thought I would tackle the drives on my FreeNAS machine. So I shut the machine down and took the side off, whereupon I found 7 disks!!!!

4 are being used for storage01 and the other *3* are for storage02. You both said that there should have been three disks and that is what zfs was looking for. I was convinced otherwise because of the "mirror" and that I built the machine about 2 or 3 years ago.

I removed all the connections and put them back and started up the machine and storage02 appeared!!! I have been able to back up the data I needed.

I really appreciate the time and effort you both put in to help me with this problem. I am sorry that I did not give the full story.

If I may ask another question. I am using ZFS to give me resilience, but this instance has shown that I do not have that on this storage pool. I am assuming that I should be using raidz1 to provide the resilience I want, e.g. to be able to loose a disk. I am still a bit confused as to why a mirror would work like it has here.

 

[Samuel Tai]

A mirror would’ve provided the resiliency you needed, if it had been constructed properly. I assume you had intended to create a 3-way mirror, but inadvertently added the 3rd disk not to the mirror, but to the top-level of the pool, which told FreeNAS to create a stripe between the existing mirror and the 3rd disk.

Since you have the data backed up, you now have an opportunity to correct that error, and to rebuild the pool to your original intent, or as a RAIDZ1, as you suggest.

 

[danb35]

Since you have the data backed up, you now have an opportunity to correct that error, and to rebuild the pool to your original intent, or as a RAIDZ1, as you suggest.

Or, alternatively, add a fourth disk as a mirror to the third--though the devs still haven't gotten around to incorporating that in the GUI, so it would need to be done from the CLI.