Uhoh, I think I'm going down the rabbit hole...

[mbro]

Oh dear! Help!

My Truenas 12 has been up a good while, zraid1 with 4 disks. I was experiencing a disk giving me some errors, attempting a replacement, and I think I'm going down the wrong path. The pool was healthy when all this started.

Shut down truenas, replaced the disk exhibiting errors. I verified the serial number of the disk that needed replacement, and triple checked. New disk is equivalent sized 4TB. Affected disk was ada2.
With the new disk in place, started up freenas, bios found the new disk (healthy), into the UI. Storage -> Pools -> Status -> ADA2 -> replace. Error. A bit of googling indicated the new disk has a new gptid, but did not assume ada2 like the old one. So I did some reading (not enough, clearly), followed advice to create some new partitions following another article, then use zpool replace pool1 <id of old disk> gptid/<gptid of /dev/ada2p2>. That sequence also errored. In my infinite wisdom, I thought I need to add the new gptid of ada2p2 to the pool. I did a zpool attach pool1 gptid/<gpti of /dev/ad2p2>. That attached the disk to the pool, but not the zraid1 volume. Ugh! Then I tried to remove the newly added volume, which told me it was not able to because it was in use or something (going by memory now because its been a bit trying and a long night).

Then I had a good idea - I'll shut it down, disconnect the newly added disk, bring it back up again. Now the volume shows UNAVAIL. Uhoh! The original 3 disks are still there, but the pool is offline. I shut it down again, plugged the disk back in, brought it back up again, the pool is still offline.

I tried zpool import, but it shows me this:

Code:

oot@nas:~ # zpool import
   pool: pool1
     id: 376450636264426360
  state: UNAVAIL
status: One or more devices are faulted.
action: The pool cannot be imported due to damaged devices or data.
config:

    pool1                                           UNAVAIL  insufficient replicas
      raidz1-0                                      DEGRADED
        gptid/9e01d44b-2b1d-11e8-bac4-4061868fdd8b  ONLINE
        ada1                                        ONLINE
        7971962589655721939                         UNAVAIL  cannot open
        gptid/a0ee5a82-7318-11e6-a72c-4061868fdd8b  ONLINE
      gptid/3596ab80-fd78-11eb-8044-4061868fdd8b    FAULTED  corrupted data
root@nas:~ #

The indentation (above) is important. The last vol is not part of the raidz1-0 volume - note that it shows as degraded (but I believe its still OK).
What shows as "FAULTED" - corrupted data - thats the gptid of /dev/ada2p2 which I created. The pool is not online currently, and not part of the raidz1-0 volume. I need to do the following, I think:
Bring the pool online? How?
Remove gptid/3596....dd8b (the faulted/corrupted but empty partition on ada2) from the pool. How do I do that if the pool is not online and won't come online? Yikes!
I suppose first I need to bring the pool online.

I do still have the disk that I removed - it was functional, but just starting to exhibit errors. I have this sinking feeling deep in my stomache... Please help!

 

[mbro]

Ok, some additional info. I decided that since the original disk that I removed hasn't exactly failed (was exhibiting errors, but still working), that I would shut down truenas, re-install the original disk, and bring it back up again. This nets me the current status:

Code:

root@nas:~ # zpool import
   pool: pool1
     id: 376450636264426360
  state: UNAVAIL
status: One or more devices are faulted.
 action: The pool cannot be imported due to damaged devices or data.
 config:

    pool1                                           UNAVAIL  insufficient replicas
      raidz1-0                                      ONLINE
        gptid/9e01d44b-2b1d-11e8-bac4-4061868fdd8b  ONLINE
        ada1                                        ONLINE
        7971962589655721939                         ONLINE
        gptid/a0ee5a82-7318-11e6-a72c-4061868fdd8b  ONLINE
      gptid/3596ab80-fd78-11eb-8044-4061868fdd8b    FAULTED  corrupted data
root@nas:~ # zpool offline pool1 gptid/3596ab80-fd78-11eb-8044-4061868fdd8b
cannot open 'pool1': no such pool
root@nas:~ # 

Ok, so now the big question - how do I remove the gptid/3596...dd8b vdev - its not actually present in the system, contains no data, and doesn't actually relate to a physical disk or a partition? But the trick is - the pool is offline, and until that reference has been removed, I can't bring it online.

 

[sretalla]

Ok, so now the big question - how do I remove the gptid/3596...dd8b vdev - its not actually present in the system, contains no data, and doesn't actually relate to a physical disk or a partition? But the trick is - the pool is offline, and until that reference has been removed, I can't bring it online.

What you have there is a Pool made of 2 VDEVs:

raidz1-0 (which is RAIDZ1, containing 4 disks)

gptid/3596ab80-fd78-11eb-8044-4061868fdd8b (a single disk... stripe).

That was a disaster waiting to happen... and now it has.

So now you're in the position of needing to rebuild the pool since you can't remove a top-level VDEV from a pool containing a RAIDZ VDEV.

Because a pool will only come online when all VDEVs are available and your single disk is not capable of going online, there's nothing you can do other than rebuild it and restore data from your backup.

 

[danb35]

So now you're in the position of needing to rebuild the pool since you can't remove a top-level VDEV from a pool containing a RAIDZ VDEV.

...and even if you could, vdev removal only works on healthy pools.

Ok, so now the big question - how do I remove the gptid/3596...dd8b vdev - its not actually present in the system, contains no data

You can't, as noted above. The only way to bring the pool online is to repair the data on that device (yes, it does contain at least some data). For that, though I don't think it will work, try zpool import -F pool1. And then, even if you are able to do that, you'll need you destroy the pool, rebuild it (in the GUI; stop messing around at the CLI), and restore your data--there's no way to recover from the way you've configured the pool.

 

[mbro]

That news fills me with dread! Is there a possibility of adding a device (the new disk) in addition to the 4 healthy ones, so that gptid3596...dd8b vdev becomes "healthy" and has something present behind it, enough so to at least bring the pool online?

The newly added vdev thats causing all of this was added and has the gptid of the "new" replacement disk that I purchased. Can I shuck it into a USB enclosure and add it to my nas, then somehow connect that physical disk to this vdev that thinks its device is corrupt? Any help or suggestions is appreciated massively!

 

[danb35]

If your pool were healthy, yes, it would be possible to add a mirror to gptid/3596ab80-fd78-11eb-8044-4061868fdd8b. It wouldn't be an ideal arrangement by any means, but it would at least give some redundancy for that disk. But the pool isn't healthy, and that disk is part of the problem. I repeat: if you're unable to bring that disk online, all data on that pool is gone.

 

[mbro]

I'm not interested in adding a mirror to that gptid, I just want to make it not be corrupt so I can bring the pool online, and make preparations to move any data that I haven't backed up. I would bring the new disk online to satisfy the need, add a mirrored disk if needed. I've really shot myself in the foot, please help stop the bleeding! Appreciate the feedback, Danb35.

 

[Heracles]

I just want to make it not be corrupt so I can bring the pool online

Too late for that...

This is like the anti-vaccine people that wait to get sick themselves and then begging for the vaccine : TOO LATE. Once sick, the vaccine can not do any good anymore.

So here, now that your single point of failure did failed, it is too late and the entire structure failed.

 

[mbro]

Granted, this is bad... I do realize the CLI things I did were not the right things to do (I blame a beer, and that it was late). My fault. Agreed, Heracles - its a bit like asking for a vaccine after you're sick...

Questions: I can put the disk that equated to the failed GPTID into a USB enclosure and attach it (I'm out of SATA interfaces)? Will the GPTID of that disk or its partitions change if its USB attached instead of via SATA interface? Is this safe to try? If it changes, can I "update" its GPTID to equate to whats shown in the pool, effectively "connecting" the physical disk with the GPTID that the pool is looking for or expecting? If that was to work, and the disk was to be "seen" by the pool, would that change its status? I'm a bit desperate.

 

[Heracles]

Hey @mbro,

Questions: I can.....

Unfortunately, nothing of that would be of any help. External USB storages work terrible under TrueNAS and ZFS. As to try to access that single drive from another system, again, it is of no use. Its filesystem, ZFS, will not make sense to most other systems and even if you put it in a system that knows about ZFS, by itself the content of that single drive is non-sensical. It will make sense only with the rest of its pool, just like in your actual TrueNAS system.

ZFS builds its pool from vDevs. For performance, it spreads the load to all of its vDev equally. It is up to the Vdev themselves to ensure redundancy because at pool level, all vDevs are always grouped without any redundancy. Should any vDev in a pool fails, the entire pool fails. Should you notice such a single point of failure in your pool BEFORE it fails, you can put a plaster on it and add redundancy in a Vdev where there was not any. But to add a redundant copy of something, you must first have an actual copy of that. In your case, you do not have copies anymore : the sole original is gone.

If you are ready to pay thousands of $$$ to an expert, you can try to have them recover the raw data from that failed drive and transfer it to another one. Such a process would be without any warranty and even if they recover 99% of the original, if the 1% missing is part of the critical ZFS hierarchy, your pool will stay offline.

Your best move now is to acknowledge "Lesson Learned" and do better next time:
Never compromise your pool with a single drive vDev
Never turn to CLI to achieve more than what you actually understand
Never consider that ANY server can be more than a single point of failure and that you always need backups (see my signature for details)

You are not the first one who tries to do backups right after loosing it all. Just be sure not to come back to this situation a second time and be more prepared for your next IT incident.

 

[mbro]

Well, I tried a few more things. I did some research into the ability to roll back volume transactions. There were some transactions that I rolled back, but the history didn't seem to go back far enough to prior to "the event" (my stupid move) that caused the problem with the array.

If you want to try rolling back transactions on your failed volume if you did something like I did, my suggestion would be: shut down your freenas, bring it up offline (or export it so it won't change any more), then follow this guidance:

Attempting to roll back zfs transactions on a disk to recover a destroyed ZFS filesystem

Here's what I was attempting (after putting it in recovery mode with /boot/loader.conf addition of:
vfs.zfs.recover=1
(and a restart)

Code:

root@nas:~ # glabel status
                                      Name  Status  Components
gptid/9e01d44b-2b1d-11e8-bac4-4061868fdd8b     N/A  ada0p2
gptid/0eee0fc0-ad94-11e5-9384-4061868fdd8b     N/A  ada2p2
gptid/a0ee5a82-7318-11e6-a72c-4061868fdd8b     N/A  ada3p2
gptid/7bd3a947-c545-11ea-b8a6-4061868fdd8b     N/A  da0p1
gptid/a0d70e26-7318-11e6-a72c-4061868fdd8b     N/A  ada3p1
gptid/0edb3a1c-ad94-11e5-9384-4061868fdd8b     N/A  ada2p1
gptid/9df8ec9f-2b1d-11e8-bac4-4061868fdd8b     N/A  ada0p1

root@nas:~ # zdb -ul /dev/gptid/9e01d44b-2b1d-11e8-bac4-4061868fdd8b
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'pool1'
    state: 0
    txg: 46939375
    pool_guid: 376450636264426360
    errata: 0
    hostid: 3813865623
    hostname: 'nas.local'
    top_guid: 16314225471850924137
    guid: 17580049385076698499
    vdev_children: 2
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 16314225471850924137
        nparity: 1
        metaslab_array: 23
        metaslab_shift: 32
        ashift: 9
        asize: 15994538950656
        is_log: 0
        children[0]:
            type: 'disk'
            id: 0
            guid: 17580049385076698499
            path: '/dev/gptid/9e01d44b-2b1d-11e8-bac4-4061868fdd8b'
            phys_path: 'id1,enc@n3061686369656d30/type@0/slot@1/elmdesc@Slot_00/p2'
            whole_disk: 1
            DTL: 4416
        children[1]:
            type: 'disk'
            id: 1
            guid: 16884205621880053317
            path: '/dev/ada1'
            phys_path: 'id1,enc@n3061686369656d30/type@0/slot@2/elmdesc@Slot_01'
            whole_disk: 1
            DTL: 4415
        children[2]:
            type: 'disk'
            id: 2
            guid: 7971962589655721939
            path: '/dev/gptid/0eee0fc0-ad94-11e5-9384-4061868fdd8b'
            phys_path: 'id1,enc@n3061686369656d30/type@0/slot@3/elmdesc@Slot_02/p2'
            whole_disk: 1
            not_present: 1
            DTL: 4414
            expansion_time: 1628998020
        children[3]:
            type: 'disk'
            id: 3
            guid: 1531281749025797575
            path: '/dev/gptid/a0ee5a82-7318-11e6-a72c-4061868fdd8b'
            phys_path: 'id1,enc@n3061686369656d30/type@0/slot@4/elmdesc@Slot_03/p2'
            whole_disk: 1
            DTL: 4413
            expansion_time: 1628997133
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3 
    Uberblock[0]
    magic = 0000000000bab10c
    version = 5000
    txg = 46939648
    guid_sum = 10968311033995811238
    timestamp = 1628999925 UTC = Sat Aug 14 21:58:45 2021
    mmp_magic = 00000000a11cea11
    mmp_delay = 0
    mmp_valid = 0
    checkpoint_txg = 0
        labels = 0 1 2 3 
    Uberblock[1]
    magic = 0000000000bab10c
    version = 5000
    txg = 46939649
    guid_sum = 10968311033995811238
    timestamp = 1628999930 UTC = Sat Aug 14 21:58:50 2021
    mmp_magic = 00000000a11cea11
    mmp_delay = 0
    mmp_valid = 0
    checkpoint_txg = 0
        labels = 0 1 2 3 
    Uberblock[2]
    magic = 0000000000bab10c
    version = 5000
    txg = 46939650
    guid_sum = 10968311033995811238
    timestamp = 1628999935 UTC = Sat Aug 14 21:58:55 2021
    mmp_magic = 00000000a11cea11
    mmp_delay = 0
    mmp_valid = 0
    checkpoint_txg = 0
        labels = 0 1 2 3 
    Uberblock[3]
    magic = 0000000000bab10c
    version = 5000
    txg = 46939651
    guid_sum = 10968311033995811238
    timestamp = 1628999940 UTC = Sat Aug 14 21:59:00 2021
    mmp_magic = 00000000a11cea11
    mmp_delay = 0
    mmp_valid = 0
    checkpoint_txg = 0
        labels = 0 1 2 3 


zpool import -T <txg_number> -F -o readonly=on pool1

Thanks for all your advice and feedback, assistance, thoughts, guidance, and pointing out the obvious - that I caused this problem... Live & learn.
Note to self: back up more often, and ensure I'm thinking with a sharp brain before doing any operations... And don't circumvent the GUI...

 

[mbro]

Brief update. With the help of an old buddy, I managed to solve this! Check it out!

Code:

root@nas:~ # sysctl vfs.zfs.max_missing_tvds=1
vfs.zfs.max_missing_tvds: 0 -> 1
root@nas:~ # zpool import -o readonly=on pool1
root@nas:~ # zpool status
  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:01:41 with 0 errors on Mon Aug  9 03:46:41 2021
config:

    NAME          STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      da0p2       ONLINE       0     0     0

errors: No known data errors

  pool: pool1
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
    repaired.
  scan: scrub repaired 217K in 21:23:11 with 0 errors on Fri Aug 13 22:23:13 2021
config:

    NAME                                            STATE     READ WRITE CKSUM
    pool1                                           DEGRADED     0     0     0
      raidz1-0                                      ONLINE       0     0     0
        gptid/9e01d44b-2b1d-11e8-bac4-4061868fdd8b  ONLINE       0     0     0  block size: 512B configured, 4096B native
        ada1                                        ONLINE       0     0     0  block size: 512B configured, 4096B native
        gptid/0eee0fc0-ad94-11e5-9384-4061868fdd8b  ONLINE       0     0     4  block size: 512B configured, 4096B native
        gptid/a0ee5a82-7318-11e6-a72c-4061868fdd8b  ONLINE       0     0     0  block size: 512B configured, 4096B native
      gptid/3596ab80-fd78-11eb-8044-4061868fdd8b    FAULTED      0     0     0  external device fault

errors: No known data errors
root@nas:~ # cd /mnt
root@nas:/mnt # ls
md_size
root@nas:/mnt # 
root@nas:/mnt # cd /mnt
root@nas:/mnt # ls
md_size
root@nas:/mnt # cd md_size
md_size: Not a directory.
root@nas:/mnt # ls
md_size
root@nas:/mnt # cd ..
root@nas:/ # ls
.cache        .rnd        compat        data        etc        media        pool1        root        tmp
.cshrc        bin        conf        dev        lib        mnt        proc        sbin        usr
.profile    boot        COPYRIGHT    entropy        libexec        net        rescue        storcli.log    var
root@nas:/ # zfs list
NAME                                                       USED  AVAIL     REFER  MOUNTPOINT
freenas-boot                                              3.18G  25.4G       23K  none
freenas-boot/ROOT                                         3.17G  25.4G       23K  none
freenas-boot/ROOT/11.3-U5                                  304K  25.4G     1.01G  /
freenas-boot/ROOT/FreeNAS-12.0-U2.1                       3.17G  25.4G     1.16G  /
freenas-boot/ROOT/Initial-Install                            1K  25.4G     1019M  legacy
freenas-boot/ROOT/default                                  277K  25.4G     1021M  legacy
pool1                                                     10.3T  3.72T      673G  /pool1
pool1/.system                                             5.69G  3.72T      773M  legacy
pool1/.system/configs-149940421d9b4df8af8e52297e068541     817M  3.72T      817M  legacy
pool1/.system/configs-646f8dae97d646cc8946ddeb0ca79d97    31.4K  3.72T     31.4K  legacy
pool1/.system/configs-ae1f6b9a44d94d15a639a40c1e91b328    22.6M  3.72T     22.6M  legacy
pool1/.system/cores                                       16.1M  1008M     16.1M  legacy
pool1/.system/rrd-149940421d9b4df8af8e52297e068541         227M  3.72T      227M  legacy
pool1/.system/rrd-646f8dae97d646cc8946ddeb0ca79d97         113M  3.72T      113M  legacy
pool1/.system/rrd-ae1f6b9a44d94d15a639a40c1e91b328         130M  3.72T      130M  legacy
pool1/.system/samba4                                      7.00M  3.72T     4.75M  legacy
pool1/.system/services                                    32.9K  3.72T     32.9K  legacy
pool1/.system/syslog-149940421d9b4df8af8e52297e068541     3.63G  3.72T     3.63G  legacy
pool1/.system/syslog-646f8dae97d646cc8946ddeb0ca79d97      208K  3.72T      208K  legacy
pool1/.system/syslog-ae1f6b9a44d94d15a639a40c1e91b328     3.77M  3.72T     3.77M  legacy
pool1/.system/webui                                       31.4K  3.72T     31.4K  legacy
pool1/backups                                              129M  3.72T      129M  /pool1/backups
pool1/iocage                                               324G  3.72T     8.98M  /pool1/iocage
pool1/iocage/download                                      931M  3.72T     31.4K  /pool1/iocage/download
pool1/iocage/download/11.2-RELEASE                         272M  3.72T      272M  /pool1/iocage/download/11.2-RELEASE
pool1/iocage/download/11.3-RELEASE                         288M  3.72T      288M  /pool1/iocage/download/11.3-RELEASE
pool1/iocage/download/12.1-RELEASE                         371M  3.72T      371M  /pool1/iocage/download/12.1-RELEASE
pool1/iocage/images                                       31.4K  3.72T     31.4K  /pool1/iocage/images
pool1/iocage/jails                                         320G  3.72T     32.9K  /pool1/iocage/jails
pool1/iocage/jails/headphones_1                            264G  3.72T     51.2K  /pool1/iocage/jails/headphones_1
pool1/iocage/jails/headphones_1/root                       264G  3.72T      264G  /pool1/iocage/jails/headphones_1/root
pool1/iocage/jails/lidarr                                 1.34G  3.72T      120K  /pool1/iocage/jails/lidarr
pool1/iocage/jails/lidarr/root                            1.34G  3.72T     1.34G  /pool1/iocage/jails/lidarr/root
pool1/iocage/jails/plex                                   50.5G  3.72T      207K  /pool1/iocage/jails/plex
pool1/iocage/jails/plex/root                              50.5G  3.72T     51.1G  /pool1/iocage/jails/plex/root
pool1/iocage/jails/radarr                                 1.72G  3.72T      131K  /pool1/iocage/jails/radarr
pool1/iocage/jails/radarr/root                            1.72G  3.72T     1.72G  /pool1/iocage/jails/radarr/root
pool1/iocage/jails/sabnzbd                                 874M  3.72T      169K  /pool1/iocage/jails/sabnzbd
pool1/iocage/jails/sabnzbd/root                            874M  3.72T      874M  /pool1/iocage/jails/sabnzbd/root
pool1/iocage/jails/sonarr                                 1.31G  3.72T      115K  /pool1/iocage/jails/sonarr
pool1/iocage/jails/sonarr/root                            1.31G  3.72T     1.31G  /pool1/iocage/jails/sonarr/root
pool1/iocage/log                                          65.4K  3.72T     65.4K  /pool1/iocage/log
pool1/iocage/releases                                     2.94G  3.72T     31.4K  /pool1/iocage/releases
pool1/iocage/releases/11.2-RELEASE                         883M  3.72T     31.4K  /pool1/iocage/releases/11.2-RELEASE
pool1/iocage/releases/11.2-RELEASE/root                    883M  3.72T      874M  /pool1/iocage/releases/11.2-RELEASE/root
pool1/iocage/releases/11.3-RELEASE                         861M  3.72T     31.4K  /pool1/iocage/releases/11.3-RELEASE
pool1/iocage/releases/11.3-RELEASE/root                    861M  3.72T      861M  /pool1/iocage/releases/11.3-RELEASE/root
pool1/iocage/releases/12.1-RELEASE                        1.24G  3.72T     32.9K  /pool1/iocage/releases/12.1-RELEASE
pool1/iocage/releases/12.1-RELEASE/root                   1.24G  3.72T     1.24G  /pool1/iocage/releases/12.1-RELEASE/root
pool1/iocage/templates                                    31.4K  3.72T     31.4K  /pool1/iocage/templates
pool1/jail                                                47.8G  3.72T      139M  /pool1/jail
pool1/jail-auto-20170517.0500-2w-clone                    5.12M  3.72T     1.76G  /pool1/jail-auto-20170517.0500-2w-clone
pool1/jail/.warden-template-9.1-RELEASE-amd64              347M  3.72T      347M  /pool1/jail/.warden-template-9.1-RELEASE-amd64
pool1/jail/.warden-template-9.1-RELEASE-amd64-pluginjail   851M  3.72T      851M  /pool1/jail/.warden-template-9.1-RELEASE-amd64-pluginjail
pool1/jail/.warden-template-pluginjail-10.3-x64            808M  3.72T      806M  /pool1/jail/.warden-template-pluginjail-10.3-x64
pool1/jail/.warden-template-pluginjail-9.2-x64             692M  3.72T      692M  /pool1/jail/.warden-template-pluginjail-9.2-x64
pool1/jail/.warden-template-pluginjail-9.3-x64             725M  3.72T      725M  /pool1/jail/.warden-template-pluginjail-9.3-x64
pool1/jail/.warden-template-standard-9.3-x64              1.90G  3.72T     1.90G  /pool1/jail/.warden-template-standard-9.3-x64
pool1/jail/couchpotato_1                                  1.91G  3.72T     2.69G  /pool1/jail/couchpotato_1
pool1/jail/headphones_1                                   6.93G  3.72T     7.72G  /pool1/jail/headphones_1
pool1/jail/plexmediaserver_1                              24.8G  3.72T     25.6G  /pool1/jail/plexmediaserver_1
pool1/jail/sabnzbd_1                                      2.22G  3.72T     3.00G  /pool1/jail/sabnzbd_1
pool1/jail/sonarr_1                                       3.19G  3.72T     3.67G  /pool1/jail/sonarr_1
pool1/jail/transmission_1                                 1.73G  3.72T     2.51G  /pool1/jail/transmission_1
pool1/media                                               8.85T  3.72T     8.85T  /pool1/media
pool1/sab_downloads                                        462G  3.72T      462G  /pool1/sab_downloads
pool1/software                                             888M  3.72T      888M  /pool1/software
root@nas:/ # cd /pool1
root@nas:/pool1 # ls
.freenas                iocage                    old_desktop                software
.Trash-1001                jail                    Public                    tftproot
backups                    jail-auto-20170517.0500-2w-clone    pxe
flexget.log                media                    rsync
Freenasupgrade                nfs_export                sab_downloads
root@nas:/pool1 # 
root@nas:/pool1 # 
root@nas:/pool1 # cd media
root@nas:/pool1/media # ls
movies        Music        tv_series
root@nas:/pool1/media # cd Music
root@nas:/pool1/media/Music # ls
Automatically Add to iTunes    Books                Mobile Applications        Music
root@nas:/pool1/media/Music # cd ..
root@nas:/pool1/media # cd ..
root@nas:/pool1 # cd ..

If you pulled a bone-head move like I did, and your zfs pool is still in tact, but a single volume that you inadvertently added has failed, alter /boot/loader.conf, add in the vfs.zfs.recover=1 parameter (unsure if its absolutely necessary, but its worked for me), then try the parameter:
sysctl vfs.zfs.max_missing_tvds=1

Then give your zpool import -o readonly=on <poolname> a try. I seriously hope this helps someone else out there - I've seen in the duration of my research a few other folks who were in the same situation, and abandoned all their data. My pool is in a degraded state, in a read-only configuration (and a questionable disk that appears to be failing), but I do now have the opportunity to copy the data off in advance of recreating the pool. Yay! Success! Open source! Massive shout out to Andrew Klaus for his help - bsd wizard! We all learned something - persistence pays off, and don't give up if you are in the same situation I was in. You might just have some hope yet!

Best of luck!

 

[Heracles]

Good that you managed to recover an access to your data.

Indeed, the only thing to do now is to copy all you can out of that still extremely hazardous situation. Put that on a separate storage and that will become your copy No1. Next thing will be to take your first complete backup : a complete copy of all your data as independent copy No2. Once you have both copies on stable supports, then you can go designing a server and a pool that will be up to the task. You then put a copy of your data in and that will become your new copy No1. Send one of your backup to a safe place (a friend, a family member, a safe at the bank, ... ) and ensure to keep you backups up-to-date.

 

[Arwen]

@mbro - Yes, newer versions of OpenZFS include the ability to import a damaged pool so that you can extract all the undamaged data off.

Based on what you said, you likely did not add any data to the pool after adding the single disk. Thus, any preexisting data would be good and should be available to be copied off without any errors.

Note that since you used the command line to import your pool, it was mounted at "/pool1". If you had done it through the GUI, (not possible in this specific case, because you needed to import R/O), it would have mounted it at "/mnt/pool1".

Last, while using an external USB enclosure is not good, my personal opinion is that in your case, it would work. At least long enough to get your data off the RAID-Z1 vDev.

 

[IOSonic]

@mbro I am immensely pleased (and amazed) that you have recovered your data and also came back to share the solution with others. Kudos for refusing to give up and digging for solutions until you found one! I feel like this should be rolled into a troubleshooting sticky.