SOLVED Pool BDD state is offline

[teztaz]

Please HELP!
Not sure how to get them back online the zpool import shows them ONLINE. Please note I am a novice with FreeNas.

System Specs:
Version: FreeNAS-11.3-U5
CPU: Dual Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
RAM: 32 GB

Disk in Pool:
da1 K8K90P7D 5.46 TiB
da2 K1K2ST3D 5.46 TiB
da3 K1KZJATD 5.46 TiB
da4 K1KZJLAD 5.46 TiB
da8 K8K9053D 5.46 TiB
da10 K8K905WD 5.46 TiB

output from zpool import
pool: BDD
id: 951754380038341644
state: ONLINE
status: One or more devices were being resilvered.
action: The pool can be imported using its name or numeric identifier.
config:

BDD ONLINE
raidz1-0 ONLINE
gptid/c0bc0058-d5fc-11ea-b418-ac1f6b81d968 ONLINE
gptid/c1cabc35-d5fc-11ea-b418-ac1f6b81d968 ONLINE
gptid/c2cbe96a-d5fc-11ea-b418-ac1f6b81d968 ONLINE
raidz1-1 ONLINE
gptid/c576a5b8-d753-11ea-888b-ac1f6b81d968 ONLINE
spares
gptid/c576a5b8-d7

 

[teztaz]

I was able to get the pool back on line it is now resilvering I use the zpool import BDD command.
There seems to be one disk causing issue da10 K8K905WD 5.46 TiB but when browse to that pool there seems to be no files there is that because it is resilvering?
Also seems I am running an older version but when I click the check for updates I get the error "Unable to connect to url https://update-master.ixsystems.com/FreeNAS/trains.txt: Automatic update check failed. Please check system network settings. " but I check the URL from my PC and it works fine and the DNS settings are the same for my FreeNAS and my PC.

 

[Samuel Tai]

The update check failure is due to a certificate expiration. See https://www.truenas.com/community/t...the-openssl-1-0-2-vs-letsencrypt-issue.95874/.

As for your pool showing empty, I'm not sure I understand the layout of your pool. Your BDD pool status appears to be cut off.

 

[teztaz]

Here is the results of the zpool status for BDD

Code:

root@ccnas:~ # zpool status
  pool: BDD
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Oct 16 22:56:09 2021
        16.7T scanned at 3.21G/s, 14.5T issued at 2.23G/s, 21.6T total
        378G resilvered, 66.92% done, 0 days 00:54:35 to go
config:

        NAME                                              STATE     READ WRITE CKSUM
        BDD                                               DEGRADED     0     0   0
          raidz1-0                                        ONLINE       0     0   0
            gptid/c0bc0058-d5fc-11ea-b418-ac1f6b81d968    ONLINE       0     0   0
            gptid/c1cabc35-d5fc-11ea-b418-ac1f6b81d968    ONLINE       0     0   0
            gptid/c2cbe96a-d5fc-11ea-b418-ac1f6b81d968    ONLINE       0     0   0
          raidz1-1                                        DEGRADED     0     0   1
            gptid/c576a5b8-d753-11ea-888b-ac1f6b81d968    ONLINE       0     0   0
            spare-1                                       ONLINE       0     0   3
              da8p2                                       ONLINE       0     0   0
              gptid/3a65c144-d5fe-11ea-b418-ac1f6b81d968  ONLINE       0     0   0
            gptid/c85ea107-d753-11ea-888b-ac1f6b81d968    FAULTED    108 4.56K   0  too many errors
        spares
          7762740217083127831                             INUSE     was /dev/gptid/3a65c144-d5fe-11ea-b418-ac1f6b81d968

errors: 9666525 data errors, use '-v' for a list

 

[Samuel Tai]

Unfortunately, your pool may be beyond saving. You have 2 failed disks in the raidz1-1 stripe, and your pool layout requires both stripes to be online for the pool to be healthy. One of the failures is dynamically swapped with a spare, but the other disk has multiple read and write failures. The resilver is for the spare. Wait for the resilver to complete, and replace the failed disk to generate another resilver.

How did you come to choose this layout? Did you intend for a mirror of raidz1 vdevs?

 

[teztaz]

I started with three disk and a spare in the BDD then wanted to add more disk and while reading it said to add more pairs to the same pool as long as they were the same size drives.

 

[teztaz]

I only see the one failed disk and it is in the raidz1-1 set up where do you see the second disk failure?

 

[Samuel Tai]

Ouch. You inadvertently striped your pool instead of expanding your existing raidz1. You would've been better off adding them as spares.

 

[teztaz]

Also how can I tell which disk has failed it does not show a disk offline in the Storage > Disk and looking at Storage>Pools it is just spinning. I think I am going to give it a day to let it finish and cross my fingers it is not lost.

 

[teztaz]

When I was adding the disk I wanted to get more storage in the same Pool how would adding more spares do that?
Just in case I have to rebuild the 10 TB of storage. ugh :-(

 

[Samuel Tai]

I only see the one failed disk and it is in the raidz1-1 set up where do you see the second disk failure?

Code:

        NAME                                              STATE     READ WRITE CKSUM
        BDD                                               DEGRADED     0     0   0
          raidz1-0                                        ONLINE       0     0   0
            gptid/c0bc0058-d5fc-11ea-b418-ac1f6b81d968    ONLINE       0     0   0
            gptid/c1cabc35-d5fc-11ea-b418-ac1f6b81d968    ONLINE       0     0   0
            gptid/c2cbe96a-d5fc-11ea-b418-ac1f6b81d968    ONLINE       0     0   0
          raidz1-1                                        DEGRADED     0     0   1
            gptid/c576a5b8-d753-11ea-888b-ac1f6b81d968    ONLINE       0     0   0
            spare-1                                       ONLINE       0     0   3 <- this means a disk has failed, and is using a dynamic spare
              da8p2                                       ONLINE       0     0   0 <- this is the failed drive
              gptid/3a65c144-d5fe-11ea-b418-ac1f6b81d968  ONLINE       0     0   0 <- this is the spare
            gptid/c85ea107-d753-11ea-888b-ac1f6b81d968    FAULTED    108 4.56K   0  too many errors
        spares
          7762740217083127831                             INUSE     was /dev/gptid/3a65c144-d5fe-11ea-b418-ac1f6b81d968 <- GUID matches the spare above

 

[teztaz]

I noticed this in my Task Manager in the UI.

 

[teztaz]

Code:

        NAME                                              STATE     READ WRITE CKSUM
        BDD                                               DEGRADED     0     0   0
          raidz1-0                                        ONLINE       0     0   0
            gptid/c0bc0058-d5fc-11ea-b418-ac1f6b81d968    ONLINE       0     0   0
            gptid/c1cabc35-d5fc-11ea-b418-ac1f6b81d968    ONLINE       0     0   0
            gptid/c2cbe96a-d5fc-11ea-b418-ac1f6b81d968    ONLINE       0     0   0
          raidz1-1                                        DEGRADED     0     0   1
            gptid/c576a5b8-d753-11ea-888b-ac1f6b81d968    ONLINE       0     0   0
            spare-1                                       ONLINE       0     0   3 <- this means a disk has failed, and is using a dynamic spare
              da8p2                                       ONLINE       0     0   0
              gptid/3a65c144-d5fe-11ea-b418-ac1f6b81d968  ONLINE       0     0   0
            gptid/c85ea107-d753-11ea-888b-ac1f6b81d968    FAULTED    108 4.56K   0  too many errors
        spares
          7762740217083127831                             INUSE     was /dev/gptid/3a65c144-d5fe-11ea-b418-ac1f6b81d968

but having the spare should have saved me right???

 

[Samuel Tai]

but having the spare should have saved me right???

That's assuming the spare is good. If the spare isn't good, your pool is dead.

 

[Samuel Tai]

When I was adding the disk I wanted to get more storage in the same Pool how would adding more spares do that?
Just in case I have to rebuild the 10 TB of storage. ugh :-(

Spares wouldn't have added space to your pool, but would've added redundancy. To add space, the proper procedure is to replace all the drives with larger ones. Extending your pool to another 3 raidz1 vdev results in striping over both vdevs, which appears to add space, but also increases risk of data loss.

If you do end up having to rebuild your pool, I recommend going with raidz2 next time, which can withstand the loss of 2 drives simultaneously.

 

[teztaz]

I noticed the folder "BDD" is now under my Root path and not my mnt path why is that? This is causing my shares to not work.

 

[Samuel Tai]

Yes, that's because you used zpool import BDD from the command line without specifying -R /mnt. You'll have to wait until the resilver completes, and then export your pool, and then use the GUI to reimport your pool.

 

[teztaz]

Looks like I may have to add drives and build a new Pool then copy the data from the root location
for raidz2 the min drive count is 3 can I have not multiple of three drives? Example if I go with 8 TB drives can I put in 5 and have a spare total of 6 drives that gives me the same space I have today.

 

[teztaz]

Yes, that's because you used zpool import BDD from the command line without specifying -R /mnt. You'll have to wait until the resilver completes, and then export your pool, and then use the GUI to reimport your pool.

seems the resilver is done and drive is back online. How do I do the export and reimport.

 

[teztaz]

I am assuming I just have it for the delete config? Does that mean I will have to redo the shares?