Replaced Failed Drive But The New Drive is not in the same mirror.

Status
Not open for further replies.
alxxer

alxxer

Dabbler
Joined
May 7, 2013
Messages
38
My system is setup is a stripped pool. ada1 failed so I removed it and added a new drive but it will not join the mirror.

Code:
[alex@freenas] /# zpool status -v
  pool: storage00
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Online the device using 'zpool online' or replace the device with
    'zpool replace'.
  scan: scrub canceled on Mon Oct 28 22:46:27 2013
config:
 
    NAME                                            STATE    READ WRITE CKSUM
    storage00                                      DEGRADED    0    0    0
      mirror-0                                      ONLINE      0    0    0
        gptid/9eb8b753-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
        gptid/9f69eedd-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      mirror-1                                      ONLINE      0    0    0
        gptid/c4caec32-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
        gptid/c54919bd-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      mirror-2                                      DEGRADED    0    0    0
        2612972933469237643                        OFFLINE      0    0    0  was /dev/gptid/ec6a123e-0761-11e3-864f-6805ca14dfe2
        gptid/ed3c2e6f-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      gptid/8a811262-0767-11e3-9186-6805ca14dfe2    ONLINE      0    0    0
      mirror-4                                      ONLINE      0    0    0
        gptid/bd426018-0c4e-11e3-b2fa-6805ca14dfe2  ONLINE      0    0    0
        gptid/bdc4a828-0c4e-11e3-b2fa-6805ca14dfe2  ONLINE      0    0    0


I hope these help
Code:
[alex@freenas] /# glabel list
Geom name: ada0p2
Providers:
1. Name: gptid/8a811262-0767-11e3-9186-6805ca14dfe2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r1w1e1
  secoffset: 0
  offset: 0
  seclength: 3902834703
  length: 1998251367936
  index: 0
Consumers:
1. Name: ada0p2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r1w1e2
 
Geom name: ada1p2
Providers:
1. Name: gptid/f28fefff-4044-11e3-8070-6805ca14dfe2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r0w0e0
  secoffset: 0
  offset: 0
  seclength: 3902834703
  length: 1998251367936
  index: 0
Consumers:
1. Name: ada1p2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r0w0e0
 
Geom name: ada2p2
Providers:
1. Name: gptid/ed3c2e6f-0761-11e3-864f-6805ca14dfe2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r1w1e1
  secoffset: 0
  offset: 0
  seclength: 3902834703
  length: 1998251367936
  index: 0
Consumers:
1. Name: ada2p2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r1w1e2
 
Geom name: ada3p2
Providers:
1. Name: gptid/ec6a123e-0761-11e3-864f-6805ca14dfe2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r0w0e0
  secoffset: 0
  offset: 0
  seclength: 3902834703
  length: 1998251367936
  index: 0
Consumers:
1. Name: ada3p2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r0w0e0
 
Geom name: ada4p2
Providers:
1. Name: gptid/9eb8b753-0761-11e3-864f-6805ca14dfe2
  Mediasize: 3998639460352 (3.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  secoffset: 0
  offset: 0
  seclength: 7809842696
  length: 3998639460352
  index: 0
Consumers:
1. Name: ada4p2
  Mediasize: 3998639460352 (3.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: ada5p2
Providers:
1. Name: gptid/c4caec32-0761-11e3-864f-6805ca14dfe2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  secoffset: 0
  offset: 0
  seclength: 5856338696
  length: 2998445412352
  index: 0
Consumers:
1. Name: ada5p2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: ada6p2
Providers:
1. Name: gptid/9f69eedd-0761-11e3-864f-6805ca14dfe2
  Mediasize: 3998639460352 (3.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  secoffset: 0
  offset: 0
  seclength: 7809842696
  length: 3998639460352
  index: 0
Consumers:
1. Name: ada6p2
  Mediasize: 3998639460352 (3.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: ada7p2
Providers:
1. Name: gptid/c54919bd-0761-11e3-864f-6805ca14dfe2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  secoffset: 0
  offset: 0
  seclength: 5856338696
  length: 2998445412352
  index: 0
Consumers:
1. Name: ada7p2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: ada8p2
Providers:
1. Name: gptid/bd426018-0c4e-11e3-b2fa-6805ca14dfe2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  secoffset: 0
  offset: 0
  seclength: 5856338696
  length: 2998445412352
  index: 0
Consumers:
1. Name: ada8p2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: ada9p2
Providers:
1. Name: gptid/bdc4a828-0c4e-11e3-b2fa-6805ca14dfe2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  secoffset: 0
  offset: 0
  seclength: 5856338696
  length: 2998445412352
  index: 0
Consumers:
1. Name: ada9p2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: da0s3
Providers:
1. Name: ufs/FreeNASs3
  Mediasize: 1548288 (1.5M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 1976647680
  Mode: r0w0e0
  secoffset: 0
  offset: 0
  seclength: 3024
  length: 1548288
  index: 0
Consumers:
1. Name: da0s3
  Mediasize: 1548288 (1.5M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 1976647680
  Mode: r0w0e0
 
Geom name: da0s4
Providers:
1. Name: ufs/FreeNASs4
  Mediasize: 21159936 (20M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 1978195968
  Mode: r1w1e1
  secoffset: 0
  offset: 0
  seclength: 41328
  length: 21159936
  index: 0
Consumers:
1. Name: da0s4
  Mediasize: 21159936 (20M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 1978195968
  Mode: r1w1e2
 
Geom name: da0s1a
Providers:
1. Name: ufsid/51fa8b722a9ec945
  Mediasize: 988283392 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 40448
  Mode: r0w0e0
  secoffset: 0
  offset: 0
  seclength: 1930241
  length: 988283392
  index: 0
Consumers:
1. Name: da0s1a
  Mediasize: 988283392 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 40448
  Mode: r0w0e0
 
Geom name: da0s1a
Providers:
1. Name: ufs/FreeNASs1a
  Mediasize: 988283392 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 40448
  Mode: r0w0e0
  secoffset: 0
  offset: 0
  seclength: 1930241
  length: 988283392
  index: 0
Consumers:
1. Name: da0s1a
  Mediasize: 988283392 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 40448
  Mode: r0w0e0
 
Geom name: da0s2a
Providers:
1. Name: ufs/FreeNASs2a
  Mediasize: 988283392 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 988364288
  Mode: r1w0e0
  secoffset: 0
  offset: 0
  seclength: 1930241
  length: 988283392
  index: 0
Consumers:
1. Name: da0s2a
  Mediasize: 988283392 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 988364288
  Mode: r1w0e1


This is what the volume status looks like
vol_status.png

I hope this info helps,

Thanks
 
warri

warri

Guru
Joined
Jun 6, 2011
Messages
1,193
How did you replace that drive? It looks like you didn't follow the instructions in the manual.

For me it looks like you added a single drive to your pool, which is now listed in the GUI as stripe. This means if the new drive fails, your pool is gone. See page 23 of Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!
Unfortunately you cannot undo this, and to fix the problem you have to recreate your pool and restore the data from a backup (Somebody correct me if I'm wrong!).

EDIT: Also your FreeNAS version seems to be quite old, since it's using the old adaXpX labels instead of GUIDs. (nevermind, looks ok in the provided CLI output) Anyway, which version are you running?
 
C

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
Nope, the OP did exactly what you think he did warri. One drive failure from losing his entire pool.

alxxer, You might want to add a mirror to that stripe disk pronto. If that disk fails you will lose everything in your pool.
 
Dusan

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
warri said:
This means if the new drive fails, your pool is gone.
Click to expand...
And that drive isn't the only current weak point. Until the missing drive in mirror-2 is replaced that remaining lone drive in mirror-2 is a single point of failure too. So, alxxer needs to add two drives ASAP.
 
alxxer

alxxer

Dabbler
Joined
May 7, 2013
Messages
38
warri said:
How did you replace that drive? It looks like you didn't follow the instructions in the manual.

For me it looks like you added a single drive to your pool, which is now listed in the GUI as stripe. This means if the new drive fails, your pool is gone. See page 23 of Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!
Unfortunately you cannot undo this, and to fix the problem you have to recreate your pool and restore the data from a backup (Somebody correct me if I'm wrong!).

EDIT: Also your FreeNAS version seems to be quite old, since it's using the old adaXpX labels instead of GUIDs. (nevermind, looks ok in the provided CLI output) Anyway, which version are you running?
Click to expand...

I'm running version 9.1.1
 
alxxer

alxxer

Dabbler
Joined
May 7, 2013
Messages
38
So, i'm reslivering the disk that was offline.

Code:
[alex@freenas] /# zpool status -v
  pool: storage00
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Oct 30 08:45:35 2013
        10.8G scanned out of 8.44T at 284M/s, 8h39m to go
        2.68G resilvered, 0.12% done
config:
 
    NAME                                            STATE    READ WRITE CKSUM
    storage00                                      ONLINE      0    0    0
      mirror-0                                      ONLINE      0    0    0
        gptid/9eb8b753-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
        gptid/9f69eedd-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      mirror-1                                      ONLINE      0    0    0
        gptid/c4caec32-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
        gptid/c54919bd-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      mirror-2                                      ONLINE      0    0    0
        gptid/06d0a5a8-4161-11e3-b720-6805ca14dfe2  ONLINE      0    0    0  (resilvering)
        gptid/ed3c2e6f-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      gptid/8a811262-0767-11e3-9186-6805ca14dfe2    ONLINE      0    0    0
      mirror-4                                      ONLINE      0    0    0
        gptid/bd426018-0c4e-11e3-b2fa-6805ca14dfe2  ONLINE      0    0    0
        gptid/bdc4a828-0c4e-11e3-b2fa-6805ca14dfe2  ONLINE      0    0    0


How to I add the new disk to the mirror, and is now a stripe?
 
alxxer

alxxer

Dabbler
Joined
May 7, 2013
Messages
38
cyberjock said:
Nope, the OP did exactly what you think he did warri. One drive failure from losing his entire pool.

alxxer, You might want to add a mirror to that stripe disk pronto. If that disk fails you will lose everything in your pool.
Click to expand...

So the reslivering completed
Code:
errors: No known data errors
[alex@freenas] /# zpool status -v
  pool: storage00
state: ONLINE
  scan: resilvered 1.66T in 6h34m with 0 errors on Wed Oct 30 15:19:57 2013
config:
 
    NAME                                            STATE    READ WRITE CKSUM
    storage00                                      ONLINE      0    0    0
      mirror-0                                      ONLINE      0    0    0
        gptid/9eb8b753-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
        gptid/9f69eedd-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      mirror-1                                      ONLINE      0    0    0
        gptid/c4caec32-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
        gptid/c54919bd-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      mirror-2                                      ONLINE      0    0    0
        gptid/06d0a5a8-4161-11e3-b720-6805ca14dfe2  ONLINE      0    0    0
        gptid/ed3c2e6f-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      gptid/8a811262-0767-11e3-9186-6805ca14dfe2    ONLINE      0    0    0
      mirror-4                                      ONLINE      0    0    0
        gptid/bd426018-0c4e-11e3-b2fa-6805ca14dfe2  ONLINE      0    0    0
        gptid/bdc4a828-0c4e-11e3-b2fa-6805ca14dfe2  ONLINE      0    0    0



But why is there an extra gptid in mirror-2?
 
C

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
That's not an extra gptid in mirror-2. If you look at the formatting mirror-2's gptids are indented. That one you put in bold is not indented relative to the 2 gptids above it because it's its own vdev. That gptid is indented to the same as the "mirror-X" signifying its its own designation.That's the disk that warri and I have said you need to get in a mirror before that disk fails. You lose that disk and you lose the pool.
 
alxxer

alxxer

Dabbler
Joined
May 7, 2013
Messages
38
Ok thanks guys got the mirror backup and running.
Code:
alex@freenas] /# zpool status -v
  pool: storage00
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Oct 31 00:09:21 2013
        306M scanned out of 8.44T at 25.5M/s, 96h21m to go
        73.8M resilvered, 0.00% done
config:
 
    NAME                                            STATE    READ WRITE CKSUM
    storage00                                      ONLINE      0    0    0
      mirror-0                                      ONLINE      0    0    0
        gptid/9eb8b753-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
        gptid/9f69eedd-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      mirror-1                                      ONLINE      0    0    0
        gptid/c4caec32-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
        gptid/c54919bd-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      mirror-2                                      ONLINE      0    0    0
        gptid/06d0a5a8-4161-11e3-b720-6805ca14dfe2  ONLINE      0    0    0
        gptid/ed3c2e6f-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0
      mirror-3                                      ONLINE      0    0    0
        gptid/8a811262-0767-11e3-9186-6805ca14dfe2  ONLINE      0    0    0
        gptid/ec6a123e-0761-11e3-864f-6805ca14dfe2  ONLINE      0    0    0  (resilvering)
      mirror-4                                      ONLINE      0    0    0
        gptid/bd426018-0c4e-11e3-b2fa-6805ca14dfe2  ONLINE      0    0    0
        gptid/bdc4a828-0c4e-11e3-b2fa-6805ca14dfe2  ONLINE      0    0    0
 
errors: No known data errors
 
C

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
PERFECT! You are back in business with redundancy :)

Well, as soon as the resilvering completes.
 
Status
Not open for further replies.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Replaced Failed Drive But The New Drive is not in the same mirror."

Similar threads

H
Replaced 3TB drives with 4TB, will not "kind of" not autoexpand
Replies
11
Views
2K
danb35
danb35
V
  • Locked
SOLVED Unable to import encrypted pool after boot device failure (Recovery key does not require passphase)
Replies
2
Views
2K
svtkobra7
S
yeliaB
Attempted to replace a failing drive with a brand-new, dead-on-arrival drive; need some help...
Replies
7
Views
2K
dlavigne
D
Aberu
  • Locked
Can't import/mount zpool in GUI after fresh install to 9.3 following cata failure (my fault)
Replies
4
Views
3K
Orca04
O
M
New disks are not recognized - FreeNAS-11.2-RELEASE-U1
2
Replies
23
Views
11K
maozefang
M
Top