replacing failing hard drive?

[KristofferD]

I've replaced my failing Seagate drive with a new WD Red but when I try and replace the new drive into the pool I can't find any member disk?

Followed this guide: https://www.youtube.com/watch?v=9YmPg_uP08w and also this one https://www.youtube.com/watch?v=c8bvtj-LQ_A
What I've done so far:
1) Change status of the bad drive from online to offline
2) Shut down machine
3) Physically change the drives, hate my choice of case right now but do love that it's so tiny.
4) Boot up again
5) Get into the freenas webui
6) Go to view volume -> status
7) I mark the new disk (got random numbers instead of adaXp2 as the other ones) and hit "replace" but the drop down menu for member drives is empty so I can't replace the bad drive?

I'm totally stuck and don't know what to do right now.
I'm using all 6 sata ports on the mb and the guide shows that they don't need to be connected at the same time?

Am running raidz2
MB: MSI B75MA-P45
CPU: i5 2320
32GB ddr 3 non ecc
Seagate Barracuda 4x2tb
WD Red 2x2tb

edit update: found out through https://www.ixsystems.com/community/threads/how-can-you-tell-if-a-zpool-is-encrypted.56213/ and some other forum posts that the pool is encrypted and I've not been using the key for ~2 years and have no clue to what passcode I've on it so I'll have to re-install it and copy all data from an up to date backup.

 

[PhiloEpisteme]

Hi @Kristoffer Danbrant. I had a VERY similar issue and posted to the forums as well as filed a bug recently while running 11.2-U2. Would you mind please posting your specifics, such as your FreeNAS version?

found out through https://www.ixsystems.com/community/threads/how-can-you-tell-if-a-zpool-is-encrypted.56213/ and some other forum posts that the pool is encrypted and I've not been using the key for ~2 years and have no clue to what passcode I've on it so I'll have to re-install it and copy all data from an up to date backup.

Is it possible you don't have a password on the key? If you rebooted the system and your pool did not require a password to decrypt that is a good sign you don't have a password. In that case you can simply rekey the pool and when you do download a backup of the key and the recovery key. Either way though, I think you should still see your new drive appear in the replace disk dropdown.

Does your system support hot-swapping?

 

[Chris Moore]

I'm using all 6 sata ports on the mb and the guide shows that they don't need to be connected at the same time?

I don't understand what you are trying to say with that. All the drives in your pool need to be connected.

edit update: found out through https://www.ixsystems.com/community/threads/how-can-you-tell-if-a-zpool-is-encrypted.56213/ and some other forum posts that the pool is encrypted and I've not been using the key for ~2 years and have no clue to what passcode I've on it so I'll have to re-install it and copy all data from an up to date backup.

As you go through the process of building again, you might want to avoid encryption if you can. It does have a good potential for creating problems.

 

[melloa]

I'm using all 6 sata ports on the mb and the guide shows that they don't need to be connected at the same time?

Are you saying the bad (removed) drive and the new (replacement) drive? The other drives are also connected to the MB, correct?

 

[KristofferD]

Hi @Kristoffer Danbrant. I had a VERY similar issue and posted to the forums as well as filed a bug recently while running 11.2-U2. Would you mind please posting your specifics, such as your FreeNAS version?


Is it possible you don't have a password on the key? If you rebooted the system and your pool did not require a password to decrypt that is a good sign you don't have a password. In that case you can simply rekey the pool and when you do download a backup of the key and the recovery key. Either way though, I think you should still see your new drive appear in the replace disk dropdown.

Does your system support hot-swapping?

It's a ~9 year old motherboard so unfortunately it doesn't support hot-swapping. I do guess I don't have a password since it never asks for it and works flawlessly when I reboot the system.
I've followed the manual to the letter but at step 3 the drop down menu is empty so I can't replace the old bad drive with the new one.
I do run the same version (11.2-U2)

I don't understand what you are trying to say with that. All the drives in your pool need to be connected.

As you go through the process of building again, you might want to avoid encryption if you can. It does have a good potential for creating problems.

I meant that I don't have any free sata ports on the motherboard so I can't have the old drives connected at the same time as the new one, it only has a couple of bad blocks but better safe then sorry.
I've some copys of my source code on the NAS and other personal data and want to be able to throw old drives without thinking about wiping them down first, that's the main reason why I opted for the encryption setting.

Are you saying the bad (removed) drive and the new (replacement) drive? The other drives are also connected to the MB, correct?

Yes, there's no free ports on the MB so I can't have the bad (removed) drive connected at the same time as the new (replacement) drive unfortunately.


edit update: ssh into the nas and ran zpool status and got:

Code:

% zpool status

  pool: MyVolume

state: DEGRADED

status: One or more devices is currently being resilvered.  The pool will

    continue to function, possibly in a degraded state.

action: Wait for the resilver to complete.

  scan: resilver in progress since Mon Mar  4 13:46:21 2019

    5.83T scanned at 2.41G/s, 3.42T issued at 1.41G/s, 7.16T total

    0 resilvered, 47.74% done, 0 days 00:45:10 to go

config:


    NAME                                            STATE     READ WRITE CKSUM

    MyVolume                                        DEGRADED     0     0     0

      raidz2-0                                      DEGRADED     0     0     0

        5033901764192954979                         OFFLINE      0     0     0  was /dev/gptid/cff542dd-5e44-11e6-993f-d43d7ef3a0c3

        gptid/d0c18b53-5e44-11e6-993f-d43d7ef3a0c3  ONLINE       0     0     0

        gptid/d19de724-5e44-11e6-993f-d43d7ef3a0c3  ONLINE       0     0     0

        gptid/d27d5e7b-5e44-11e6-993f-d43d7ef3a0c3  ONLINE       0     0     0

        gptid/d35e60a8-5e44-11e6-993f-d43d7ef3a0c3  ONLINE       0     0     0

        gptid/d42f8861-5e44-11e6-993f-d43d7ef3a0c3  ONLINE       0     0     0


errors: No known data errors


  pool: freenas-boot

state: ONLINE

  scan: scrub repaired 0 in 0 days 02:06:54 with 0 errors on Wed Feb 27 05:51:54 2019

config:


    NAME        STATE     READ WRITE CKSUM

    freenas-boot  ONLINE       0     0     0

      da0p2     ONLINE       0     0     0


errors: No known data errors

I'm thinking if I could run this command and get it to work that way:
zpool replace -f MyVolume 5033901764192954979 ada5p2

 

[PhiloEpisteme]

edit update: ssh into the nas and ran zpool status and got:
% zpool status
pool: MyVolume
state: DEGRADED

For readability do you mind surrounding those code blocks in code tags?

that's the main reason why I opted for the encryption setting.

I've seen some folks recommend against encryption. I use it, and haven't had any issues yet. I tested the various situations heavily prior to putting any data on the encrypted pool and I keep backups. You'll want to make sure that you have a backup of the key and the recovery key. The FreeNAS guide will show you how to do that. Without that, if you have to reinstall the operating system for whatever reason, you may be in trouble. This is true even if your pool does not require a password. Keeping a backup of the config file is not a substitute for a backup of the encryption keys.

I'm thinking if I could run this command and get it to work that way:
zpool replace -f MyVolume 5033901764192954979 ada5p2

Do you have a backup of your data? If not, you should make one.

I'm not sure what else to suggest given that what fixed it for me was hot-swapping the drive. I think the bug I experienced is getting fixed in U3 and if you're experiencing the same thing it may work for you to wait for U3 and try again. Either way, backups are what will save you if things go awry.

 

[KristofferD]

For readability do you mind surrounding those code blocks in code tags?

Edited the post with code blocks now, sorry about that.

I've seen some folks recommend against encryption. I use it, and haven't had any issues yet. I tested the various situations heavily prior to putting any data on the encrypted pool and I keep backups. You'll want to make sure that you have a backup of the key and the recovery key. The FreeNAS guide will show you how to do that. Without that, if you have to reinstall the operating system for whatever reason, you may be in trouble. This is true even if your pool does not require a password. Keeping a backup of the config file is not a substitute for a backup of the encryption keys.


Do you have a backup of your data? If not, you should make one.

I'm not sure what else to suggest given that what fixed it for me was hot-swapping the drive. I think the bug I experienced is getting fixed in U3 and if you're experiencing the same thing it may work for you to wait for U3 and try again. Either way, backups are what will save you if things go awry.

I've it all backup on AWS S3 bucket but it would take days if not a week to download all the data, we're talking about ~6T of actual data on a 100mbps/10mbps connection :/
The NAS is rocking raidz2 with 5 out of 6 working drives so it's no panic so I could wait and hope that U3 fixes it, as I said I also have a backup of all the stuff, the really important stuff is copyed to at least 3 different servers for total security.

 

[Chris Moore]

It's a ~9 year old motherboard so unfortunately it doesn't support hot-swapping.

Perhaps it is time to upgrade your hardware. It was never a server board to begin with.

I do guess I don't have a password since it never asks for it and works flawlessly when I reboot the system.

From what you have shown us, it does not appear to be an encrypted pool. What makes you think it is?

If it was encrypted, the zpool status would look like this:

Code:

% zpool status 
  pool: Test
state: ONLINE
  scan: resilvered 3.10T in 16h59m with 0 errors on Mon Mar  4 13:46:21 2019
config:

  NAME                                                STATE     READ WRITE CKSUM
  Test                                                ONLINE       0     0     0
    raidz2-0                                          ONLINE       0     0     0
      gptid/c8688176-4668-11e4-b49f-d05099264f68.eli  ONLINE       0     0     0
      gptid/c8e2f1e9-4668-11e4-b49f-d05099264f68.eli  ONLINE       0     0     0
      gptid/c9cc35d5-4668-11e4-b49f-d05099264f68.eli  ONLINE       0     0     0
      gptid/ca449ed0-4668-11e4-b49f-d05099264f68.eli  ONLINE       0     0     0
      gptid/cabb7d11-4668-11e4-b49f-d05099264f68.eli  ONLINE       0     0     0
      gptid/cb39f216-4668-11e4-b49f-d05099264f68.eli  ONLINE       0     0     0

errors: No known data errors

Instead, your pool looks like this:

Code:

% zpool status
  pool: MyVolume
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Mar  4 13:46:21 2019
    5.83T scanned at 2.41G/s, 3.42T issued at 1.41G/s, 7.16T total
    0 resilvered, 47.74% done, 0 days 00:45:10 to go
config:

    NAME                                            STATE     READ WRITE CKSUM
    MyVolume                                        DEGRADED     0     0     0
      raidz2-0                                      DEGRADED     0     0     0
        5033901764192954979                         OFFLINE      0     0     0  was /dev/gptid/cff542dd-5e44-11e6-993f-d43d7ef3a0c3
        gptid/d0c18b53-5e44-11e6-993f-d43d7ef3a0c3  ONLINE       0     0     0
        gptid/d19de724-5e44-11e6-993f-d43d7ef3a0c3  ONLINE       0     0     0
        gptid/d27d5e7b-5e44-11e6-993f-d43d7ef3a0c3  ONLINE       0     0     0
        gptid/d35e60a8-5e44-11e6-993f-d43d7ef3a0c3  ONLINE       0     0     0
        gptid/d42f8861-5e44-11e6-993f-d43d7ef3a0c3  ONLINE       0     0     0

errors: No known data errors

Do you see the difference?

I've followed the manual to the letter but at step 3 the drop down menu is empty so I can't replace the old bad drive with the new one.
I do run the same version (11.2-U2)

There may be a bug in the new GUI. It may be that you need to login to the legacy GUI to complete the drive replacement.
If you need assistance, please respond back to the forum.

Yes, there's no free ports on the MB so I can't have the bad (removed) drive connected at the same time as the new (replacement) drive unfortunately.

You can add more ports to connect drives if you want to, but it might be better to think about a newer system board.

Keeping a backup of the config file is not a substitute for a backup of the encryption keys.

Because the keys (for security reasons) are not supposed to be stored in the configuration database. Kind of like leaving your keys in your car.

The NAS is rocking raidz2 with 5 out of 6 working drives so it's no panic so I could wait and hope that U3 fixes it,

I wish you would be more interactive in the effort to troubleshoot this problem