Unable to bring drive online

[Ralms]

Hi all,

I recently decided to move to FreeNAS, I've bought a HP Server DL360p and I've been exploring it.

Currently I have 5 drives connected, in a Z1 pool.
The drives are connected on ada0, da0, da1, da2, da3.
To test the resilience of FreeNAS, I've physically unplugged the drive on da0, which is on a Hotswap bay.

The controller and FreeNas quickly noticed the drive being disconnected, the Pool is now marked as "degraded".
I've not done anything to that drive, simply plugged it back in, in hopes FreeNas would just pick it up and the Pool to return to healthy state.
As that didn't happen, I've rebooted the server however still the same status.

Then I checked the Pool status and attempted to bring it "Online", but I get the error "[MiddlewareError: Disk cannot be set to online in encrypted pool.] "

At last I've disconnected the Pool without deleting any configuration.
Imported the Pool back and the status is still degraded and when trying to bring the drive "Online" I get the same error.

Please note that I can access the data on the Pool and I have the encryption keys.

Any idea?

Thanks,
Ralms

 

[Chris Moore]

The drive is encrypted. You have to decrypt the drive before you can join it back to your pool.

 

[Ralms]

The drive is encrypted. You have to decrypt the drive before you can join it back to your pool.

Hi Chris,

Thanks for the tip.

I've been doing some research and trying some things although I'm struggling a bit here.

First I'm a bit confused with the "/dev/" drives, this is because when I use the new GUI and go to "Storage > Disks" I have the names as I mentioned above "ada0, da0, da1, da2, ..", however when I go to "Storage > Pools" and I check my Pool status I have "ada0p2, da0p2, da1p2, ...".
Additionally, in the shell, doing "ls /dev" I see that I have for all drives the normal one, "da0" but also "da0p1" and "da0p2". This seems to have happened after I rebooted the server and disconnected and re-imported the Pool.
I'm a bit confused for the reasoning for all these.

Either way, following your tip, I tried to do the following:
"sudo geli attach -k /data/geli/<Key guid>.key -p /dev/da0p2"
and I get the error:
geli: Cannot access da0p2 (error=1).

Any tips?

Thank you for your help.

 

[Ralms]

The drive is encrypted. You have to decrypt the drive before you can join it back to your pool.

Ok this is dumb and looks like a GUI issue....

So after keeping researching this issue I found out about "zpool" command in the shell.

I ran the following:

Code:

% zpool status Mars
  pool: Mars
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: none requested
config:

        NAME                                                STATE     READ WRITE CKSUM
        Mars                                                DEGRADED     0     0     0
          raidz1-0                                          DEGRADED     0     0     0
            gptid/5fa3d1fc-5f22-11e9-a7e8-2c768a5571c4.eli  ONLINE       0     0     0
            872516724093501408                              OFFLINE      0     0     0  was /dev/gptid/66f8be9f-5f22-11e9-a7e8-2c768a5571c4.eli
            gptid/6ea29009-5f22-11e9-a7e8-2c768a5571c4.eli  ONLINE       0     0     0
            gptid/7629b8a5-5f22-11e9-a7e8-2c768a5571c4.eli  ONLINE       0     0     0
            gptid/7ddfaf08-5f22-11e9-a7e8-2c768a5571c4.eli  ONLINE       0     0     0

errors: No known data errors
%

After this I ran:

Code:

% sudo zpool online Mars 872516724093501408
Password:
%

And the Pool become Online and Healthy fine:

Code:

% zpool status Mars
  pool: Mars
 state: ONLINE
  scan: resilvered 6.93M in 0 days 00:00:04 with 0 errors on Tue Apr 16 17:32:56 2019
config:

        NAME                                                STATE     READ WRITE CKSUM
        Mars                                                ONLINE       0     0     0
          raidz1-0                                          ONLINE       0     0     0
            gptid/5fa3d1fc-5f22-11e9-a7e8-2c768a5571c4.eli  ONLINE       0     0     0
            gptid/66f8be9f-5f22-11e9-a7e8-2c768a5571c4.eli  ONLINE       0     0     0
            gptid/6ea29009-5f22-11e9-a7e8-2c768a5571c4.eli  ONLINE       0     0     0
            gptid/7629b8a5-5f22-11e9-a7e8-2c768a5571c4.eli  ONLINE       0     0     0
            gptid/7ddfaf08-5f22-11e9-a7e8-2c768a5571c4.eli  ONLINE       0     0     0

errors: No known data errors
%

So it seems that the GUI is not considering that it already had the key for the encrypted disk, etc.

 

[Chris Moore]

So it seems that the GUI is not considering that it already had the key for the encrypted disk, etc.

It should have prompted you for the password like the command line did, but it would not assume that the password for the "unknown" disk was the same as the password for the other disks it was already using.
You might want to submit a feature request / bug report on this. It could be more intuitive, or just give some more informative prompt.
Thanks for posting your results.

 

[Chris Moore]

PS. It is probably a good idea to go through and update your encryption keys for the pool. We have seen a user loose access to the pool after doing a disk replace and reboot. Encryption can be very tricky and it is designed to keep you out if you are not properly authorized, so you want to be sure you have all the keys to get back in.

 

[Ralms]

PS. It is probably a good idea to go through and update your encryption keys for the pool. We have seen a user loose access to the pool after doing a disk replace and reboot. Encryption can be very tricky and it is designed to keep you out if you are not properly authorized, so you want to be sure you have all the keys to get back in.

I'm using encryption with keys only, password is blank just so I don't need to worry about anything if I have to RMA a drive although Raid-Z1 should already do that.

Maybe the fact that the encryption password is blank is why the GUI failed.

The password prompt in shell was due to running Sudo, was not letting me bring the drive up without it.

I can update the keys, however in this case it should have remained the same as the disk was not actually replaced, simply I removed it from the hot swap bay with the system running and reconnected it back for testing purposes.

If I can reproduce this issue, where would I submit it?

 

[Chris Moore]

If I can reproduce this issue, where would I submit it?

Here is the link to the manual, but it has not been updated since the bug reporting system was changed.
https://www.ixsystems.com/documentation/freenas/11.2/system.html#support

Here is the link to the new bug tracker: https://jira.ixsystems.com/projects/NAS/issues/NAS-101365?filter=allopenissues

 

[Apollo]

Normally, if I am not mistaken, removing the drive then inserting it back, upon reboot the drive should still be enable to reattach itself.
Even though the drive is encrypted, it must have a means to be recognized before decryption takes place.

 

[Chris Moore]

Normally, if I am not mistaken, removing the drive then inserting it back

If the drive is not encrypted, you can take it out and put it back in and the pool will automatically resilver with no user intervention required.

upon reboot the drive should still be enable to reattach itself.

In hot-swap bays, you don't need a reboot, it just picks it up and goes.

Even though the drive is encrypted, it must have a means to be recognized before decryption takes place.

No. It just doesn't do that. It is only because it is encrypted that something needed to be done here. FreeNAS is kind of dumb in some ways. But it is a good thing. It does not "know" what credentials to use on the encrypted disk you put in and it does not attempt to use the credentials it already has for the other disks. FreeNAS must be told what to do. If you put a fully new disk in, it would not automatically do anything with it (like start to resilver) you would need to give instructions to tell FreeNAS how to behave. Part of that 'dumb' behavior is on purpose because you might have some other plan in mind for that disk and FreeNAS is giving you the chance to choose.

 

[RedWing82]

Hello, so I appear to be having a somewhat similar issue. I recently decided to move FreeNAS to a new Gigabyte GA-7PESH2 motherboard and dual XEON processors. After setting everything up in my new case I transferred my drives (6 x 4TB WD Reds) from my previous FreeNAS box. They are encrypted and I have the backup keys. One of the drive cables was loose when I started up FreeNAS for the first time. I got the message that my pool was degraded. I went back, plugged in the cable but the drive still shows as disabled. When I use the GUI to online the drive I get

Code:

Error: Traceback (most recent call last):

  File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 219, in wrapper
    response = callback(request, *args, **kwargs)

  File "./freenasUI/api/resources.py", line 890, in online_disk
    notifier().zfs_online_disk(obj, deserialized.get('label'))

  File "./freenasUI/middleware/notifier.py", line 1064, in zfs_online_disk
    assert volume.vol_encrypt == 0

AssertionError

If I run zpool status and zpool online from the shell I get

Code:

root@freenas:~ # zpool status Volume_1
  pool: Volume_1
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 0 days 05:41:52 with 0 errors on Sun Mar 31 05:41:54 2019
config:

        NAME                                                STATE     READ WRITE CKSUM
        Volume_1                                            DEGRADED     0     0     0
          raidz2-0                                          DEGRADED     0     0     0
            gptid/0bf4a45c-0f3e-11e8-bcef-d05099c0ad5a.eli  ONLINE       0     0     0
            gptid/0d62b0cb-0f3e-11e8-bcef-d05099c0ad5a.eli  ONLINE       0     0     0
            gptid/0ed639aa-0f3e-11e8-bcef-d05099c0ad5a.eli  ONLINE       0     0     0
            gptid/105faa6a-0f3e-11e8-bcef-d05099c0ad5a.eli  ONLINE       0     0     0
            11293198003029721267                            UNAVAIL      0     0     0  was /dev/gptid/11cc547b-0f3e-11e8-bcef-d05099c0ad5a.eli
            gptid/1338d78a-0f3e-11e8-bcef-d05099c0ad5a.eli  ONLINE       0     0     0

errors: No known data errors
root@freenas:~ # zpool online Volume_1 11293198003029721267
warning: device '11293198003029721267' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present

I don't get the option to put a password in. Do you think the drive is faulty now? I've never had a problem before this and it just seems like the situation is very similar.

Thanks