Help needed - Pool is OK, but drives no longer show in UI

[Craig White]

Hi members,

Ok, so I made a mistake just before the holidays... I used the command line to replace/resilver my failing drives. The pool is operating fine, but now nothing shows in the GUI for drives or volumes. I'd like to get that working again, so I'm looking for advice.

Background / setup info..

I've had a long running home FreeNAS system (still on v9.2.1.9), it's a 4 drive Raidz1 config with 3TB WD Red drives.

One drive was failing horribly and a second was getting errors. I panicked and rushed out to get replacements right away. Bad timing since I was planning on updating and increasing the capacity of the system in a few weeks over the holidays. Anyway, I picked up 4x 6TB drives. However, I replaced the two failing drives with the command line zpool tools instead of the GUI. (Yes, I now know that was a big f-up! Had a lot going on at the time and was so worried about losing my data). Anyway, I've read this can be fixed by replacing the drive 'in-place' again using the GUI, but unfortunately, the drives don't show up anymore in the GUI. (View Disks and View Volumes are both blank pages).

So, I'm looking for advice.

First - does anyone know why the UI wouldn't show the disks, or how to debug this?

I was thinking of trying to update to a newer version of FreeNAS to hopefully help. Does this seem like a good approach?
I currently have a HD as the boot disk, so I'm guessing I could just unplug it and boot from a USB key, that way I can revert if anything is an issue. Is my thinking here correct - as long as I don't upgrade the ZFS pool?

Is it best to start the USB key at 9.2.1.9 and then upgrade from there by doing the "Save Config" / "Upload Config".

Here's the current zpool status: You can see the 2 drives I replaced are showing as ada2 & ada4. The other two are still the older 3TB drives.

Code:

    
NAME                                            STATE     READ WRITE CKSUM
    ocean                                           ONLINE       0     0     0
      raidz1-0                                      ONLINE       0     0     0
        gptid/b482690c-f05a-11e3-87ca-001cc0bf19b4  ONLINE       0     0     0
        ada2                                        ONLINE       0     0     0
        gptid/b61dd0bb-f05a-11e3-87ca-001cc0bf19b4  ONLINE       0     0     0
        ada4                                        ONLINE       0     0     0

Any advice is appreciated!

Thanks,
Craig

 

[Etorix]

I would try and solve the issue with the current version before updating FreeNAS.

Have you tried exporting the pool and reimporting from the GUI?

 

[Craig White]

Thanks for the suggestion. I will look into the export/re-import.
Is it fairly risk-free? Main thing I don't want to lose is the data. Second most important is all the shares setup.

 

[Patrick M. Hausen]

Here's the current zpool status: You can see the 2 drives I replaced are showing as ada2 & ada4.

You have replaced the entire raw disk devices into your pool. You shouldn't. Always create a GPT partition table and use a partition of type freebsd-zfs for the pool. The UI would have done this automatically.

If you want to fix that situation I can help. We will need to take the drives offline, create the partition table, put them in the pool again. But that would put a risk on your data because you have only single redundancy.

A risk free replacement would need one more additional drive.

 

[Craig White]

If you want to fix that situation I can help. We will need to take the drives offline, create the partition table, put them in the pool again. But that would put a risk on your data because you have only single redundancy.

A risk free replacement would need one more additional drive.

Thanks Patrick - Yes, I'm aware I screwed up in the panic! Your help would be appreciated. One reason I didn't replace the other two 3TB drives is to try and clean this up. This means I've got two spare 6TB drives on hand, and there's a spare SATA port on the motherboard.

So, with the right technique, I figured we could replace the two messed up 6TB drives with the new ones (one at a time).

Let me know if this makes sense and how to proceed.
Thanks!

 

[Patrick M. Hausen]

Great. Then please add one of the spare disks and afterwards post the output of zpool status and gpart list.

 

[Craig White]

Great. Then please add one of the spare disks and afterwards post the output of zpool status and gpart list.

I added one the of spare 6TB drives. Here's the output requested, plus a couple others:

Code:

[craig@nas1 ~]$ zpool status
  pool: ocean
 state: ONLINE
  scan: scrub repaired 0 in 9h16m with 0 errors on Mon Jan 18 12:21:04 2021
config:

    NAME                                            STATE     READ WRITE CKSUM
    ocean                                           ONLINE       0     0     0
      raidz1-0                                      ONLINE       0     0     0
        gptid/b482690c-f05a-11e3-87ca-001cc0bf19b4  ONLINE       0     0     0
        ada2                                        ONLINE       0     0     0
        gptid/b61dd0bb-f05a-11e3-87ca-001cc0bf19b4  ONLINE       0     0     0
        ada5                                        ONLINE       0     0     0

errors: No known data errors

Code:

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 625142447
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: ada0s1
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 32256
   Mode: r0w0e0
   rawtype: 165
   length: 988291584
   offset: 32256
   type: freebsd
   index: 1
   end: 1930319
   start: 63
2. Name: ada0s2
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 988356096
   Mode: r1w0e1
   attrib: active
   rawtype: 165
   length: 988291584
   offset: 988356096
   type: freebsd
   index: 2
   end: 3860639
   start: 1930383
3. Name: ada0s3
   Mediasize: 1548288 (1.5M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1976647680
   Mode: r0w0e0
   rawtype: 165
   length: 1548288
   offset: 1976647680
   type: freebsd
   index: 3
   end: 3863663
   start: 3860640
4. Name: ada0s4
   Mediasize: 21159936 (20M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1978195968
   Mode: r1w1e2
   rawtype: 165
   length: 21159936
   offset: 1978195968
   type: freebsd
   index: 4
   end: 3904991
   start: 3863664
Consumers:
1. Name: ada0
   Mediasize: 320072933376 (298G)
   Sectorsize: 512
   Mode: r2w1e4

Geom name: ada0s1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1930256
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: ada0s1a
   Mediasize: 988283392 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 40448
   Mode: r0w0e0
   rawtype: 0
   length: 988283392
   offset: 8192
   type: !0
   index: 1
   end: 1930256
   start: 16
Consumers:
1. Name: ada0s1
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 32256
   Mode: r0w0e0

Geom name: ada0s2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1930256
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: ada0s2a
   Mediasize: 988283392 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 988364288
   Mode: r1w0e1
   rawtype: 0
   length: 988283392
   offset: 8192
   type: !0
   index: 1
   end: 1930256
   start: 16
Consumers:
1. Name: ada0s2
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 988356096
   Mode: r1w0e1

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   rawuuid: b471524a-f05a-11e3-87ca-001cc0bf19b4
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada1p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   rawuuid: b482690c-f05a-11e3-87ca-001cc0bf19b4
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: ada1
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: ada4
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada4p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   rawuuid: b60c1fcd-f05a-11e3-87ca-001cc0bf19b4
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada4p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   rawuuid: b61dd0bb-f05a-11e3-87ca-001cc0bf19b4
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: ada4
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Code:

[craig@nas1 ~]$ glabel status
                                      Name  Status  Components
                             ufs/FreeNASs3     N/A  ada0s3
                             ufs/FreeNASs4     N/A  ada0s4
                    ufsid/53580c07160b47c5     N/A  ada0s1a
                            ufs/FreeNASs1a     N/A  ada0s1a
                            ufs/FreeNASs2a     N/A  ada0s2a
gptid/b482690c-f05a-11e3-87ca-001cc0bf19b4     N/A  ada1p2
gptid/b61dd0bb-f05a-11e3-87ca-001cc0bf19b4     N/A  ada4p2

The new drive (based on S/N) came in at /dev/ada3

Code:

[craig@nas1 ~]$ sudo smartctl -i /dev/ada3
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p15 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD60EFRX-68L0BN1
Serial Number:    WD-WX11D3778E9F
LU WWN Device Id: 5 0014ee 20eb9af2a
Firmware Version: 82.00A82
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Tue Jan 19 13:08:57 2021 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

 

[Patrick M. Hausen]

1. gpart backup ada1 | gpart restore -F ada3
2. gpart list ada3

 

[Craig White]

1. gpart backup ada1 | gpart restore -F ada3
2. gpart list ada3

Code:

[root@nas1] /mnt/ocean/users/craig# gpart list ada3
Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 11721045134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: ccedd414-5a89-11eb-8f32-001cc0bf19b4
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada3p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: cd57e9ce-5a89-11eb-8f32-001cc0bf19b4
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: ada3
   Mediasize: 6001175126016 (5.5T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

 

[Patrick M. Hausen]

So we replace ada2 with the second partition of your new disk:
zpool replace ocean ada2 gptid/cd57e9ce-5a89-11eb-8f32-001cc0bf19b4

If that works as intended you can monitor the resilver progress with zpool status ocean.

 

[Craig White]

So we replace ada2 with the second partition of your new disk:
zpool replace ocean ada2 gptid/cd57e9ce-5a89-11eb-8f32-001cc0bf19b4

If that works as intended you can monitor the resilver progress with zpool status ocean.

Replace started. Question - the partition layout we copied was from a 3TB drive. Will the partition get expanded to consume the 6TB drive at some point, perhaps when the four drives are done and the autoexpand kicks in?

And if I haven't said it enough - a HUGE thanks for the help!

Code:

[root@nas1] /mnt/ocean/users/craig# zpool replace ocean ada2 gptid/cd57e9ce-5a89-11eb-8f32-001cc0bf19b4
[root@nas1] /mnt/ocean/users/craig# zpool status
  pool: ocean
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Jan 19 14:21:57 2021
        34.4M scanned out of 8.77T at 4.91M/s, 520h16m to go
        8.45M resilvered, 0.00% done
config:

    NAME                                              STATE     READ WRITE CKSUM
    ocean                                             ONLINE       0     0     0
      raidz1-0                                        ONLINE       0     0     0
        gptid/b482690c-f05a-11e3-87ca-001cc0bf19b4    ONLINE       0     0     0
        replacing-1                                   ONLINE       0     0     0
          ada2                                        ONLINE       0     0     0
          gptid/cd57e9ce-5a89-11eb-8f32-001cc0bf19b4  ONLINE       0     0     0  (resilvering)
        gptid/b61dd0bb-f05a-11e3-87ca-001cc0bf19b4    ONLINE       0     0     0
        ada5                                          ONLINE       0     0     0

errors: No known data errors

 

[Patrick M. Hausen]

Question - the partition layout we copied was from a 3TB drive. Will the partition get expanded to consume the 6TB drive at some point, perhaps when the four drives are done and the autoexpand kicks in?

Take care of that after all 4 drives are replaced with their 6 TB counterparts. I assume you can work on your own from here? By copying the partition table you get a swap partition and a freebsd-zfs partition of size 3 TB. The gptid is the "rawuuid" part you get from gpart list.
Just go one disk at a time now until the resilver is complete.

The procedure for resizing for maximum capacity is for e.g. partition 2 of ada3: gpart resize -i 2 ada3. A quick gpart show ada3 should show a 6 TB partition afterwards. Then when you did that for all 4 disks, possibly (I have not yet tried that with OpenZFS 2.0) the zpool will be expanded automatically. For FreeNAS 11 the magic incantation was e.g.: zpool online -e ocean gptid/cd57e9ce-5a89-11eb-8f32-001cc0bf19b4.

Good luck!

 

[Craig White]

Take care of that after all 4 drives are replaced with their 6 TB counterparts. I assume you can work on your own from here? By copying the partition table you get a swap partition and a freebsd-zfs partition of size 3 TB. The gptid is the "rawuuid" part you get from gpart list.
Just go one disk at a time now until the resilver is complete.

The procedure for resizing for maximum capacity is for e.g. partition 2 of ada3: gpart resize -i 2 ada3. A quick gpart show ada3 should show a 6 TB partition afterwards. Then when you did that for all 4 disks, possibly (I have not yet tried that with OpenZFS 2.0) the zpool will be expanded automatically. For FreeNAS 11 the magic incantation was e.g.: zpool online -e ocean gptid/cd57e9ce-5a89-11eb-8f32-001cc0bf19b4.

Good luck!

Thanks again - I will work through re-silvering the disks. So hopefully after I get those first two done, they will show up in the GUI again.
I had succeeded with expanding this system once before. It started with 1TB drives way back!