Confusing pool disk replacement

[marcevan]

AFAIK the guides say to offline a disk add a new one and do a replace.

If I have 8 bays and 8 disks in there, it's impossible. I have to offline a disk then physically replace it.

I did that and used Zpool add Media da5

But in the GUI it shows the disk da5 (in Storage / Disks) as n/a and not in Media.

In CLI, zpool status Media:


Media DEGRADED raidz2-0 DEGRADED gptid ONLINE gptid ONLINE gptid ONLINE gptid ONLINE gptid ONLINE da5 ONLINE


They are all online, but da5 didn't make it into the RaidZ2. It's on the same indent as raidz2-0

 

[danb35]

AFAIK the guides say to offline a disk add a new one and do a replace.

No, they don't. Assuming the disk you're replacing isn't completely dead, there are two documented methods:

• If you have a spare bay
  • Install the new disk
  • In the GUI, go to pool status, select the disk to replace, in the drop-down menu choose Replace, select the new disk in the pop-up window
  • Wait for resilver to complete
  • Remove the old disk
• If you don't have a spare bay
  • Offline the old disk
  • Remove the old disk
  • Install the new disk
  • In the GUI, go to pool status, select the (offline) disk to replace, in the drop-down menu choose Replace, select the new disk in the pop-up window
  • Wait for resilver to complete

In no case should you be doing anything at the CLI.

Zpool add Media da5

Yeah, you just killed your pool's redundancy (edit: and since you decided to do it at the CLI, you didn't have any warning--the GUI would have warned you, and might not have even let you do it at all). You've added da5 as a single-disk vdev, striped with your degraded RAIDZ2 vdev. When da5 fails, you'll lose all the data on your pool. You also failed to properly partition the new disk, but that's a lesser issue.

 

[marcevan]

I first went to the pool in the GUI but the disk missing (former da1) was gone so couldn't be "replaced"

This same as I was seeing in reddit (offline a disk and it's poof gone!).

So is there a way to do this one more time? E.g., shut down and attach the old failing drive on SATA from mobo and see if it gets a replace option?

 

[danb35]

I first went to the pool in the GUI but the disk missing (former da1) was gone so couldn't be "replaced"

There should have been a placeholder, like this:

So is there a way to do this one more time?

Unfortunately, no. vdev removal landed with FreeNAS 11.2, IIRC, but it's only applicable when the pool consists only of mirror or single-disk vdevs. Once you've added a single-disk stripe vdev to your RAIDZn vdev, your only option is to back up your data, destroy and rebuild the pool, and restore.

 

[marcevan]

I think what happened was the SSD that was attached (purpose to mirror to USB to replace the USB - but there's threads about that not working so well) was added as it's also there with status unavail.

There was no /dev/gptid/____________ FAULTED entry

Now ironically I've been debating removing the LSI card (SAS/SATA) as while it's got 8 Sata port connections, it's only been reading 6 of my 8 disks anyway. I can literally use 8 of the SATA ports on the motherboard, form a new pool (Content) and then copy the data from pool Media to pool Content then I'd be whole again but just on mobo SATAs.

Would that work?

 

[danb35]

Would that work?

Don't see why not.

 

[marcevan]

Ahh, got a break. Shut it down and put the "offline" disk back on mobo sata.

Rebooted and it came up "offline" ready to replace. Will turn off and add another disk on mobo Sata and use that as replacement and the pool should be back.

Then I'll do one more shut down and put all the disks on mobo and will have 8 of them so can expand the pool to the +2 more.

 

[marcevan]

Progress in many fronts:

Drives/Pools

1. The 2 4TB drives that LSI nor mobo would read were DOA. Tried in a 2nd system and weren't even seen in BIOS - RMA
2. I used that space to resilver one bad sector 3TB drive with new 4TB drive
3. Now have one more 4TB drive give CAM and sector issues, so resilvering that for next ~6 hrs then will remove and return 3 WD 4TB drives to NewEgg under RMA tomorrow.
4. I will then have space for 1 4TB drive again (6 on pool, 1 dangerously new vdev unbalanced to raidz2) so will add that as mirror in the zdev using zpool attach poolname olddisk newdisk and old = da5 and new will be TBD but probably da8

Big thanks to Chris Moore who had a few 4TB Seagate Constellation disks on the ready so will use until they experience errors and likely be in 6 or 8TB land at that time.

CPU Cooler

1. My bad in getting the absurdly large Noctua NH-U14S CPU cooler which is about 6mm above my chassis sides.
2. New Noctua arrives today so will time that with Drives/Pools Step #3

Racking

1. Supermicro board wouldn't boot up fully when racked and believe that's because it needs dedicated mouse, keyboard, and video (possibly 2 of 3) so have spare USB keyboard and mouse and can bring those to rack. Now issue is video. I had rejumpered JMG1 to 2-3 for HDMI (works on my office monitor) but whenever it was in 1-2 it's for VGA and wouldn't give video to my KVM screen. Will try to attach directly with VGA/VGA cable from motherboard to KVM instead of the KVM's cable.

Will post update after racking attempt as Drives/Pools/CPU cooler should be done by 4pm (EDT). Then onto racking!

 

[danb35]

zpool attach poolname olddisk newdisk

If you're using TrueNAS, you should be able to do this from the GUI--you really shouldn't be messing with your pool from the CLI, though I guess if you're about to scrap the pool anyway that isn't as critical.

 

[marcevan]

Got it.

I'll use Storage/Pools/Status, and tag da5 with "extend" and use new 4TB drive to be attached post re-silvering, reboot.

That will add a 2nd disk to Vdev, but will that force a mirror or just JBOD 2 disks?

 

[danb35]

It would be a mirror.

 

[marcevan]

Gracias for all the help...none taken too personally and direct feedback is appreciated.

 

[marcevan]

Status and remaining issues:

1. Chassis was racked last night after CPU cooler installed, Media pool resilvered and VDEV single 4TB drive was mirrored.
2. Returning 3 of 4 4-TB drives (Western Digital NAS Red 3.5") that had 2 DOA and 1 sector and CAM issues. Worst single-purchase luck ever.
3. Identified that VGA port on motherboard is for IPMI only. If I setup IPMI with a different IP than TrueNAS it becomes the TrueNAS IP. Went with disabled IPMI and nothing on VGA so it's headless which isn't terrible since I can ssh in and have GUI.
4. 10G Intel X550-v2 is still showing 1000T on TrueNAS dashboard. Should be 10G-T but since I don't have a 10G client nor switch this is a future problem unless someone has a solution.