Moving drives between pools?

[jkingaround]

Semi-complicated situation here so please bear with me.

I have a 4U server with 24 bays / drives. running TrueNAS-12.0-U7 with 3 pools split as follows:
- Pool1: 8 x 8tb
- Pool2: 8 x 4tb
- Pool3: 8 x 4tb

Now, I'm in the process of buying new 8tb drives as replacements for the 4tb drives that fail or when 8tbs go on sale. This is so that when i have replaced all of the 4tbs with 8tbs the pool auto expands (i've done this to expand the size of Pool1 and created Pool3 w/ the replaced 4tbs to fill all my drive bays).

I replaced 2 additional 8tb drives into Pool2 and took 2 x 4tb OUT. All is good at this point and all is working as expected.

However, at some point in the middle of the night one of the 4tbs went offline but a few minutes later i get another email saying the alert had cleared itself. However, something got screwed up and one of the 8tb jumped to Pool3 (genuinely have no idea how this happened). So now I have 1 8tb each in Pool2 and Pool3, where I really want 2 x 8tb in a single pool so it can expand.

so currently:

- Pool2: 7 x 4tb, 1x 8tb
- Pool3: 7 x 4tb, 1x 8tb

but i want this:

- Pool2: 6 x 4tb, 2 x 8tb
- Pool3: 8 x 4tb

I need to move the oddball 8tb over from Pool3 into Pool2, but I'm not sure the best way to do this. I do have old 4tb drives (on the verge of death) but I do not have any spare bays.

I get that this is confusing so please ask questions if anything needs clarification.

Thanks in advance.

 

[EnKo]

It is very curious that the system switched the 8TB drive during replacement from one drive to another. Are you sure you did the replacement process correct? The way I did it recently was to set one old disc offline, removed it, put the new one in the same bay, told the system to replace the old disc by the new on, waited for resilvering and repeated this step for every old disk. You can replace the 8 TB drive from your Pool3 by a 4TB drive in this way as well.

 

[jkingaround]

It is very curious that the system switched the 8TB drive during replacement from one drive to another. Are you sure you did the replacement process correct? The way I did it recently was to set one old disc offline, removed it, put the new one in the same bay, told the system to replace the old disc by the new on, waited for resilvering and repeated this step for every old disk. You can replace the 8 TB drive from your Pool3 by a 4TB drive in this way as well.

I'm pretty sure I did it correctly, and it was working perfectly until there was a hiccup with a 4tb drive that temporarily went offline.

Is there any way to do it in purely software and not physically removing disks and putting a 4tb drive back in? (i don't trust it to fully resilver as it was on it's way out the door)

 

[EnKo]

Now when I think about your problem again, did you replaced an drive of Pool2 and Pool3 at the same time? I replaced mine really one by one within the whole system. You can set the 4TB drive previously in Pool3 and the 8TB drive should be in Pool2 offline and replace the 4TB drive by the 8TB drive and the 8TB drive by the 4TB drive (by software).

/core/coretutorials/storage/disks/diskreplace/

 

[jkingaround]

Now when I think about your problem again, did you replaced an drive of Pool2 and Pool3 at the same time? I replaced mine really one by one within the whole system. You can set the 4TB drive previously in Pool3 and the 8TB drive should be in Pool2 offline and replace the 4TB drive by the 8TB drive and the 8TB drive by the 4TB drive (by software).

/core/coretutorials/storage/disks/diskreplace/

Per that documentation: "This disk must be installed in the TrueNAS system and not part of an existing storage pool."

Yeah, I tried to offline a 4tb from pool2, and offline the 8tb in pool3 but it failed to replace/swap them. I don't think I can do this without a man in the middle drive.

How do I fully remove a disk from a pool? It's still part of the pool when I offline it and when I wipe it. So i think that's why it failed.

 

[EnKo]

In theory you can replace them by setting the "Force" option. If this is not working properly you can remove the disc from your server and delete the filesystems e. g. on a personal computer. Therfore you have to mount it in the pc or use an adapter, if you have one. There should be a build in function for this in TrueNAS as well, but I don't know how to use it. I usually use the graphical interfaces of my pc for these purposes.

 

[artlessknave]

this sounds a bit like you have added the drive by its hardware location somehow, which can change. are all of your drives using GPTID, or are any of them showing, like ada0/da0/nmve0/etc?

Code:

zpool status -v

can really explain things.
if it were me...i would nuke it all and restore from backups...but many of the people asking for help here didn't give themselves that option.

 

[jkingaround]

this sounds a bit like you have added the drive by its hardware location somehow, which can change. are all of your drives using GPTID, or are any of them showing, like ada0/da0/nmve0/etc?

Code:

zpool status -v

can really explain things.
if it were me...i would nuke it all and restore from backups...but many of the people asking for help here didn't give themselves that option.

yeah one of them is listed but it's in the "daXp2" format. i've done all drive replacements through the webui, so i don't think so? but unsure.

just an update:

i ended up having to physically switch drives and i was able to use an 8tb that was slated as a future replacement:
1) offlined a 4tb from pool2, physically hot swap with new 8tb, resilver.
2) offlined the rogue 8tb from pool3, replace with removed 4tb. tried to wipe it via freenas and do a replace without reboot / hotswap like above but failed as it still thought it was part of the old pool since it recognized the serial number). did a longer wipe, rebooted and now pool3 is resilvering and my drives are where they need to be.

still need to put the replaced 8tb back into pool2 but that's after it resilvers and all is good.

 

[artlessknave]

every once in a while someone, somehow, replaces a disk and gets device name instead of the GPTID, and swears it was the GUI.
how that happens, I do not know; the only way I know of to do that is to replace it manually.
IIRC

Code:

gpart destroy -F /dev/disk-you-wont-miss

alternatively, you might be able to make a brand new pool on the disk you want to wipe, and use force to force it. gpart destroy is probably easier though.
merely using wipe commands, or DD, will not get all the GPT off the disk, as GPT stores itself in a few places to make it hard to wipe by accident.