Need help with trying to replace the drive to grow the pool

[doutatsu]

Hello everyone o/

I've had my FreeNAS system for a while, with a RAIDZ1 setup of 3 WD Reds each of 2TB. As I was running out of storage, I wanted to replace disks one by one to grow the pool, following the guide here. Now, because I didn't have any more open SATAs on the machine, I used an external enclosure to connect the drive through a USB3. All I went good, I used the replace drive through the GUI, went through resilver and saw my pool green, with the replaced drive.

The issue came when I went to remove the old disk, and then put the disk from the enclosure inside the NAS. After turning on the system again, my disk was unavailable and the pool degraded... This is how the Pool Status looked:

And then looking into disks:

So after trying a couple of commands, checking the zpool status through ssh, I got to this error:

One or more devices could not be used because the label is missing or invalid

Looking more into this, it seems that guid wasn't assigned to the new drive, hence it can't be just moved across? This is where I am unsure of what is going on and how to fix this. I tried exporting and importing the pool, as some suggested, to no avail. I think I can manually label the drive through bash, but that did sound like the right thing to do. I would really appreciate help with this

 

[garm]

You used USB to resilver the drive.. hence ZFS knows nothing of the drive when you put it back in. USB dosent expose the SATA bus to the system but emulates a drive of the same size. This is one of many reasons people keep saying not to use USB with ZFS.
You need to put the pool back in working order if possible and use a proper connection to redo the resilver.

 

[doutatsu]

You need to put the pool back in working order if possible and use a proper connection to redo the resilver.

Well, I don't have any other SATA ports, so this was the only way to do it. Is it not possible to just fix the labelling? The data integrity is there, the poll works correctly while connected through the USB. Surely, there is a way to just move the drive into the NAS without having to do resilvering again?!

I've tried btw to add the old disk back, but perhaps because the data changed since then, its now also unavailable. So if I to do a resilver, it'll have to be with just 2 drives in place. Can I just wipe this new, 3rd disk, connect it into the NAS and then get the data just from the 2 disks?

 

[danb35]

Well, I don't have any other SATA ports, so this was the only way to do it.

No, it isn't--you could always offline one of your existing disks. That isn't ideal, as you lose redundancy that way, but it's certainly a possibility.

3rd disk, connect it into the NAS and then get the data just from the 2 disks?

Yes, of course, as long as there aren't any data errors on those two disks--that's the point of RAIDZ.

Sometimes USB enclosures/docks do stuff (technical term) with the data, such that what's seen when the disk is plugged into the dock isn't the same as what's seen when it's plugged into SATA. Sometimes they don't. My general (albeit limited) experience has been "they don't", but that clearly isn't what's happening with you.

 

[doutatsu]

Yes, of course, as long as there aren't any data errors on those two disks--that's the point of RAIDZ.

No there are none. So I am not going to lose any data if I do that? I get that's the point of the redundancy, but I feel like resilvering from the degraded state might have its risks?

 

[danb35]

So I am not going to lose any data if I do that?

No, you shouldn't lose any data--as you say, that's the point of redundancy. There is a risk, and that's the risk that there is a data error (or disk failure) while you're resilvering--this is one of the reasons we generally recommend RAIDZ2 over RAIDZ1. But barring disk failure or data error, you'll have all your data.

 

[doutatsu]

Didn't know RAIDZ2 is recommended, maybe I should expand the number of drivers I have in the future and switch to it.

I've had a suggestion to fix this issue by doing a manual import, but changing how these are labelled. Specifically by doing an import with this flag: `zpool import -d /dev/gptid Void`, so that the pool will use partuuid based labelling instead of kernel name-based. Does that make sense?

 

[doutatsu]

Alright, for the sake of keeping the history on what I did: I've decided to try the relabelling. What I did is as follows:

Code:

# Exported the pool in GUI
1. zpool import -d /dev/gptid Void
2. zpool status -v Void
# online and showing GPTIDs instead of volatile paths like sd* or ada*
# Power Off
# Swap Disk from USB to SATA
# Power On
3. zpool status -v Void
# online since the GPTIDs don't change when changing slots

Unfortunately, this didn't quite work out on the swap side. Doing the import did assign a `gptid` to that new disk. But after the swap, I was back to that disk being unavailable. Moreover, doing import in CLI made it so that GUI was unaware of the pool import and hence it was not shown on the GUI.

I had to put the HDD back into the enclosure, then export on the CLI, to then import on the GUI to get back to where I was, with a healthy RAID. The unique GPTIDs did remain though:

At this point, it seems like resilvering, as if that drive failed, is the only way to fix this now...

 

[doutatsu]

To close off this thread:

I went ahead and did a resilver. I realised I did an extra SATA port hidden away, so I first replaced the new disk with the old one, so that its back to the state the NAS was before I attempted to grow the pool. I then followed the same steps as in documentation, except using on mobo SATA port instead of the enclosure. All went well and I could then replace the old disk with the new disk, this time with labels in order and with no issues.

So the moral of the story: Do not use HDD enclosure to replace a disk, make sure to use the onboard connection. Lesson painfully learned