Replacing Disk does not work in TrueNAS Core v13.0?

[Koala166]

Hello, I am new to TrueNAS/ZFS with zero experience in UNIX so I decided to to some testing before I entrust my data on it. I used TrueNAS Core 13.0 Release version. The hardware I use is not relevant so I won't bother you with unnecessary details.

So for my tests I created a ZVolume with 5 drives in ZRAID 1 configuration, and added some files to it. Everything worked as expected.

Then I tried to simulate a catastrophic drive loss: I booted into Windows, and I formatted one of the 5 drives. I wanted to see how the drive replacement and resilvering procedure would work in practice.

I booted back into TrueNAS, and I saw that the system correctly warned me that one drive in the Zpool was damaged/missing. So then I tried to replace it following the instructions in the online manual.

It did not work. Clicking the 'Replace Disk' button does absolutely nothing (with either the "Force" checkbox enabled or not). No error message, no nothing. The "Replacing disk" Dialog box remains open with no feedback at all. (The 'Cancel" button works for what's worth.)

I know TrueNAS is free and I can't expect everything to work as intended, but as first impressions go this is not ideal.

 

[Samuel Tai]

Unfortunately, this is a known issue with the 13.0-RELEASE, and proves once again why you shouldn't ever use a .0 release for anything mission critical. There is a work-around with a CLI tool: https://www.truenas.com/docs/core/corereleasenotes/#cli-disk-replacements.

If you want reliability and a better experience, try 12.0-U8.1 instead.

 

[Koala166]

Thank you very much for letting me know. I am very surprised that such a serious bug made it on a stable release channel and has not yet been patched with a hotfix. This is the equivalent of releasing a car that does not allow you to replace its tires.

Since I have zero prior experience with TrueNAS, I have to ask: is it usual to have such serious bugs on core features on the "stable" release channel?

Also, how easy is it to downgrade to version 12.0-U8.1? Do I just install it on top of version 13, or do I have to do some preparation to save the TrueNAS settings? (Not that I mind losing everything and starting from scratch since this NAS has no data on it. But I'd like to know in case of future issues where I have to downgrade and preserve all my data and settings.)

 

[Samuel Tai]

Save your settings under System->General->Download Config. After the reinstall with 12.0-U8.1, restore your settings under System->General->Upload Config. Remember to save secrets with the config to preserve passwords.

As for functionality bugs on .0 releases, this is par for the course in the entire software industry. TrueNAS isn't an especially egregious offender here. The Release Notes for 13 don't recommend using 13.0 for critical deployments until 13.0-U2 or later.

You did read the release notes, didn't you? If not, then you're guilty of installing under false assumptions, and you should learn from this experience for next time you deal with server software.

 

[Koala166]

Thank you for all the info. This is the fist time I used any kind of NAS software, and I assumed that core features were tested and working on stable releases. Now I know better. I will downgrade and will not install new versions until I read in the forum that they are safe to use.

Again, thank you very much for your time!

 

[danb35]

TrueNAS isn't an especially egregious offender here.

iX have a bad habit of shipping "release" versions of their software with what should have been show-stopping bugs, going back (at least) to the days of 9.2. Yeah, they're a pretty egregious offender. This doesn't quite rise to the level of The Release That Must Not Be Named (in which disk replacement wasn't in the GUI at all), but this is a pretty major fail on their part. Again.

And sure, we can say "never install .0" (and lots of us do), but that's frankly stupid--it suggests that it's OK for them to ship badly-broken software as though it were release-ready, and it's the users' fault for believing them.

 

[PPatla]

I tried the release notes CLI instructions, but I am getting an error that partitions exist on the replacement drive.

I previously completed a wipe in the GUI of the drive to be the replacement drive and it said the wipe successfully completed.

Is there a "force" extension that can be used with the replace drive script like the option that is available in the GUI version of replace?

Seems like I may have to wait for U1 to upgrade my 2x4TB drive mirror to 2x8TB drive mirror.

Thanks,
Pat

 

[Samuel Tai]

I tried the release notes CLI instructions, but I am getting an error that partitions exist on the replacement drive.

I previously completed a wipe in the GUI of the drive to be the replacement drive and it said the wipe successfully completed.

Is there a "force" extension that can be used with the replace drive script like the option that is available in the GUI version of replace?

Seems like I may have to wait for U1 to upgrade my 2x4TB drive mirror to 2x8TB drive mirror.

Thanks,
Pat

From the Shell, run gpart destroy -F <disk device> to whack the drive's partition table. For example, if your replacement disk is ada3, run gpart destroy -F ada3. If you typo the drive ID, you'll whack the partition table on the wrong drive. There is no safety net here.

 

[PPatla]

root@freenas:~ # gpart destroy -F ada2
gpart: arg0 'ada2': Invalid argument
root@freenas:~ #

That didn't work for me. I did it both from the shell and I SSH'd into the system.

 

[sretalla]

gpart destroy -F ada2

did you mean gpart destroy -F /dev/ada2 ? I guess that would have worked better.

 

[husky1055]

I would like to thank the OP for this thread. I am a freenas/truenas user for a fairly long time and have replaced and resilvered disks a number of times. Installed 13.0, read the notes but never did I suspect that replacing a disk would not work. Was about to replace a disk with a bigger capacity. So happy I found this thread!!!

 

[Rakrul]

I'm running TrueNAS-13.0-U1 and replace disk worked fine for me. I have 4 x 4TB disks and I replaced a 4 TB disk with a 16TB disk in the hopes of increasing capacity, but it didn't change. I can see the new 16TB disk is in the pool, it's online and everything looks dandy, so is this just a silly misunderstanding on my end where I need to replace at least half of the disks to increase storage capacity?

(I'm pretty clueless when it comes NAS in general and zfs/TrueNAS specifically, so feel free to point me in the right direction and I'll RTFM. :)

 

[Redcoat]

so is this just a silly misunderstanding on my end where I need to replace at least half of the disks to increase storage capacity?

Actually you need to replace ALL of the disks before the pool capacity will increase.

 

[Rakrul]

Actually you need to replace ALL of the disks before the pool capacity will increase.

Oh ok. I guess I have to wait for this to finish then. :)
Thanks for the fast reply!

 

[danb35]

Oh ok. I guess I have to wait for this to finish then. :)
Thanks for the fast reply!

That feature won't do what you're asking about. RAIDZ expansion will let you turn, say, a 5-disk RAIDZ into a 6-disk RAIDZ. It won't increase the capacity of that 5-disk RAIDZ when you replace a single disk in it with a larger one.

 

[husky1055]

It's made clear that replacing a disk with a larger capacity will not increase the pool capacity. To increase the capacity of the pool one has to replace all disks with the larger capacity disks. This I have done. To add a disk to a pool, can I just create a vdev with the new disk and expand the pool with the new vdev. I was never able to expand my pool short of deleting it and create a new one with the additional disk, then risk of losing all my data.

 

[Rakrul]

That feature won't do what you're asking about. RAIDZ expansion will let you turn, say, a 5-disk RAIDZ into a 6-disk RAIDZ. It won't increase the capacity of that 5-disk RAIDZ when you replace a single disk in it with a larger one.

Yeah I know, but then I can add my 16 TB disk to the pool. I currently have 4 x 4 TB disks in the pool (I replaced the 16TB disk back to the recently unused 4 TB disk again so it's back to the original 4 x 4TB setup).
Hopefully expanding the pool with a disk (or two) doesn't have to be of the same size of the disks already in the pool? I.e. I have 4 x 4TB in the pool and want to expand the pool with either one or two 16TB disks. I know some capacity will probably go to waste due to data parity, if the feature allows this.

It's made clear that replacing a disk with a larger capacity will not increase the pool capacity. To increase the capacity of the pool one has to replace all disks with the larger capacity disks. This I have done. To add a disk to a pool, can I just create a vdev with the new disk and expand the pool with the new vdev. I was never able to expand my pool short of deleting it and create a new one with the additional disk, then risk of losing all my data.

My problem is that I have 4 disks in the pool, and my chassis can fit max 6 disks. Obviously I didn't read the manual well enough to realize that I can't expand disks after the pool is created. However this is the feature they're working on, that I linked to.
I actually have 6 disks in my chassis now, but only 4 is in use.

 

[Patrick M. Hausen]

If you add a single disk to the pool as a single disk vdev you will get the added capacity but will lose all redundancy. That disk fails, your data is toast. All of it.

 

[Rakrul]

If you add a single disk to the pool as a single disk vdev you will get the added capacity but will lose all redundancy. That disk fails, your data is toast. All of it.

I don't think we're talking about the same thing now? I'm talking about the zfs expansion feature that's being developed.

RAIDZ Expansion feature by ahrens · Pull Request #12225 · openzfs/zfs

Motivation and Context This feature allows disks to be added one at a time to a RAID-Z group, expanding its capacity incrementally. This feature is especially useful for small pools (typically wit...

github.com

It would allow me to expand the pool with additional disks, but I'm not sure if it allows me to add different sizes of disks than already in the pool.

 

[Patrick M. Hausen]

You can add a larger disk but only the size of the smallest disk in the RAIDZ vdev will be used.