Replacing Disk does not work in TrueNAS Core v13.0?

[Rakrul]

You can add a larger disk but only the size of the smallest disk in the RAIDZ vdev will be used.

Damn.
Thanks for clarifying though.

 

[Patrick M. Hausen]

RAIDZ expansion means you can turn e.g. a 4 disk RAIDZ2 into a 5 disk RAIDZ2. All other constraints still hold, it's still a RAIDZ2. Like all disks have to be of the same size or the smallest one will determine the space used on all of them.

 

[Rakrul]

Yup. Apparently my reading abilities are non-existent. :) Thanks again.

 

[Arwen]

Let us be clear on one point. Growing storage in a pool is by vDev.

For example, replacing all the disks in a single vDev with larger disks, will increase the storage of a pool by the amount of the smallest disk remaining in the vDev. So changing out 2 x 4TB disks in a Mirror vDev with a 6TB & a 8TB, will only increase storage by 2TB, which is available on both the new replacement disks.

Or adding a vDev, of storage type will increase storage. Storage type does not include SLOG, L2ARC, Special DeDup and may not include Special Meta Device(s), depending on if small files are allowed on the Special Meta Device(s).

So a server with 6 disk slots, that uses 2, 2 x Mirrored vDevs can grow either by replacing a both disks in a single vDev. Or adding another pair of disks in a new vDev.

Thus, many times we ask for the hardware and software configuration when people ask questions.

 

[Redcoat]

Apparently my reading abilities are non-existent. :)

Actually, they're probably fine. "all disks have to be of the same size or the smallest one will determine the space used on all of them" - this characteristic of a vDev isn't mentioned in the current TN13 manual. It was in TN11's manual but didn't make it into this one that I (or a pdf search tool) can find.

 

[dashtesla]

Literally just had a drive failure and I went for a quick hard drive swap while my food is waiting and was faced with all this kinda ruined my evening.. but still there's more:

Last login: Wed Sep 7 03:13:20 on pts/1
FreeBSD 13.1-RELEASE-p1 n245406-814eb095751 TRUENAS

TrueNAS (c) 2009-2022, iXsystems, Inc.
All rights reserved.
TrueNAS code is released under the modified BSD license with some
files copyrighted by (c) iXsystems, Inc.

For more information, documentation, help or support, go here:
http://truenas.com
Welcome to TrueNAS

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@vm7[~]# python3 replace_disk.py POOL-B-Z1 /dev/gptid/b05d4c81-ee2a-11ec-861e-00155d75a501 da2
Traceback (most recent call last):
File "/root/replace_disk.py", line 44, in <module>
main()
File "/root/replace_disk.py", line 40, in main
print('Replace initiated.')
File "/root/replace_disk.py", line 19, in main
assert '13.0-RELEASE' in c.call('system.version')
AssertionError
root@vm7[~]#

Instructions from here: https://www.truenas.com/docs/core/corereleasenotes/#cli-disk-replacements

 

[dashtesla]

Save your settings under System->General->Download Config. After the reinstall with 12.0-U8.1, restore your settings under System->General->Upload Config. Remember to save secrets with the config to preserve passwords.

As for functionality bugs on .0 releases, this is par for the course in the entire software industry. TrueNAS isn't an especially egregious offender here. The Release Notes for 13 don't recommend using 13.0 for critical deployments until 13.0-U2 or later.

You did read the release notes, didn't you? If not, then you're guilty of installing under false assumptions, and you should learn from this experience for next time you deal with server software.

.1 now. What's next .2 .3??? :(

Truly disappointed to read this 'official' statement, It's certainly not standard for the software industry to release a major feature breaking bug on a stable release it's basic stuff that would've been tested before an official release, there's beta branches for that and if it's not ready then it should stay beta until it is.

 

[dashtesla]

So I got tired of trying to fix that python script as it's not my job anyway so I went ahead and replaced the disk manually for anyone that might need it this is what it was like for me but you'd need your own values obviously:

root@vm7[~]# zpool replace POOL-B-Z1 /dev/gptid/b05d4c81-ee2a-11ec-861e-00155d75a501 da2
root@vm7[~]#

It will hang the terminal for a few seconds and then return normally. Reboot.

 

[sretalla]

The GUI should be working since U1... have you tried it?

 

[dashtesla]

The GUI should be working since U1... have you tried it?

Obviously... (Severus Snape). And no it still doesn't work for me which is the entire reason i posted I updated to the latest version before trying to replace the disk.

Last login: Wed Sep 7 03:13:20 on pts/1
FreeBSD 13.1-RELEASE-p1 n245406-814eb095751 TRUENAS

you can see it above.

 

[sretalla]

Obviously...

What's not obvious at all is that you tried that at the GUI...

I can see just fine that you tried running the script (according to the provided instructions for 13.0, hence using a script intended only for 13.0), but that script fails due to a check of uname -a (or similar) not matching that version on your system (seems you're on U2).

What I'm asking is that you stop using the script on versions after 13.0 (U1 and U2) and use the GUI (and nothing else).

The issue with the GUI is documented on the release notes page you linked as being fixed in U1 (https://ixsystems.atlassian.net/browse/NAS-116217)

I imagine that zpool status -v shows your pool has a strange status now for that disk.

 

[dashtesla]

What's not obvious at all is that you tried that at the GUI...

I can see just fine that you tried running the script (according to the provided instructions for 13.0, hence using a script intended only for 13.0), but that script fails due to a check of uname -a (or similar) not matching that version on your system (seems you're on U2).

What I'm asking is that you stop using the script on versions after 13.0 (U1 and U2) and use the GUI (and nothing else).

The issue with the GUI is documented on the release notes page you linked as being fixed in U1 (https://ixsystems.atlassian.net/browse/NAS-116217)

I imagine that zpool status -v shows your pool has a strange status now for that disk.

Well I am running the latest version and it didn't work for me so clearly hasn't been fixed otherwise I wouldn't have bothered coming here right?

I can't test any further as I just did it manually and as for status well it's some junk smr seagate drives so it's still at 90% resilver but that's no one's fault.. and the pool is fine. I will run the status later after the resilver is done.

 

[sretalla]

Well I am running the latest version and it didn't work for me so clearly hasn't been fixed otherwise I wouldn't have bothered coming here right?

You still haven't addressed the first question I asked you with a clear response.

The script is meant only for people running 13.0 (not 13.0-U1 or 13.0-U2 as you currently are). So to reiterate, you should not be using that script.

In order to replace a disk in 13.0-U2, you should use the GUI... again, you showed no evidence that you even tried that, so I'm not clear when you say "it's not working" if you mean the GUI process or the script (which is expected to not work on the version you're using).

You posted an example CLI that is not correct form and proposed it as a solution that others might like to follow (I suggest that they should not).

it's some junk smr seagate drives

If you're using SMR drives, you're not going to be happy in the long term, particularly at your next resilver or scrub when zfs starts marking those drives as faulted because they can't keep up.

It seems to me like there's not much point continuing this discussion further. I wish you luck with your resilver and the safety of your pool data (as good design won't save you if you're using SMR disks).

 

[dashtesla]

You still haven't addressed the first question I asked you with a clear response.

The script is meant only for people running 13.0 (not 13.0-U1 or 13.0-U2 as you currently are). So to reiterate, you should not be using that script.

In order to replace a disk in 13.0-U2, you should use the GUI... again, you showed no evidence that you even tried that, so I'm not clear when you say "it's not working" if you mean the GUI process or the script (which is expected to not work on the version you're using).

You posted an example CLI that is not correct form and proposed it as a solution that others might like to follow (I suggest that they should not).


If you're using SMR drives, you're not going to be happy in the long term, particularly at your next resilver or scrub when zfs starts marking those drives as faulted because they can't keep up.

It seems to me like there's not much point continuing this discussion further. I wish you luck with your resilver and the safety of your pool data (as good design won't save you if you're using SMR disks).

The GUI isn't working that's painfully obvious considering if it was i wouldn't be here right?

It's a small 4x5TB 2.5inch HDD pool with a 256GB L2 ARC SSD it's mostly for documents and it's not a massive pool just a Z1, I have another pool with 12TB CMR drives for multimedia it finished fine btw nothing else no report really no errors all data is there etc.

I also backup this small pool to LTO cartridges and mirror offsite so im not concerned about data loss or resilver time.

 

[sretalla]

The GUI isn't working that's painfully obvious considering if it was i wouldn't be here right?

I have no idea why you're here... you have provided evidence that a script doesn't work on a version it's not supposed to work on, some incorrect advice to others, a generally unsatisfied tone and nothing else to help us to help you.

If the GUI wasn't working, it would have helped to share the details provided by the GUI, not the script.

I suspect the GUI is fine as I see evidence from others in the forum that it works, although I haven't yet had occasion to test it myself, so I assume either you're doing something wrong when using the GUI or there's some issue with your install.

I've already advised you to think about your use of SMR (and I suggest you stop it if you care about your data), so I have nothing more to offer you unless you want to document your problem with the GUI.

 

[dashtesla]

I have no idea why you're here... you have provided evidence that a script doesn't work on a version it's not supposed to work on, some incorrect advice to others, a generally unsatisfied tone and nothing else to help us to help you.

If the GUI wasn't working, it would have helped to share the details provided by the GUI, not the script.

I suspect the GUI is fine as I see evidence from others in the forum that it works, although I haven't yet had occasion to test it myself, so I assume either you're doing something wrong when using the GUI or there's some issue with your install.

I've already advised you to think about your use of SMR (and I suggest you stop it if you care about your data), so I have nothing more to offer you unless you want to document your problem with the GUI.

There is no error in the GUI you click replace disk the big blue button and nothing happens using the latest version, i already have replaced my disk manually as you know since I don't want to put that copy of my data at risk for the sake of troubleshooting and waiting around for a better solution and I don't think anyone would either if their data was at risk and it was their only copy and they had to replace a drive immediately and not wait for a fix and community support to eventually start their resilver I think would you've done the same here.

As for what to look for if you're curious i'm sure you can test everything yourself and if it's working and it's only my installation that's fkd up then no problem at least most people won't face this even though i've done nothing but install truenas 13.0 when it was the latest and click update using the GUI and then try to replace the disk after a reboot that's literally all you'd have to do to replicate my installation, fresh install did absolutely nothing it's not like there's an RNG in place to decide if the installation is gonna be glitchy or work as expected right? :)

Well it's up to ixsystems devs to test and debug not us they know what to look for in their own GUI, at least ZFS itself is working as expected and if i have any more issues with truenas i'll just switch to a standard linux installation for the future if a GUI is getting in my way, which reminds me there's also truenas scale but i haven't had much success with it previously the permissions weren't working as expected over SMB unlike core.

I'm responsible for my data and what I use so thanks for the advice but it's fine, I understand you might just be letting people know for their sake so they don't end up using SMR drives in general but I did it on purpose as I wanted to use spare hardware for this specific installation and I'm aware how SMR drives work and the resilver time to expect.

 

[sretalla]

I'm aware how SMR drives work and the resilver time to expect

I'm not sure how you can understand SMR and elect to use it on purpose with ZFS, but it's your own (data's) funeral, so have at it. It's not about the (too long) resilver times (although that's reason enough on its own to not want SMR), the drives will be marked as faulted if they can't respond fast enough and you'll lose your pool if too many of them are faulted.

As for what to look for if you're curious i'm sure you can test everything yourself and if it's working and it's only my installation that's fkd up then no problem at least most people won't face this even though i've done nothing but install truenas 13.0 when it was the latest and click update using the GUI and then try to replace the disk after a reboot that's literally all you'd have to do to replicate my installation, fresh install did absolutely nothing it's not like there's an RNG in place to decide if the installation is gonna be glitchy or work as expected right? :)

OK, well I did just that... I've now tested on a fresh install of 13.0-U2 and the GUI replace works exactly as expected. No issues.

That seems to make it clear something's not right on your end.

 

[sretalla]

i've done nothing but install truenas 13.0 when it was the latest and click update using the GUI and then try to replace the disk after a reboot that's literally all you'd have to do to replicate my installation, fresh install did absolutely nothing it's not like there's an RNG in place to decide if the installation is gonna be glitchy or work as expected right? :)

Maybe you haven't refreshed the cache on your browser since the update and you're still running parts of the GUI from 13.0 as a result even with the new backend. Try a SHIFT + Refresh in your browser and see if that helps.