Degraded boot-pool not reported on Cobia?

[tannisroot]

Saw this post on Reddit, which made me concerned:

Reddit - The heart of the internet

www.reddit.com

Heads up : TrueNAS Scale does not show any alarm if one of the mirrored drive on your boot pool is failed. I'm on Scale 23.10.0.1. Nothing in alarms, nothing on the dashboard.

I've ran my system in degraded mode like this for months. I've discovered the failure only when I visited the page System -> Startup -> Status and clicked on my mirror-0 to expand the view.

I'm very disappointed.

Can anyone else confirm on Cobia 23.10.1, by maybe plugging the cable out one of the devices from the bool pool mirror while the server is offline? I might check this myself when I need to install a new drive to my server, but maybe someone will beat me to it.

 

[CacheMeIfYouCan]

Hi @tannisroot, I'm the OP of this post on Reddit. Thanks for posting it here, I will follow along.

 

[Kris Moore]

Oof, that is indeed a potentially serious issue. @CacheMeIfYouCan is your system still in this state and can you generate a debug file and make the bug ticket? I'll make sure this gets looked at ASAP.

 

[probain]

This has been reported by me back on december 22. NAS-125855
Fix has PR. But I don't know why they're waiting to implement it in 23.12.2, instead of 23.13.1.1. which was scheduled for last week.

Edit.
Should be the correct ticket now.
Mobile linking isn't the most reliable. I'll do the actual link when I get home

 

[CacheMeIfYouCan]

Sure but I have replaced the defective drive yesterday. I guess I could reinstall the defective one or first try to just unplug the new one and see if it replicates the same symptoms.

My boot drives are not hot-swappable so it will have to wait until night time. Could you remind me how to take the debug files?

 

[CacheMeIfYouCan]

This has been reported by me back on december 22. NAS-124961
Fix has PR. But I don't know why they're waiting to implement it in 23.12.2, instead of 23.13.1.1. which was scheduled for last week.

Could you double check the reference you provided above? It doesn't seem to be related.

 

[Kris Moore]

Debugs are under System Settings -> Advanced -> Save Debug.

Sure but I have replaced the defective drive yesterday. I guess I could reinstall the defective one or first try to just unplug the new one and see if it replicates the same symptoms.

My boot drives are not hot-swappable so it will have to wait until night time. Could you remind me how to take the debug files?

 

[CacheMeIfYouCan]

I haven't reinstalled the failed drive yet but I had a look at the logs and I think there's already all the info that you're looking for. How could I send you the debug file privately?

 

[HoneyBadger]

Bonus points for the username, I had a sensible chuckle.

Normally you would do Report A Bug and attach the debug there - but since the linked issue above doesn't seem to be the right one (it's about disk burn-in?) drop me a DM if you like, I can make sure @Kris Moore gets it.

 

[Kris Moore]

Yep, once you make the ticket, a link will be created which lets you upload a private debug file. You just need to associate it with your particular ticket, and it'll be iX engineer eyes only.

https://ixsystems.atlassian.net/servicedesk/customer/portal/15/group/37/create/153

 

[probain]

I've updated with the correct link, and subsequent correct issue. I was trying to link from my phone. And that experience was aparently far from ideal. CORRECT LINK NAS-125855
Do note that there is a (closed) duplicate issue referenced with, that matches yours. NAS-125863

Ping for visibility
@CacheMeIfYouCan @Kris Moore @HoneyBadger

 

[CacheMeIfYouCan]

Yep, once you make the ticket, a link will be created which lets you upload a private debug file. You just need to associate it with your particular ticket, and it'll be iX engineer eyes only.

https://ixsystems.atlassian.net/servicedesk/customer/portal/15/group/37/create/153

Done, it is recorded as https://ixsystems.atlassian.net/browse/NAS-126748

 

[Kris Moore]

Appreciate it folks. I've make sure all the tickets are linked now, the debugs can come in handy if we are trying to catch various manners in which this fails. Looks like they may all have a similar root-cause. Slated for fixing in 23.10.2.

 

[probain]

Appreciate it folks. I've make sure all the tickets are linked now, the debugs can come in handy if we are trying to catch various manners in which this fails. Looks like they may all have a similar root-cause. Slated for fixing in 23.10.2.

It's actually quite rewarding to do the bug reports, since they listened to and get fixed. This is very different to how the big faceless enterprises.

If us testing and reporting keeps everyone using this great software. Then it's a win for everyone.

 

[Kris Moore]

It's actually quite rewarding to do the bug reports, since they listened to and get fixed. This is very different to how the big faceless enterprises.

If us testing and reporting keeps everyone using this great software. Then it's a win for everyone.

We think so as well. All companies big and small make mistakes and all software has bugs. But we try to own up to those and be open and transparent in the resolution as well. I encourage all users to keep reporting things as they find them. We can't guarantee fixing every single thing in that moment, but we'll do our best to make the software better and better.