ZFS: can't read MOS of pool foo

[aufalien]

Dear community,
I'm a long time FreeNAS user and normally have ZERO issues. And those that I do have, I end up fixing.

However I have a troubling issue today.

One of my TrueNAS servers rebooted in the middle of the night and can no longer complete a boot cycle.

I was receiving errors of;
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool foo

So, I simply downloaded the latest 12.0-U5.1 as I was running 12.0-U4.

I then installed it on a new USB and chose to import my existing pools.

All went very well and I left the system up but in a few hours it rebooted.

The pool foo is simply a very large data pool and not the boot pool. It seems to work for a bit and then the system reboots.

While I do have my configs backed up, I like the idea of starting clean and simply importing what's there and then customizing my env.

At any rate I have done this about 2 times with the same results.

I did search and have read many posts that are similar and point to faulty boot media.

The system will remain up for a bit and the pool foo is accessible so that I can do some troubleshooting. But I'm fairly clueless here.

My foo pool is simply a raidz2 of 60 drives and this pool has performed flawlessly for a few years now.

Any guidance would be an early holiday gift so thanks in advance.

 

[NugentS]

Hardware?

 

[aufalien]

Thank you NugentS. The system has been up now longer than usual however it's not in general population meaning no active NFS serving just yet.

At any rate I didn't wish to vomit my system info in my initial post so here is some data;
2x Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
128GB Ram
60x Seagate Exos ST8000NM0055 8TB drives
10Gb Solarflare
9300-4i4e SAS3 card

The above is arranged in 5x SuperMicro JBODs connected via SAS3.

It's not a lot of info so let me know what else you'd like. In this go around I've enabled autotune to see if things improve and perhaps they have?

I normally don't use autotune and manually tweak but thought to try something different.

My pools are;
misc - used for system data set
void - used for file serving via NFS
* I would prefer to set void for my system data set but am leery of changing it now as the system has been up for a good bit. I had void as my system dataset previously.

zpool status
pool: boot-pool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
da60p2 ONLINE 0 0 0
errors: No known data errors

pool: misc
state: ONLINE
scan: resilvered 1.25M in 00:00:00 with 0 errors on Mon Sep 27 18:27:08 2021
config:
NAME STATE READ WRITE CKSUM
misc ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/a0ce38fa-3ceb-11e9-bc10-ac1f6b2ca160 ONLINE 0 0 0
gptid/a184ac1f-3ceb-11e9-bc10-ac1f6b2ca160 ONLINE 0 0 0
errors: No known data errors

pool: void
state: ONLINE
scan: scrub repaired 0B in 16:13:58 with 0 errors on Sun Sep 26 16:14:02 2021
config:
NAME STATE READ WRITE CKSUM
void ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/6119170f-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/6294e412-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/637bfd37-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/63c6eddf-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/65409ada-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/6738f191-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/5ec3be54-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/600f5377-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/60c2e016-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/64b026e8-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/6647dda4-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/68114201-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
gptid/610fac0c-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/62138af7-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/679fa162-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/697f4779-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/6a08fb6e-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/6a4628be-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
raidz2-3 ONLINE 0 0 0
gptid/6a4f89e7-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/6b8bd0d9-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/6c327042-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/6c81e301-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/89be3534-d44b-11eb-a961-ac1f6b2ca160 ONLINE 0 0 0
gptid/6f10d34a-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
raidz2-4 ONLINE 0 0 0
gptid/6f29bf62-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/6fe24be2-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/7165a4a9-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/724d80eb-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/7317241a-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/72fa6c49-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
raidz2-5 ONLINE 0 0 0
gptid/75416831-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/757be331-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/765d7c55-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/765369e7-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/76edaedf-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/7810c752-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
raidz2-6 ONLINE 0 0 0
gptid/78c0eb6e-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/78df497f-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/7a6ad70e-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/7b3de7df-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/7b6d34d8-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/7d0d5e12-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
raidz2-7 ONLINE 0 0 0
gptid/7de107d8-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/7d95410e-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/7ed5bc5c-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/807e8a01-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/80881b0c-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/81795bb3-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
raidz2-8 ONLINE 0 0 0
gptid/824a024b-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/835bd31b-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/843093f9-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/848a85d9-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/8628c603-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/86314bb5-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
raidz2-9 ONLINE 0 0 0
gptid/8731234e-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/874befe0-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/87ff25bb-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/88b27853-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/8898c833-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
gptid/88bbecf0-d1bc-11ea-a738-ac1f6b2ca160 ONLINE 0 0 0
errors: No known data errors

 

[NugentS]

How are the disks arranged. You can't have 60 disks in a single chassis - therefore they have to be in shelves (multiple). How are the vdevs arranged amongst the shelves - could you be losing (for a brief duration) a single vdev due to a shelf or cabling error?

Just random thoughts I'm afraid

 

[aufalien]

Oh yes of course.

I've 5x SuperMicro 12 bay chassis connected via SAS3 to each other.

Each chassis has 2x vdevs that are raidz2 each.

Then all of that is of course striped. Very standard stuff.

BTW, is there a way to see those errors I mentioned earlier in a log other than during boot?

These errors to be specific;
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool void

I'm simply baffled that I can import the pool even though I have the above error.
Maybe I can initiate a pro support request with TrueNAS? I don't mind paying as this seems out of my toolset.

 

[NugentS]

Honestly - its well out of mine too

 

[aufalien]

N

Honestly - its well out of mine too

No worries. It's a good/bad one indeed.

 

[NugentS]

You could try replacing the boot device with a USB on a temporary basis - to see if that is the issue. What are you booting from?

 

[aufalien]

Well, the boot device is a USB and I've tried 3.

 

[Spearfoot]

Your problem is out of my league, too. The little searching I've done makes me think this might be metadata corruption on your pool.

Have you tried scrubbing the pool? I've seen suggestions to scrub the pool; export it; import it; scrub it a second time; export it; then import it a final time. That's going to take a long, long time on 60 disks...

Have you done regular smartctl tests on the disks? I assume you have, but it never hurts to ask.

 

[aufalien]

Your problem is out of my league, too. The little searching I've done makes me think this might be metadata corruption on your pool.

Have you tried scrubbing the pool? I've seen suggestions to scrub the pool; export it; import it; scrub it a second time; export it; then import it a final time. That's going to take a long, long time on 60 disks...

Have you done regular smartctl tests on the disks? I assume you have, but it never hurts to ask.

Well, that's not a bad idea.

I'm pretty good about setting SMART tests and will initiate one manually soon. Especially if it goes down once more.

So about your suggestion;
a) Scrub
b) Export
c) Import
d) Scrub

In that order I assume?

 

[Spearfoot]

Well, that's not a bad idea.

I'm pretty good about setting SMART tests and will initiate one manually soon. Especially if it goes down once more.

So about your suggestion;
a) Scrub
b) Export
c) Import
d) Scrub

In that order I assume?

Yes, sir, and then export and re-import a final time after the last scrub. The last export & import may not be necessary... but they couldn't hurt, right?

Do you have the system dataset located on the pool? If so, maybe you should move it over to your boot pool. I know it's not a good idea to put it on USB boot media (because of wear & tear) but "desperate times call for desperate measures", and it would only be for a few days. Just a thought...

 

[aufalien]

My dataset is on a pair of mirrored Samsung SSDs.

Back in the 9.x days, I had odd issues replicating +100TB pools with the standard dataset location so I use a separate dedicated location.

Thanks again for the intriguing idea. I'll do it.

 

[Spearfoot]

My dataset is on a pair of mirrored Samsung SSDs.

Back in the 9.x days, I had odd issues replicating +100TB pools with the standard dataset location so I use a separate dedicated location.

Thanks again for the intriguing idea. I'll do it.

No need to move it to your USB boot pool, then. Well, unless the Samsung SSDs are botched!

I'd turn off autotune, too, and most other services (except SSH) while you're doing the scrubbing. Keep it simple and give the pool a chance to scrub, right?

 

[aufalien]

Welp, an update, my scrubs show no errors and SMARTS show no errors as well.

The server panics/reboots during moderate to heavy NFS loads. This has not been an issue for years.

I've swapped memory and even doubled memory. It's a fresh install with no autotune although I like some of the tuned values that autotune introduces.

What I like to do is enable it so that values are introduced. I then disable it and remove those values I prefer not having.

At any rate in this example, no tuning has been done.

I've attached a screen shot during the panic as I'm not sure how to articulate it in a post. I must say that my FreeNAS 11.x servers seem more reliable then my TrueNAS 12.x servers. I've about 8 all around the facility in various levels of OS version.

 

[Spearfoot]

Welp, an update, my scrubs show no errors and SMARTS show no errors as well.

The server panics/reboots during moderate to heavy NFS loads. This has not been an issue for years.

I've swapped memory and even doubled memory. It's a fresh install with no autotune although I like some of the tuned values that autotune introduces.

What I like to do is enable it so that values are introduced. I then disable it and remove those values I prefer not having.

At any rate in this example, no tuning has been done.

I've attached a screen shot during the panic as I'm not sure how to articulate it in a post. I must say that my FreeNAS 11.x servers seem more reliable then my TrueNAS 12.x servers. I've about 8 all around the facility in various levels of OS version.

Thanks for the update, and sorry you're having these problems.

Regarding autotune, I was simply advising that you turn it off during your 'recovery scrubs', to keep things simple. By all means, run it if you want to.

Seems like NFS is the culprit. Would you post detailed information about your NFS setup? Maybe one of the smart folks here will be able to offer some salient advice.

Like you, I've found 11.2-U8 to be rock-solid and stable; I run it on all my servers except my 'sandbox'. Is reverting to 11.2-U8 possible on this server? Meaning, have you kept the older ZFS pool feature flags on the server's pools? I'd be curious to see if your NFS server runs okay on the older version.

 

[aufalien]

Well, keep in mind that this sever has been pretty solid for years, even with using 12.x.

It's standard NFSv3, 16 threads or more depending on my mood.

It's serving;
- home directories
- a reference library of texture and assets for various 2D and #D warez
- a quicktime library
- a software repository as little to no software is installed locally on workstions

I've about 300ish NFS clients each on 1Gb.

The server itself is 10Gbx2/trunk.

I was async (trying sync with no stability improvement) as in my env and ZFS being COW based I don't really see a need for a SLOG/sync.

We've had several power outages where my large 240V APCs would drain and the server would go by by, no biggy.

All in all it's sorta hard to corrupt it. I think they have a NFS bug in 12.

You know, I've even pulled a JBOD in a 6 JBOD set up, that's 2 vdevs on a live system, yanked power etc... She still came back up fine.

I've done all sorts of abuse testing as I sorta like killing these things. No problem, ZFS came back up.

I'm thinking about trying to enable it's firewall and ONLY allow NFS, ssh, https traffic to it. I always disable mDNS etc...

 

[aufalien]

I'll also be looking into importing this pool into OpenZFS/Linux and moving away from FreeNAS/TrueNAS/FreeBSD.

My previous bug reports go unanswered and the fact that FreeBSD doesn't support 1M R/W NFS sizes is abysmal at this point.

What's even more abysmal is that seemingly no one here has ever brought this up with IX Systems who can then speak with the upstream provider.

My over all view on the state of anything IT is that software is made more feature rich instead of more stable.

Instead of fixing what's broken, the industry as a whole simply introduces more widgets, bells and whistles.

The TrueNAS GUI is bit much in terms of a full featured 3D animated film as well.

I'm sure it suites other tech jockey's but not me so much.

 

[aufalien]

Thanks for the update, and sorry you're having these problems.

Would you know how I might downgrade the pool from TrueNAS 12 to FreeNAS 11?
Unfortunately I had upgraded the pools when upgrading to TrueNAS 12.

This would be the very easiest thing to do as like you said 11.2 was rock solid and I regret not being more careful in switching to TrueNAS 12. So this is on me 100%.

Also is it possible to replicate (not rsync) the v12 pool to v11.2 dataset? I may have a spare server that could fit the bill.

I've not done this and was curious on your thoughts.

 

[Spearfoot]

Would you know how I might downgrade the pool from TrueNAS 12 to FreeNAS 11?
Unfortunately I had upgraded the pools when upgrading to TrueNAS 12.

This would be the very easiest thing to do as like you said 11.2 was rock solid and I regret not being more careful in switching to TrueNAS 12. So this is on me 100%.

If you upgraded the pool's feature flags, then I don't know any way to revert other than to re-install 11.2, re-create the pool w/ FreeNAS 11.2-U8, and then restore the data from backup. You've got a lot of data on a lot of disks -- 60 disks, right? -- so that would be a daunting prospect.

It might be easier to shake down and fix your NFS configuration gremlins instead. Wish I could help you there, but my NFS setup is dirt-simple and 'just works', so I don't have any advice to offer.

Perhaps if you post details about your NFS problems, someone more knowledgeable can help out.

EDIT: I meant to say: "details about your NFS configuration" -- not "problems"