TrueNAS Scale incorrectly reporting Mixed Capacity VDEVS

[Brandito]

Can you PM me a debug?

What is a Debug and Why does (NickF think) it Matters?

Disclaimer: This is a living document and I expect to revise it often-ish It's also a pretty half-way mix of my personal opinions and also presenting important information. Understanding the "Debug" in TrueNAS Systems FreeNAS, TrueNAS, TrueNAS...

www.truenas.com

sent, thanks for the help btw

 

[NickF]

Hmm that's weird. There is no history on this system of that pool being imported. Cache file?

Code:

2023-11-18.10:10:16 zpool import -N -f boot-pool
2023-11-18.10:27:51 zpool import -N -f boot-pool
2023-11-20.03:45:02 py-libzfs: zpool scrub boot-pool
2023-11-20.16:05:31 zpool import -N -f boot-pool
2023-11-21.09:54:32 zpool import -N -f boot-pool
2023-11-21.10:37:08 zpool import -N -f boot-pool
2023-11-21.10:44:42 py-libzfs: zfs snapshot  boot-pool/ROOT/Clean-23.10.0.1@for-debug
2023-11-21.10:44:42 py-libzfs: zfs clone -o mountpoint=/tmp/tmp3n0s4dxv boot-pool/ROOT/Clean-23.10.0.1@for-debug
2023-11-21.10:44:42 zfs destroy boot-pool/for-debug

Can you try:

Code:

zpool export -f Home
zpool import -o readonly=on -fR /mnt Home

 

[Brandito]

Can't get passed the export

Code:

root@truenas[~]# zpool export -f Home
cannot export 'Home': pool is busy

 

[NickF]

At this point we can bang on it harder or, ideally, you can perform recovery from another system that has never touched this pool. Given we know there are potential hardware problems that may be the safest way.

Alternatively if that's not an option, install TrueNAS fresh on a new boot drive and remove the existing one for now. Dont lose or wipe the existing one. Then the system will not try to import it for sure.

 

[Brandito]

At this point we can bang on it harder or, ideally, you can perform recovery from another system that has never touched this pool. Given we know there are potential hardware problems that may be the safest way.

Alternatively if that's not an option, install TrueNAS fresh on a new boot drive and remove the existing one for now. Dont lose or wipe the existing one. Then the system will not try to import it for sure.

I should have a boot option already to boot without this pool imported.

My only question is, do I need to worry about importing to the txg that I'm mounted with now?

 

[NickF]

I should have a boot option already to boot without this pool imported.

My only question is, do I need to worry about importing to the txg that I'm mounted with now?

It should attempt to roll back to a known good TXG when you mount with -fR and read only. We'll see what it says.

If it doesnt try this:

Code:

zpool import -f -o readonly=on -T 3080103 Home

 

[Brandito]

It should attempt to roll back to a known good TXG when you mount with -fR and read only. We'll see what it says.

If it doesnt try this:

Code:

zpool import -f -o readonly=on -T 3080103 Home

While I can load up the iso in ipmi, I can't get truenas to install a fresh boot environment to my bootpool so I'll need to do that when I get home from work and can install a spare drive to boot from

 

[NickF]

While I can load up the iso in ipmi, I can't get truenas to install a fresh boot environment to my bootpool so I'll need to do that when I get home from work and can install a spare drive to boot from

Cool. Thats safer option anyway. Let me know and @ mention me when you figured out where you are if you need help with next steps.

 

[Brandito]

Cool. Thats safer option anyway. Let me know and @ mention me when you figured out where you are if you need help with next steps.

@NickF I got a fresh copy of truenas installed on a spare drive and once in the shell I get this when trying to mount the zpool

Code:

root@truenas[~]# zpool import -o readonly=on -fR /mnt Home
cannot mount 'Home/Media': Input/output error
Import was successful, but unable to mount some datasets#

The datasets all show up in the webui but the pool does not show up under storage.

 

[Brandito]

Could this be a permissions issue? I was getting a lot of errors before anytime I even clocked the dataset under datasets in the webui mostly pertaining to user quotas and permissions. Now that it's read only nothing

 

[Brandito]

This is concerning for me. Could the block cloning bug in 2.2 be the cause of my corruption? Can we expect Truenas to release an update to added this fix?

Release zfs-2.2.1 · openzfs/zfs

Supported Platforms Linux: compatible with 3.10 - 6.6 kernels FreeBSD: compatible with releases starting from 12.2-RELEASE Changes Note: We recommend that anyone running zfs-2.2.0 to please upda...

github.com

 

[Kris Moore]

This is concerning for me. Could the block cloning bug in 2.2 be the cause of my corruption? Can we expect Truenas to release an update to added this fix?

Release zfs-2.2.1 · openzfs/zfs

Supported Platforms Linux: compatible with 3.10 - 6.6 kernels FreeBSD: compatible with releases starting from 12.2-RELEASE Changes Note: We recommend that anyone running zfs-2.2.0 to please upda...

github.com

Fix is queued (Along with ZFS 2.2.1) for TrueNAS SCALE 23.10.1 already. That is still a couple weeks out, we'll be reviewing soon to see if we can do anything sooner in the meantime.

 

[NickF]

@Brandito did you ever manage to get past that issue? Did you ever try to import an older TXG to see if you can get access?

 

[Brandito]

@Brandito did you ever manage to get past that issue? Did you ever try to import an older TXG to see if you can get access?

I can try an even earlier one, didn't know if that would work

 

[Brandito]

@Brandito did you ever manage to get past that issue? Did you ever try to import an older TXG to see if you can get access?

Actually, I don't know that I can now, when I type zdb -u -l Home the TXGs are all much higher than when I started, probably from not mounting the pool read only. Can I still go back to the ones I had before?

 

[mav@]

Could the block cloning bug in 2.2 be the cause of my corruption?

Lookign through the thread I haven't seen an single backtrace or debug or anything that would shed some light on the cause of the system crashes. But I doubt it. At least not the sorts of issues reported so far. So far there are two know issues: 1) ZIL corruption on encrypted datasets, that prevents later pool import in non-read-only, and 2) silent user data corruptions, creating ranges of zeroes if several operations including block cloning done to a file (about) same time. The first I think I've diagnosed and the patch is in review now . The second is still being debugged. But none of those should cause problems for even read-only pool imports you see.

 

[Brandito]

@Brandito did you ever manage to get past that issue? Did you ever try to import an older TXG to see if you can get access?

Replying to this comment again

Code:

root@truenas[~]# zpool import -f -o readonly=on -T 3080103 -R /mnt Home 

This seems to be the final bit of magic to get it to mount my Media dataset. So far the directory structure looks good. I should have 10 drives sitting at my house when I get home tonight ready to receive a zfs send.

That leads me to my next question, how should I do this? I have 2 snapshots for that dataset, and 80-90TB to send. I don't believe I can break it down by directory with zfs send, but is there a way to do it so if for whatever reason the process needs to stop (power outage) it can be resumed? I've not messed with zfs send until this past week.

Or is there a better way other than zfs send that doesn't compromise the data? Would it be as simple and a cp from one dataset to another?

 

[Patrick M. Hausen]

If creating new snapshots is not possible due to the read-only import, I recommend rsync over cp, because rsync can ve interrupted and restarted an arbitrary number of times.

rsync -avz is what covers most use cases. Drop the -z if it's a local copy and not over the network.

 

[Brandito]

If creating new snapshots is not possible due to the read-only import, I recommend rsync over cp, because rsync can ve interrupted and restarted an arbitrary number of times.

rsync -avz is what covers most use cases. Drop the -z if it's a local copy and not over the network.

Is it best to still try to break up the transfer by directory? Will I know if and what is corrupt during the transfer?

If this was a typical backup of known good data I'd have fewer questions, so please bear with me.

Thanks again to everyone getting me this far!

 

[Patrick M. Hausen]

If the data is served by ZFS in a corrupt state to rsync without an error message, then probably (my guess) zfs send and receive would not behave differently. Advantage of rsync is that you can go by directory which you cannot with zfs send. Only full datasets with the latter.

My expectation is that rsync will transfer the files and upon encountering a broken one and hence an I/O error from ZFS, it will lig something like "skipped" and continue or abort altogether. Check the manpage for options - if there are any - to control this.

If abort is the only thing it can do, you can build an exclude list that you add to as you hit broken files. Rsync will always continue from where it stopped so it's good for an iterative process.