Help Required - (Seemingly) Corrupt Array

[jetpigeon]

Good Morning All - Apologies that my first post is this, but I need some help... Please excuse my lack of knowledge; whilst I'm technically savvy my TrueNAS/FreeNAS knowledge is not great.

My hardware is (apologies I don't know the specific MB manufacturer etc..);
- Old Datto appliance.
- i5-4440 / 16GB RAM / 1238GB SSD Boot Volume / 4x Hitachi HDS5C3020ALA632 2TB HDDs.

I have a single pool running over two mirrored pairs:
pool1 ONLINE
mirror-0 ONLINE
gptid/24d7222f-6408-11e9-82f8-fcaa14913c0e ONLINE
gptid/2dd7e7ad-6408-11e9-82f8-fcaa14913c0e ONLINE
mirror-1 ONLINE
gptid/377f6874-6408-11e9-82f8-fcaa14913c0e ONLINE
gptid/3fad091f-6408-11e9-82f8-fcaa14913c0e ONLINE

On this I have (had!) a single iSCSI extent running in device mode on a single zvol on this pool. I have a single VMWare server which connects to this pool over two 'portals'. The single extent contains a single VMFS volume used by VMware.

All has been working perfectly fine for months on end until one day, it didn't. Upon connecting the machine to a screen I found it was boot-looping, with a panic when mounting the volume 'panic: Solaris(panic): zfs: allocating allocated segment(offset=1172842725376 size=4096) of (offset=1172488953856 size=4292273992). I only have a photo of this so can't type it all out.

This was on FreeNAS - I have since run an in-place upgrade to TrueNAS latest stable build.

I found that the only way I could get the system to boot was by pulling the disks before powering on. I can then insert the disks (with hot-swap enabled in bios) and then see and attempt to mount the volume.

I am now at the point where I can mount the volume as read only using: zpool import -F -f -o readonly=on.

This appears to mount the pool fine, and it appears as mounted as read only when I run zpool list.:

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 111G 5.08G 106G - - - 4% 1.00x ONLINE -
pool1 3.62T 1.31T 2.32T - - 0% 36% 1.00x ONLINE -


But VMWare just hangs for ages when rescanning (possibly because it's trying to write to the volume) and eventually show the volume as size 0B?

Also having tried to mount the volume using the windows iscsi initiator it also seems to hang indefinitely.

Really not sure what is going on here, the disks seem to be in good health - and I obviously run two mirrored pairs for redundancy.

Your thoughts and expertise would be greatly appreciated. I just want to mount the VMFS volume somewhere and copy all the data off so I can re-build the NAS.

Any further info you need let me know.

Thank you in advance.

 

[jetpigeon]

I wonder if it might be possible to try and mount the VMFS volume in TrueNAS itself? Thus bypassing iscsi etc.?

 

[sretalla]

I wonder if it might be possible to try and mount the VMFS volume in TrueNAS itself?

No, TrueNAS has no installed filesystem for VMFS.

You will likely need to use a ZFS send | recv to get the data/ZVOL to a healthy/writable pool and then mark it as rw.

I suggest that the error you mentioned indicates that there is indeed some problem with your disks (at least one of them).

Were you able to get smartctl -a /dev/daX from any of them?

 

[jetpigeon]

Great, thanks sretalla! I have just hooked up a 4TB USB HDD and created a new pool on that - I've disabled iSCSI for now and am just getting to grips with the syntax for a ZFS send/receive to the new pool now.

Have run smartctl -a /dev/adb0,1,2,3 and all seem to come back with 'SMART overall-health self-assessment test result: PASSED' alongside a bunch of other info. However drilling deeper into that, it looks like adb0 has suffered some form of error at some point:

<code>Error 101 occurred at disk power-on lifetime: 6631 hours (276 days + 7 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 80 48 39 80 02 Error: UNC at LBA = 0x02803948 = 41957704</code>

 

[jetpigeon]

Sorry - completely failed to properly use the

Code:

 flags there...

 

[jetpigeon]

Sorry for multiple posts but a couple of questions as I continue down this rabbit hole:

- I cannot seem to do a zfs send/recieve as I have no snapshot? And I can't create a snapshot as the pool is read only? Is there another way I can accomplish this?
- Given that adb0 was showing some error; perhaps I could try mounting the volume normally with that disk disconnected? Or would that leave me open to further issues? Want to ask those with experience first before I ruin any slim chance I might have had of recovering the data.

I do find it strange that neither VMWare nor the windows software iSCSI HBAs are able to see the volume at all in read-only. I'd expect Windows to at least see the 'disk' even if it can't read VMFS.

Finally; whist mounting in read-only on the console I noticed it says:

Code:

GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Encryption: hardware

I'm pretty certain this isn't an encrypted pool, and that same message does not appear when I mount the pool over SSH. So hopefully it's just some console level message. But if it is, where in the filesystem would I find the key? Have done a recursive search for *.key and it only came back with 4 files, none of which appear to be related to pool encryption. There are also no 'failed to authenticate' messages - so assuming the pool is mounted and readable..

Thanks again.

 

[jetpigeon]

Me again... A small update - I've managed to start the zfs send/recv to clone the zvol over. Just waiting for that to finish now, I expect it will take a while.

Fingers crossed VMWare will be able to read the volume once I've mounted that in iSCSI.

My next question will be; how do I ascertain what caused the failure in the first place? And what might I be able to do to prevent it in future? The NAS is backed by UPS and I know it's not suffered any power failure recently. And I had (perhaps incorrectly) assumed that zfs coupled with the mirrored redundancy would allow the OS to handle and either drop a disk if it starts experiencing read/write errors?

Thanks.

 

[jetpigeon]

Next hurdle; the send/recv failed after quite some time with

Code:

warning: cannot send 'pool1/zvol1': Input/output error
cannot receive new filesystem stream: incomplete stream

Are there some flags I can use to override / force this to work?

The command I used was

Code:

zfs send pool1/zvol1 | zfs recv pool2/zvol2

Assuming it will create the currently non-existent zvol2 on pool2.

Thanks

 

[sretalla]

You may need to use the -F on the recv side to force it to destroy the existing one and allow the new one to write in its place.

It may be that you have some unreadable blocks in the ZVOL, so it will probably halt at that point in the stream... not exactly sure what to recommend there, but it would imply that you will have an equally corrupt copy even if it finishes.

 

[AlexGG]

I would recommend you use ddrescue to clone four of the original disks to four spares. Then, proceed to tinker with the spares. Avoid exercising original disks, of which at least one looks like it has a mechanical problem of some sort or other. Also if you botch something on the clones, you can go back and make another set.

 

[jetpigeon]

Cheers guys. I tried the -F on recv and unfortunately that seems to have failed with the same error. I was hoping it would be able to send the whole zvol, complete with errors in the stream. Is there anything similar to '-F' for the zfs send command?

Cheers for the ddrescue idea AlexGG; I'll look into that. Sadly I don't have any spare disks of this of this capacity at the moment.

Id hoped that as at least 3 of the disks seem completely fine I'd be able to use one of the healthy striped pairs to pull the data from. Might try mounting the volume from just that pair as a last resort, but I guess zfs would have already identified the corrupt blocks and failed over to the 'good' disks if that were an option?

Failing all of the above I'll just have to accept that the data is gone, rebuild and pull back the important stuff from cloud backups; which will take some time! Time to introduce a local backup...

I'm just bemused as to how this failure happened with seemingly no warning and what appear to be 3 healthy disks! Could be my fault in some way I guess but I am considering swapping this thing out for an off-the-shelf Synology or something!?

 

[sretalla]

I'm just bemused as to how this failure happened with seemingly no warning and what appear to be 3 healthy disks!

Were you running regular (short and long) SMART tests on all of your disks? If you don't, a disk appears to be fine until an attempt at working with a bad block fails.

 

[sretalla]

Is there anything similar to '-F' for the zfs send command?

I think you can use dd, with the switch conv=noerror,sync (maybe also iflag=direct) I haven't tested that on FreeNAS, but did just use it today on Kali Linux.

 

[jetpigeon]

Were you running regular (short and long) SMART tests on all of your disks? If you don't, a disk appears to be fine until an attempt at working with a bad block fails.

Probably not to be fair - unless TrueNAS does this out-the-box then no, I wasn't. I'll have a read up on that, probably something I should be doing and skipped over when setting it all up.

I think you can use dd, with the switch conv=noerror,sync (maybe also iflag=direct) I haven't tested that on FreeNAS, but did just use it today on Kali Linux.

Awesome, many thanks, I'll give that a go. So would that be a case of just piping the output of dd into zvol recv? (Apologies I only have a pretty basic working knowledge of *linux FreeBSD and even less on block-level copy tools!) If you have a rough example dd command that I could try to replicate what I'm doing with

Code:

zfs send pool1/zvol1 | zfs recv -F pool2/zvol2

that would be greatly appreciated.

Thanks again.

*edit: sorry I forget that TrueNAS isn't Linux!

 

[AlexGG]

So would that be a case of just piping the output of dd into zvol recv?

No, not really. dd and ddrescue both copy disk to disk (or disk to file, but I'm thinking disk to disk). The goal of the process is to get a copy of the pool which has no unstable (or suspected unstable) disks in it and also to get a second chance if you screw up something during your recovery attempt. Now you have one damaged pool on four drives. Once dd or ddrescue complete, you will have two copies of the same damaged pool, on eight drives. The process does not make the pool working, and is not intended to, and dd is not aware of ZFS, or any other filesystem for that matter.

 

[jetpigeon]

Thanks, cool yep that makes sense.

I have had a further breakthrough. Such a small thing but I can now at least see my files; it turned out that 'pool1' always kept mounting in / rather than /mnt.

I noticed that in the GUI Pool1 was erroneously showing as online under /mnt/pool1, when infact it was not there but simply under /pool1. In my iSCSI config the extent was pointing at the visible 'zvol1', but I guess it was looking for /mnt/poo1/zvol1, which was not there!

I exported and re-imported the pool, this time using:

Code:

zpool import -F -f -o readonly=on -R /mnt pool1

and hey-presto! The volume is visible!

I'm not out of the woods yet because VMWare still cannot read the datastore, but it does at least see a RO only volume of 2.8TB which I'm expecting to see. I can also mount and explore my data using DiskInternals VMFS recovery. However the license for that is $700 which is too much for me and my homelab...

Now looking at mounting the iSCSI target in linux and using vmfs-tools to extract my vmdk files. Will see how it goes.

Cheers again for help and pointers so far, much appreciated.

 

[jetpigeon]

Finally sorted; I managed to mount the RO volume in Ubuntu using open-iscsi and vmfs6-tool.

From there I could simply copy out the all of my files and re-import into alternative VMWare storage.

I've come across two VMDK files that always fail to copy at the same point, presumably due to file-system corruption. But the bulk of my data has thankfully been recoverable. I think in future though I'll use NFS rather than iSCSI as have many more options open to me to mount the volume and read the data natively on TrueNAS.

If anyone can advise or point me in the direction of best practice to prevent this from re-occuring again, or how to monitor to catch it before it happens again, would be much appreciated. Cheers.