Datasets lost after reboot

[Collerico]

Hi everyoone, this is my first post I hope I used the right thread.
I use truenas-SCALE-22.12-BETA.1 virtual on unraid server with assigned 8core, 16gb ram (ecc) and 3 pools that I use for different uses.
This morning I had to reboot to do an update on the physical host and on reboot I noticed that the process to load one of the 3 pools took a long time, at the time of reboot I noticed that most of the datasets on this pool were lost and the space used went from around 70% to 36%. this pool is a raidz1 of 4 4tb wd red disk with a 16gb vdisk for deduplication table (maybe this was the problem at the beginning)
in the dataset menu I see only two of the 5 datasets that should be there, one of which is about 1tb now completely empty.
I searched on the forum and on the web but I did not find anything, luckily some of these datasets had replicas and I should be able to recover the data, I am left with the problem on two datasets, of one I only have snapshots, I see the process still present but I don't know how and if I can restore it, I don't have any backup or replicas of another one because I use it as a temporary storage for documents to be sorted.
there are very important things that I need to recover, is there anything that comes to your mind could help me? I'm also okay with having to browse disks and manually copy files to another disk / pool
sorry for the long message but I'm desperate for what happened.

Thanks in advance

 

[joeschmuck]

That is one odd problem, datasets disappearing. Have you tried to reboot again? The entire machine. I don't have much hope that it would fix it but it's worth a shot. I hope this isn't an issue with the Beta. Hopefully you have a backup of all your data when you are running a Beta.

16GB RAM and deduplication is terrible, you need to stop using deduplication in my opinion. I couldn't say that caused the issue but it's definitely not a good setup with such little RAM.

It sounds like you are running TrueNAS on a hypervisor but didn't state which one it was.

 

[Collerico]

That is one odd problem, datasets disappearing. Have you tried to reboot again? The entire machine. I don't have much hope that it would fix it but it's worth a shot. I hope this isn't an issue with the Beta. Hopefully you have a backup of all your data when you are running a Beta.

16GB RAM and deduplication is terrible, you need to stop using deduplication in my opinion. I couldn't say that caused the issue but it's definitely not a good setup with such little RAM.

It sounds like you are running TrueNAS on a hypervisor but didn't state which one it was.

My hypervisor is unraid 6.11.1, i've tried a reboot but nothing changed, i also tried exporting and reimporting the pool but no changes, i've noticed that all operation regarding this pool is extremely slow, for now I have almost given up on recovering the data because im running out of idea.

 

[Samuel Tai]

Let me guess. You passed through disks to TrueNAS instead of an HBA? This unfortunately is a known path for losing data.

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

The only proven method for passing disks to a TrueNAS VM is to pass through a whole HBA, and then attach disks to the HBA. Unfortunately, you're going to need to use ZFS recovery tools to salvage your data. Klennet ZFS Recovery is probably your best bet.

 

[Collerico]

Let me guess. You passed through disks to TrueNAS instead of an HBA? This unfortunately is a known path for losing data.

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

The only proven method for passing disks to a TrueNAS VM is to pass through a whole HBA, and then attach disks to the HBA. Unfortunately, you're going to need to use ZFS recovery tools to salvage your data. Klennet ZFS Recovery is probably your best bet.

Im using a pci-e sata controller with a Marvel 88SE9215 controller directly passed trough the vm, i dont know if that give problem with truenas or not, i just checked if it was freebsd compliant but i never had a problem like this in 2+ year using it

 

[Samuel Tai]

Im using a pci-e sata controller with a Marvel 88SE9215 controller directly passed trough the vm, i dont know if that give problem with truenas or not, i just checked if it was freebsd compliant but i never had a problem like this in 2+ year using it

Unfortunately, cheap PCI-E SATA controllers have a poor track record of data safety with TrueNAS.

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

 

[Collerico]

Ok, i understand the problem, i think we can close this thread

 

[joeschmuck]

m using a pci-e sata controller with a Marvel 88SE9215 controller directly passed trough the vm, i dont know if that give problem with truenas or not, i just checked if it was freebsd compliant

You are using Debian, not FreeBSD. That won't really help you but it's important to know the operating system you are using.

pci-e sata controller with a Marvel 88SE9215 controller

I'm using a Marvel 88SE9235 controller which to date has been working very stable for TrueNAS Core (FreeBSD), but I'm also using ESXi 7.0 which is a very mature hypervisor. I pass the entire card through. I have no clue if it will work reliably whenever I migrate over to TrueNAS Scale (Debian), I hope it will but I'll be retaining a backup of my data for a while until it's proven itself, or I install an HBA I already have. I don't use it because it creates a lot of heat and my current controller is working fine.

 

[Zhang-Hokkien]

Hi everyoone, this is my first post I hope I used the right thread.
I use truenas-SCALE-22.12-BETA.1 virtual on unraid server with assigned 8core, 16gb ram (ecc) and 3 pools that I use for different uses.
This morning I had to reboot to do an update on the physical host and on reboot I noticed that the process to load one of the 3 pools took a long time, at the time of reboot I noticed that most of the datasets on this pool were lost and the space used went from around 70% to 36%. this pool is a raidz1 of 4 4tb wd red disk with a 16gb vdisk for deduplication table (maybe this was the problem at the beginning)
in the dataset menu I see only two of the 5 datasets that should be there, one of which is about 1tb now completely empty.
I searched on the forum and on the web but I did not find anything, luckily some of these datasets had replicas and I should be able to recover the data, I am left with the problem on two datasets, of one I only have snapshots, I see the process still present but I don't know how and if I can restore it, I don't have any backup or replicas of another one because I use it as a temporary storage for documents to be sorted.
there are very important things that I need to recover, is there anything that comes to your mind could help me? I'm also okay with having to browse disks and manually copy files to another disk / pool
sorry for the long message but I'm desperate for what happened.

Thanks in advance

Since TrueNAS scale is Linux based, you can follow this video to analyze your log files to figure out exactly what is happening and what the problem might be:

Analyzing Linux logs to troubleshoot storage subsystem with LSI IT mode HBA SAS controller

In this video I'm going to show how to analyze the Linux log files to troubleshoot a storage subsystem with an LSI IT mode HBA SAS controller. In this case, ...

youtu.be