SOLVED Accidentally deleted datasets

[rockybulwinkle]

First let me explain my setup:

I have two installations of TrueNAS. One is local (named Titan and mirrored pool "data0"), and another is remote (named Atlas and mirrored pool "data0-backup"). Of note, Atlas only has 4GB of RAM. I am well aware that this is not supported and is the next thing I'm going to upgrade, but I really don't think this is the cause of my issue today.

Titan normally replicates to Atlas.

Last week, Atlas had a boot drive failure. The boot-pool wasn't in a mirror and I had no backups. The drive still worked "well enough" to boot and create a backup of the configuration, so I made that, and reinstalled TrueNAS. In the process I upgraded Atlas from Core 12 to 13.

Now, where the trouble really starts:

This weekend, I made an initial backup of a family member's data to Atlas (I'm building them a NAS for christmas and wanted to have that initial backup pre-loaded).

When I got home, I tried to create a replica using the Web UI from Atlas to Titan. This was done through Titan's UI as a pull from Atlas. I think I misconfigured the replication's destination as my root dataset initially (I think this is when my data started getting deleted). I corrected it, and tried starting the replication again, which failed.

At that point I decided to make the replica using the command line, unaware that I had deleted a bunch of datasets on Titan. I *think* around 6GB was written to Titan before I noticed the missing datasets.

As soon as I noticed this, I killed the replication process from Atlas to Titan, and tried to export Titan's data pool. It wouldn't let me, because apparently I'd set the system dataset to my data pool instead of boot-pool... I moved the system dataset to boot-pool, tried exporting again, rebooted, finally exported.

I tried importing against the last transaction before the destruction, but it says "cannot import 'data0': one or more devices is currently unavailable".

I can still import data0 on Titan, but several datasets are gone, and all common snapshots for replication to Atlas are gone.

I have put a hold on all snapshots on Atlas. I have not lost any data yet as far as I'm aware, just damaged my local NAS.

Is there any way I can restore my datasets? I really don't want to have to restore from the remote backup because it has a super slow upload speed (home internet). Of course I will if I have to (either by uploading or by sneakernet).

I do have a single drive locally that can hold all of what's left of data0, though I'm going to hold off on trying anything more until hopefully someone experienced can give me advice.

 

[rockybulwinkle]

Here's my full hardware specs:
Titan:
motherboard: ASRock B450 Pro4
CPU: Ryzen 3600
RAM: 32GB ECC
Pool: 2xWD100EMAZ-00WJTA0 in a mirror (shucked WD 10TB drives)
Disk controller: using the AMD chipset's SATA controller
Network: Realtek RTL8111/8168/8411 (I know this isn't recommended... but it hasn't caused me problems yet afaik and I am assuming problems would be network reliability, not data integrity).

Atlas:
Motherboard: Supermicro X9SCA-F
CPU: Xeon E31230@3.2GHz
RAM: 4GB ECC (again, I'm aware this isn't supported. It's next on my upgrade list)
Pool: 2xWDC WD140EDGZ-11B1PA0 in a mirror (shucked WD 14TB drives)
Disk controller: Intel 6 series/C200 (motherboard's built-in controller)
Network: Intel 82574L

 

[jgreco]

Supermicro X9SCI

If you had an X9SCA, you'd need either an E3-1235 CPU (w/ video) or a GPU. So you almost certainly have the X9SCI, which for whatever reason does like to get cross-identified as both boards.

 

[rockybulwinkle]

If you had an X9SCA, you'd need either an E3-1235 CPU (w/ video) or a GPU. So you almost certainly have the X9SCI, which for whatever reason does like to get cross-identified as both boards.

I checked my order details, it was an X9SCA-F. It has an onboard VGA adapter:
root@atlas[~]# lspci | grep -i vga 04:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)

 

[rockybulwinkle]

Baring somehow recovering my local NAS, I'm going the restore-from-offsite route. The replica is hosted at a family member's house, so I'm sending a USB drive to them, they'll plug it in, I'll make a replica, then they'll mail it to me.

Currently my datasets are unencrypted; this seems like a prime time to get them encrypted if I wanted to. Are there any downsides to ZFS's built-in encryption besides the obvious, "don't lose your password", and that I'll have to enter the password on reboots? Both servers have the AES ISA extensions.

 

[rockybulwinkle]

Why has this been moved to "legacy software releases"? Atlas is on TrueNAS-13.0-U3.1 and Titan on TrueNAS-SCALE-22.12-BETA.2.

Edit: Got servers backwards

 

[jgreco]

Because the layout of the forums is terrible, most likely.

 

[joeschmuck]

and Atlas on TrueNAS-SCALE-22.12-BETA.2.

I upgraded Atlas from Core 12 to 13.

Which one is it? Just sayin' you only mentioned Core in previous postings.

 

[rockybulwinkle]

Which one is it? Just sayin' you only mentioned Core in previous postings.

Thanks, I got the server names backwards in post #6.

 

[joeschmuck]

So you are running Core on one and Scale on the other.

Atlas is on TrueNAS-13.0-U3.1 and Titan on TrueNAS-SCALE-22.12-BETA.2.

And this is the configuration. Sounds good. I wasn't trying to be a prick about it, although after I read it back I felt like maybe I was. Sorry about that. I only saw conflicting information and wanted it to be accurate. Prior to posting #6, there was no mention of TrueNAS Scale which is where I was confused. Anyway, all is good.

It sounds like you have a plan to move forward and recover your data. It's a good thing for offsite backups! I should get one of those established.

 

[rockybulwinkle]

So you are running Core on one and Scale on the other.

And this is the configuration. Sounds good. I wasn't trying to be a prick about it, although after I read it back I felt like maybe I was. Sorry about that. I only saw conflicting information and wanted it to be accurate. Prior to posting #6, there was no mention of TrueNAS Scale which is where I was confused. Anyway, all is good.

It sounds like you have a plan to move forward and recover your data. It's a good thing for offsite backups! I should get one of those established.

It's all good, these sorts of details can matter for data recovery options.

Yeah I'm glad I had the backup. I'd definitely recommend you get one going, like you said. Also I've learned the value of having an on-site backup.

My plan going forward is to keep the mailed drive as an on-site cold backup. I'll reestablish a common snapshot between my local and remote backups, and put a hold on them so it won't easily be deleted accidentally. I'd be in a lot better shape right now, I think, if I had done these things.

 

[rockybulwinkle]

Yeah I'm glad I had the backup. I'd definitely recommend you get one going, like you said.

Really should get one going, at least for the most precious data (family photos, etc), even if the backup itself isn't fully redundant. By and far better than no backup at all!

 

[greyone]

Here's my full hardware specs:
Titan:
motherboard: ASRock B450 Pro4
CPU: Ryzen 3600
RAM: 32GB ECC
Pool: 2xWD100EMAZ-00WJTA0 in a mirror (shucked WD 10TB drives)
Disk controller: using the AMD chipset's SATA controller
Network: Realtek RTL8111/8168/8411 (I know this isn't recommended... but it hasn't caused me problems yet afaik and I am assuming problems would be network reliability, not data integrity).

Atlas:
Motherboard: Supermicro X9SCA-F
CPU: Xeon E31230@3.2GHz
RAM: 4GB ECC (again, I'm aware this isn't supported. It's next on my upgrade list)
Pool: 2xWDC WD140EDGZ-11B1PA0 in a mirror (shucked WD 14TB drives)
Disk controller: Intel 6 series/C200 (motherboard's built-in controller)
Network: Intel 82574L

Hello,
I intend to assemble almost the same TrueNAS Scale server as your Titan server and I have few questions to you. My goal is to migrate current setup to new ECC RAM enabled setup.

1. Will it work with this motherboard B450 Pro4 R2.0 ?
2. What type of ECC RAM are you using? I intend to use this one KSM26ED8/16HD.
3. Your Ryzen 3600 is just 3600 and not X?

Thank you in advance fot your help.

 

[rockybulwinkle]

Hello,
I intend to assemble almost the same TrueNAS Scale server as your Titan server and I have few questions to you. My goal is to migrate current setup to new ECC RAM enabled setup.

1. Will it work with this motherboard B450 Pro4 R2.0 ?
2. What type of ECC RAM are you using? I intend to use this one KSM26ED8/16HD.
3. Your Ryzen 3600 is just 3600 and not X?

Thank you in advance fot your help.

This is off topic, you should open a new thread in the General/Hardware section of the forum.

 

[rockybulwinkle]

Hello,
I intend to assemble almost the same TrueNAS Scale server as your Titan server and I have few questions to you. My goal is to migrate current setup to new ECC RAM enabled setup.

1. Will it work with this motherboard B450 Pro4 R2.0 ?
2. What type of ECC RAM are you using? I intend to use this one KSM26ED8/16HD.
3. Your Ryzen 3600 is just 3600 and not X?

Thank you in advance fot your help.

Let me clarify. Will it work? Yeah, probably! But yours is a separate discussion from what's talked about here, and for your own benefit, you should open a new thread. Detail your hardware selection like you already have, along with your intended use cases.

 

[rockybulwinkle]

I finally got my situation back in order a couple weeks ago.