Pool Offline after removing Dedup Disk

[Topshot]

Hello,
I am very new to TrueNAS and just did the same thing the person on this thread did.

I lost my Dedup VDev Disk and Pool UNAVAIL

I added an Optane SSD for Dedup VDev some days ago and tested dedup performance, but it seems not very good. Then I try to delete the SSD from zpool and use it for other things, but failed. So I shut down the TrueNAS host and unplugged the SSD then erased it for other systems. When I turn on...

www.truenas.com

Show : See My TrueNAS Disk config

TrueNAS Core Version - TrueNAS-12.0-U8.1
boot 128 GB SSD
Pool in question:
vDev 16TB WD Red
removed Dedub 240 GB SSD
current Dedub 128 GB SSD

Short Version
I removed and formated a Dedub Disk and my Pool appears as Offline.

Long Version
I have a pool and wanted to add a Dedub. Used an SSD that was way too big to test it out. Added a new smaller SSD to the pool as a Dedub disk, removed the bigger SSD and erased it to use it as a Cache disk (erasing it before testing was not very smart). Result is, that my Pool is Offline.

The good thing I still have all the data on my PC the bad thing it's scattered and a couple of TB.

Question
Did anything change in TrueNAS Core in the last year, making it possible to repair/export or import the pool or do I have to wipe my disk and lose 2 weeks of copy & pasting data.
(as the person in above mentioned thread had to do)

Secondary Question
Am I able to replace Dedub & Cache disks if they break not loosing Data as I might just did?
Should I add Dedub & Cache Drives to my NAS for the reason I just mentioned (sounds risky)?

I hope I included all needed information & Thank you for helping out :)

 

[jgreco]

I removed and formated a Dedub Disk and my Pool appears as Offline.

Whoops.

I have a pool and wanted to add a Dedub. Used an SSD that was way too big to test it out. Added a new smaller SSD to the pool as a Dedub disk,

I haven't tried this, but, ... what?!?

Again, not having tried this, I would expect that you'd need to add a mirror to the existing dedup disk. Adding a smaller one suggests it added it as a second dedup vdev, which seems weird but I guess at scale there could be reasons to need that ability. The data on the dedup SSD is unique in that it contains the pointers to the actual target block, etc., so loss of the dedup table is fatal for the pool.

removed the bigger SSD and erased it to use it as a Cache disk

It allowed you to remove a mandatory component of your pool without giving you a warning? If so, this absolutely needs to be brought up as a bug.

(erasing it before testing was not very smart).

Well, yes, but, damn, it should absolutely have warned you, at least, or probably should have refused to let you do it at all, since that's a pool killer.

 

[Topshot]

Thanks for the answer.
As you said big Whoops ^^ But you learn with mistakes.

It allowed you to remove a mandatory component of your pool without giving you a warning? If so, this absolutely needs to be brought up as a bug.

I removed it and had to format it in another PC because I have not found a way of erasing a disk that is in use in TrueNAS. I still don't get how to remove a disk from a pool (like Dedub or Cache etc.). So no bug just user error in this case.

you'd need to add a mirror to the existing dedup disk

Is that a thing and how would you do it, I saw no option while installing the second Dedub SSD (was not looking for one though).

that's a pool killer

So I guess there is no way back and I have to format everything and start at the beginning?
Leaving my last question open, why would you want to risk your data for a Dedub and is it the same thing for Cache?

 

[Arwen]

L2ARC Cache device(s) are simple, and only contain copies of recently used data & metadata. Generally L2ARC Cache device(s) are intended to be on faster device(s) than the main pool devices. The regular pool vDevs have the "original" copies, so any loss of a L2ARC Cache is non-fatal. Their can be small issues, (not really problems), in importing a pool with a missing or failed L2ARC Cache vDev. But. the pool can be imported without data loss.

De-Dup, (note the spelling is with a P / Pee, as in De-Duplicate, not B / Bee), devices are critical and part of the pool. This is why it is best practices to have the De-Dup, (or other special vDev), built with the same redundancy amount as the main pool. Meaning if you are using RAID-Z2 on the main pool, you would use 3 way mirroring of any De-Dup vDev or special vDev.

De-Dup is generally not recommended for casual users because of memory and management constraints. Pools with De-Dup enabled can get slower over time, as the De-Dup table increases in size.

Next, you can't generally shrink vDevs. Thus, going from large De-Dup device to a smaller device would likely entail full backup, (preferably multiple backups), re-configure pool, and full restore. I don't know if you can use ZFS to remove a De-Dup device. And then add a newer, smaller De-Dup device.

---

If you really did yank, (aka removed the SSD without using the GUI tools), the De-Dup device, and then wiped it, your data is almost certainly gone for good. (At least as far as the TrueNAS instance is concerned.)

This is one reason I sometimes suggest that new users try TrueNAS, (Core or SCALE), in a VM first. And read lots of the Resources and sticky forum threads. ZFS is different enough from anything that came before, their is lots of potential gotchas to new users.

 

[Topshot]

Thanks for the detailed answer :)
yeah I figured that the data is lost (on my Pool), formatted everything and started to copy my data back on a new created pool.

Would be awesome if there was a way to remove De-Dup Disks but I guess hard to implement with a lot of complexity.
Another way would be to warn the user about the risk when installing a De-Dup.

De-Dup, (note the spelling is with a P / Pee, as in De-Duplicate, not B / Bee)

Small Typo on my end but Bees are more important than Duplication anyway :)

With the last answer I would close this thread, got my answer that the pool was lost for good and will not install a De-Dup again.

 

[AlexGG]

If you really did yank, (aka removed the SSD without using the GUI tools), the De-Dup device, and then wiped it, your data is almost certainly gone for good. (At least as far as the TrueNAS instance is concerned.)

TrueNAS instance yes, but the dedup table is not really required to read the filesystem. It is only required for writes. From the standpoint of data recovery (I mean expensive specialized software), missing dedup vdev is not a factor.

 

[winnielinnie]

TrueNAS instance yes, but the dedup table is not really required to read the filesystem. It is only required for writes. From the standpoint of data recovery (I mean expensive specialized software), missing dedup vdev is not a factor.

How is data recovery even feasible if multiple files are using the same (duplicate) records? Isn't the Dedup table necessary to know what belongs to what?

Record #100456 is part of File A, File G, and File Y.

Without the Dedup table (and hence the Dedup vdev which has been nuked), how would any file recovery tool recover File A, File G, and File Y? It might be able to read Record #100456, but that's not enough, right? There's no longer anything to connect this same record to the several different files.

EDIT: I might be misunderstanding the Dedup table.

I thought it also played a role in pointing records to files, but that's not the case? It's still the same filesystem metadata (per dataset) that does this, regardless if Dedup has been enabled on a pool or not?

Then if that is the case, why is losing the Dedup vdev such an "irreversible pool killer"?

What is stopping ZFS from detecting the loss of the Dedup vdev and then proceeding to work as normal, but this time without checking for duplicate records upon writing new data? (Akin to an emergency failsafe mode.)

Why does your entire pool go down just because you lost the ability to "skip writing already-existing records" without the Dedup table? Why can't ZFS just intelligently detect the loss of the Dedup vdev and simply disable the deduplication feature from that point forwards?

 

[AlexGG]

How is data recovery even feasible if multiple files are using the same (duplicate) records? Isn't the Dedup table necessary to know what belongs to what?

No. The dedup table is not needed at all to figure out what data belongs to what file. Each block pointer tree, associated with each specific file, contains enough data to determine what is what and where it is, without consulting the dedup table. The dedup table is (maybe) required for writes (and probably not even that). Frankly I did not look into the dedup table itself in any detail. In other words, the filesystem is not implemented the way you expect it is.

 

[winnielinnie]

The dedup table is (maybe) required for writes (and probably not even that).

I edited my above post and added another hypothetical question.

Because if this is true (about Dedup only in play for "writes"), then I don't understand why losing the Dedup vdev kills your entire pool. You would think there would be fail-over/failsafe mechanisms in place for losing such a vdev.

 

[AlexGG]

It's still the same filesystem metadata (per dataset) that does this, regardless if Dedup has been enabled on a pool or not?

Yes.

I don't understand why losing the Dedup vdev kills your entire pool. You would think there would be fail-over/failsafe mechanisms in place for losing such a vdev

Me too.

When I did the tests earlier today, I fully expected to be able to mount pool readonly via some switch of zpool import, but no.

Arguably, there may be a problem writing some metadata, which may also be deduplicated (why not).

 

[winnielinnie]

And another thing...

If Deduplication can be "disabled" without killing your pool, that's the equivalent to removing a Dedup vdev and disabling Deduplication.

There are even guides on enabling/disabling Deduplication on your ZFS pool, with the caveat that "Disabling deduplication does not remove deduplication on already existing data, however, new writes will not use the space-saving efficiency of deduplication." They never mention anything about how "You will lose everything if you disable it after it has already been enabled!"

You'd think in the decades that ZFS has been developed and improved, there would be a built-in mechanism that simply... disables Deduplication when it detects that the Dedup vdev is missing; keeping your existing data safe.

 

[AlexGG]

There are even guides on enabling/disabling Deduplication on your ZFS pool

There is a significant difference between disabling something in controlled manner, when you have unlimited time to do whatever cleanup you want and make things consistent with whatever requirements you want for safe subsequent use, and having a significant part of the filesystem yanked from under you without any chance to take a last look at it. These things are extremely complex, with many interactions and possibilities to take into account.

 

[Topshot]

fail-over/failsafe

Wouldn't storing a copy of the dup-table on the pool itself be a solution. It would only be used if the main de-dup fails but helps safe all the data, and in the case of a fail you could easily recover on a new de-dup disk.

 

[AlexGG]

Wouldn't storing a copy of the dup-table on the pool itself be a solution. It would only be used if the main de-dup fails but helps safe all the data, and in the case of a fail you could easily recover on a new de-dup disk.

No. The dedup table needs very frequent updates. Moving it to a specialized device (read: SSD mirror) provides the required performance. If you have a copy on the main pool, you must update the copy, which negates the improvement of having a dedicated SSD mirror.

 

[HoneyBadger]

When I did the tests earlier today, I fully expected to be able to mount pool readonly via some switch of zpool import, but no.

In cases like this, it's often beyond simple zpool import flags and well into the Here Be Dragons world of Forbidden Tunables, that open you up to the possibility of returning corrupted data. Feel free to read up on zfs_max_missing_tvds if you want to play with fire.

Would be awesome if there was a way to remove De-Dup Disks but I guess hard to implement with a lot of complexity.

They never mention anything about how "You will lose everything if you disable it after it has already been enabled!"

Note that we're not only talking about controlled removal as @AlexGG mentioned, but different situations.

Disabling deduplication with an internal DDT (not on a separate vdev) is done by simply disabling the setting, but this doesn't rehydrate any existing deduped data and until it's done you still have the overhead of the DDT in RAM.

Removing a dedup vdev (even in a controlled manner) doesn't automatically disable deduplication. You will still need to disable it as a ZFS feature. But removing the ddt vdev(s) means changing the on-disk location of the ddt, which means you need to meet the requirement of "top-level vdev removal" and one of the major caveats there is "no top-level RAIDZ allowed"

So if you have a RAIDZ2 pool and add a mirrored dedup/special vdev - you're stuck with it. Only way out of that at present is "back up data, delete pool, recreate."

 

[Arwen]

Disabling De-Dup is not fatal, it's just like disabling compression. All new writes will not use that feature. But, as has been pointed out, disabling De-Dup does not re-hydrate any existing de-duplicated blocks. Nor eliminate the overhead for processing the existing de-duplicated blocks.