pool import hangs at boot

[Davide Zanon]

Like long import times and memory requirements

Now I'm convinced more than ever that my problem is about insufficient memory,
I started the pool import on Friday at 12:48 and ran arcstat every 2 seconds to see
where it goes while importing:

It seems the system hanged about 23 hours later because it ran out of memory,
now the system is unresponsive and I must shut it down forcefully.
I wonder if I'm not setting things right, I thought these settings would be enough
to limit ARC usage to 12gb/24gb min/max:

Code:

sysctl vfs.zfs.arc_min=12884901888
sysctl vfs.zfs.arc_max=25769803776

Maybe I'm missing something else that is eating up memory during import?
I don't have other services running though, so I don't know why 21gb ARC
is enough to kiil the system, if I lower the max ARC usage would it just take
longer to import the pool or will it result on errors/problems?

 

[Davvo]

Why mess with the default ARC configuration?

 

[Arwen]

I don't think that the De-Dup table takes space inside of the ARC. But, I could be wrong.

 

[Davide Zanon]

Why mess with the default ARC configuration?

I don't think that the De-Dup table takes space inside of the ARC

I'm just trying to import the pool somehow, since ARC is eating up all the memory
I thought to limit its usage to let the system import the pool and get it over with.
Please do suggest other viable options, my efforts right now are only try&guess.
Thanks

 

[Davvo]

I'm just trying to import the pool somehow, since ARC is eating up all the memory

As it should, it's standard ZFS procedure to use all the system's memory.

I thought to limit its usage to let the system import the pool and get it over with.

I don't see how it could help? Quite possibily it's causing instability.

As a side note, I do remember DDT tables having to be stored in RAM. That might be the issue.

Can you please list your full hardware specs?

 

[Davide Zanon]

As it should, it's standard ZFS procedure to use all the system's memory.

But in my case it's not helping since it's making the system unusable.
How can I get out of this mess? (besides upgrading the system - I can't)

 

[Redcoat]

How full was the pool (what was the "Used Space" %) before you started the delete process that you mentioned in your first post of this string?

 

[Davide Zanon]

How full was the pool (what was the "Used Space" %) before you started the delete process that you mentioned in your first post of this string?

very full (almost 90% - I know, my bad), that's why I started the delete process.

 

[Davvo]

But in my case it's not helping since it's making the system unusable.

It's not the ARC fault if the system is unusable as it's dynamically allocated: if the system needs RAM, the ARC shrinks.

 

[Davide Zanon]

if the system needs RAM, the ARC shrinks.

Then I don't understand the output of arcstat, it clearly says "avail -491M",
I thought it was that that was eating all of system memory but if you say
it shrinks then I don't know what to to look at.

 

[Davvo]

Then I don't understand the output of arcstat, it clearly says "avail -491M",
I thought it was that that was eating all of system memory but if you say
it shrinks then I don't know what to to look at.

You changed the default behaviour, likely breaking something, thus obtaining such output.
Standard ZFS behaviour is use of ALL of the system memory, with none or very little unused. There is no concept of eating all memory.

 

[HoneyBadger]

Yes to all that.
This was a Netgear system that came with an IR-firmware, RAIDZ1 and dedup enabled.
A while ago I dumped the proprietary Netgear OS for Freenas (and now TrueNAS) but
I have to face with all these design problems that it came with originally.
Right now the data amounts to almost 50tb and I can't move it elsewhere to change
the RAIDZ1 and I can't disable dedup due to the nature of backups the system receives.
I just need to import it again and delete some stuff, I'll let you know tomorrow if the
operation finished successfully.

Thanks

Before I try to work out an estimated time-to-import, is this 50TB of logical data stored (before dedup) or 50TB of used space (after dedup?)

But spoiler alert, it isn't pretty. You're likely IOPS-bound on your deduplication table import and very likely don't have sufficient memory in the system. Quick napkin math puts you at 122GiB worth of DDT if that Netgear was using old defaults for recordsize.

 

[Davvo]

Nuking that will feel so good.

 

[Davide Zanon]

50TB of used space (after dedup?)

This

122GiB worth of DDT if that Netgear was using old defaults for recordsize

Is there a way to find out the recordsize before importing a pool?
I'm trying to retrieve a 64gb system to move the disks to, but that's
as far as I can go right now without buying a new system.

Nuking that will feel so good

I'll throw a party for sure, you're all invited!

 

[HoneyBadger]

This

Oof. The napkin-math about 122G DDT is probably on the money then.

Is there a way to find out the recordsize before importing a pool?
I'm trying to retrieve a 64gb system to move the disks to, but that's
as far as I can go right now without buying a new system.

Don't believe so, unless I can find a way to query zfs properties of an exported pool through ZDB - if it can be done from a top-level metadata approach without having to pull the DDT in, it may be possible - but the default pool/dataset recordsize in ZFS without manual override was 128K.

64G will help vs. the 24G you had specified as the earlier limit, but that's still only 50% of the needed RAM, so your DDT lookups have a 50% chance of having to go to disk; unless you manage to luck out and have all of the records living in a particular segment that stays live. I'd really recommend trying to get a hold of a system with 128G at a minimum. Try reaching out to a local Linux/Unix user group, and see if someone is willing to help you out with loaner hardware? Someone may have an older DDR3-era system they're willing to let you use for this purpose.

I'll throw a party for sure, you're all invited!

The food should be good if the popular thread in Off-Topic is any indication!

Cooking/Baking!

I can't cook/bake but when I do - I like to do a cheesecake cupcakes - this https://www.cookingforengineers.com/recipe/181/Cheesecake-Cupcakes helps a lot - not hard. The website hasn't been updated/active in a LONG time .. - Anyone got recipes that are easy to follow? I done pizza, but I...

www.truenas.com

 

[Davvo]

@Davide Zanon with a 64GB system you can slap a single 250GB SSD and use it to load the DDT just for the time necessary in order to get this sorted out. Likely cheaper than getting a different system.

 

[HoneyBadger]

@Davide Zanon with a 64GB system you can slap a single 250GB SSD and use it to load the DDT just for the time necessary in order to get this sorted out. Likely cheaper than getting a different system.

Unfortunately that horse has left the barn, crossed the field, and trotted down the road. A DDT/special vdev has to be in place beforehand in order to get the metadata written there, and one definitely can't be attached to an exported pool - so we come back to the same issue of the long import process.

Discarding the previous transactions (that included the delete operations) and rewinding to that state might be possible, but that's potentially going to put the pool in an even more fragile state. Doing a manual import with the -n (for noop) to test and -o readonly=on options if that succeeds might mitigate that, but I'm hesitant to suggest it as anything other than as a last resort.

 

[Davide Zanon]

A DDT/special vdev has to be in place beforehand in order to get the metadata written there

If I manage to import the pool can I attach the SSD then? I have loads of spare (non-enterprise) SSDs.

 

[Arwen]

If I manage to import the pool can I attach the SSD then? I have loads of spare (non-enterprise) SSDs.

Yes and no. The existing De-Dup table entries will likely stay exactly where they are. Only new entries will use the new Special vDev for De-dup.

Next, you need to make sure the Special vDev for De-Dup has at least the minimum redundancy as the main data vDev(s). For example;
RAID-Z1 - at least 2 way Mirror for the Special vDev for De-Dup
RAID-Z2 - at least 3 way Mirror for the Special vDev for De-Dup
...

This is because loss of ANY Special vDev will mean LOSS of the ENTIRE pool, no if, ands or buts.

Note that Cache / L2ARC, SLOG, or Hot Spare are not included in the above statement. They have different failure effects... Loss of Cache / L2ARC is completely harmless, except speed of reads. A Hot Spare not in use is also harmless. Loss of a SLOG at any time is also harmless, except after un-expected reboot when Sync writes were pending. Then, some data loss occurs, but is limited.

 

[HoneyBadger]

The existing De-Dup table entries will likely stay exactly where they are.

They will definitely stay exactly where they are if no actions are taken. Only a write that updates that specific DDT record (writes another duplicate copy or deletes an existing one) would move it.