Resilvering: Out of swap space -> processes getting killed

[Wanderhoden.]

Hi there,

so this is my first thread here, so please don't get mad if I miss some of the rules :p

I am currently upgrading my 5x2 TB zfs pool to 5x6 TB. FreeNAS began to kill processes (like bhyve and the GUI) during the first resilver. The local console throws an error every ~5 minutes, saying that the system is out of swap space.

1. Is it possible that data gets corrupted / lost, when there's not enough swap space left while resilvering?
Of course I created a data backup before detaching the first drive, but that option should only be the last resort.

2. How can I fix that problem? Enlarging the swap file or upgrading the available memory in the first place?
I still have a 480 GB SSD lying around, that I could use as a temporary swap drive.

Specs:
- FreeNAS 11.1U6
- Skylake Celeron G3900 @2,8GHz
- Gigabyte GA-X150M-PRO ECC
- 16 GB DDR4-ECC @2133MHz
- Pool: 5x 2TB drives, all mixed up (Toshiba & Seagate, 2.5 and 3.5"), currently at 82% capacity

 

[kdragon75]

what is running on the NAS? If you have bhyve running VMs, you may want to stop them during the resilver. Same for jails.

 

[Wanderhoden.]

Currently no jails and just 1 VM with minimum requirements, that I already shut down before the resilvering process.

 

[Chris Moore]

or upgrading the available memory in the first place?

Probably need more memory, but we need to know what RAIDz level you are using, RAIDz1 or RAIDz2?
Also, do you have "Autotune" enabled?

 

[Wanderhoden.]

I'm running a RAIDz2 on that pool. Autotune was disabled and the swap size was 2GB per drive.
I upgraded the RAM to 24GB and deactivated the high priority resilvering. Now the system seems stable while rebuilding the pool.

I just don't understand why the system becomes unavailable during resilvering. Even 16GB should have been more than sufficient.

 

[HoneyBadger]

Does the GUI report that you're actually using all of your available swap space? (Although if the GUI is crashing, you may need to use the command line and check manually using free)

 

[Wanderhoden.]

Sometimes netdata was working and it says, that the memory and the swap space are maxed out at 100 percent.

 

[joeschmuck]

You should submit a bug report for this, this problem should never occur in such a refined product and with the system min specs being 8GB RAM and you were running twice that with a SWAP space of 2GB per data drive.

 

[Chris Moore]

That's why I was asking if auto-tune had been enabled because even if it was disabled the settings would still be in there and that would cause it to more aggressively use swap space.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Wanderhoden.]

I finally managed to solve the issue. I tried:

- Upgrading the RAM from 16 to 24 GB
- Disabling the High Priority Resilvering under Storage -> Resilver Priority
- Manually enlarging the swap space from 2GB to 6GB per drive

The system stabilized, but now it's using 0% Swap and only 26% RAM (including ARC it should be 100% RAM usage, shouldn't it?)

 

[Chris Moore]

Manually enlarging the swap space from 2GB to 6GB per drive

Unless you deleted your entire pool and started from scratch, the actual swap space on disk will not have changed. It would require re-partitioning the drives.

Disabling the High Priority Resilvering under Storage -> Resilver Priority

This was probably more effective than anything.

including ARC it should be 100% RAM usage, shouldn't it?

If you rebooted the system, ARC would be dumped and would refill gradually based on use.

now it's using 0% Swap

Not using swap is the desired state.

I have limited my ARC memory utilization to keep some RAM available for running VMs.

 

[Chris Moore]

PS. ZFS on Linux puts the ZFS partition first and a (my term) cushion partition at the end of the drive. FreeNAS creates a swap partition at the beginning of the drive and the ZFS partition after, filling the remainder of the drive. I have changed my system to use a SSD drive for swap and I have no swap partition on my data drives.

 

[Wanderhoden.]

Unless you deleted your entire pool and started from scratch, the actual swap space on disk will not have changed. It would require re-partitioning the drives.

This was probably more effective than anything.

If you rebooted the system, ARC would be dumped and would refill gradually based on use.

Not using swap is the desired state.

View attachment 26115
I have limited my ARC memory utilization to keep some RAM available for running VMs.

I deleted none of my pools, so the swap size still should be 2 GB.
I also think that changing the resilver priority did the job.

I'm opening up a bug report, as soon as my drive upgrade is finished.
The resilver priority should be high when it's set to that value, but it shouldn't chew up your system as it did with mine

 

[pro lamer]

ZFS on Linux puts the ZFS partition first and a (my term) cushion partition at the end of the drive.

Do you mean OMV's ZFS plugin?

Sent from my mobile phone

 

[Chris Moore]

Do you mean OMV's ZFS plugin?

No. I have never used Open Media Vault. I really don't know anything about that software. I manage some servers at work that use RedHat Enterprise Linux with the Open ZFS (ZoL) installed to allow use of ZFS storage pools. If you simply point the current version of ZFS for Linux at the disks and create a pool, ZFS will automatically partition the drives with the larger ZFS partition first and a small secondary partition that has no purpose that I can discover. I discovered that behavior as part of my testing prior to system deployment, so I manually partition the drives and tell ZFS to use the partition for creating the pool to keep ZFS from creating the 'default' partition scheme. When you have over 100 drives in a server, you don't want to waste 2GB per disk on a partition that is not used. I have not tested this behavior on FreeNAS because I don't need to create pools from the command line, but I know that the GUI (middle-ware) creates the partitions on disk before it creates the pool and you can adjust (or eliminate) the size of the swap partition through the setting in the Advanced menu, however that only changes things if set before pool creation. I have done a fair bit of testing with the systems I manage to determine how they will behave under certain circumstances. I manually configure a SSD with a swap partition on it and adjust my pool disks to have no swap partition. I also make adjustments to try and prevent the system from hitting swap, but I still have swap space in the event of unplanned occurrences.

 

[Ender117]

No. I have never used Open Media Vault. I really don't know anything about that software. I manage some servers at work that use RedHat Enterprise Linux with the Open ZFS (ZoL) installed to allow use of ZFS storage pools. If you simply point the current version of ZFS for Linux at the disks and create a pool, ZFS will automatically partition the drives with the larger ZFS partition first and a small secondary partition that has no purpose that I can discover. I discovered that behavior as part of my testing prior to system deployment, so I manually partition the drives and tell ZFS to use the partition for creating the pool to keep ZFS from creating the 'default' partition scheme. When you have over 100 drives in a server, you don't want to waste 2GB per disk on a partition that is not used. I have not tested this behavior on FreeNAS because I don't need to create pools from the command line, but I know that the GUI (middle-ware) creates the partitions on disk before it creates the pool and you can adjust (or eliminate) the size of the swap partition through the setting in the Advanced menu, however that only changes things if set before pool creation. I have done a fair bit of testing with the systems I manage to determine how they will behave under certain circumstances. I manually configure a SSD with a swap partition on it and adjust my pool disks to have no swap partition. I also make adjustments to try and prevent the system from hitting swap, but I still have swap space in the event of unplanned occurrences.

I thought it was recommended to keep that cushion in event of slight drive size mismatch? But of course if you can stick to one model it won't be necessary

 

[Chris Moore]

I thought it was recommended to keep that cushion in event of slight drive size mismatch? But of course if you can stick to one model it won't be necessary

Absolutely. I do this at my own risk and I am not suggesting that anyone should follow my lead in this.
I just think that that idea would make more sense if the cushion was at the end of the drive, like ZFS on Linux does by default. That is not how FreeNAS makes the partition though. It makes the swap partition a fixed size based on the way it is defined in the system settings, with a default of 2, then it partitions the remainder of the drive for ZFS. If your drive size doesn't match what you are replacing, you are still going to have a problem and the existence (or lack thereof) of the swap partition isn't going to help.

 

[pro lamer]

If your drive size doesn't match what you are replacing, you are still going to have a problem and the existence (or lack thereof) of the swap partition isn't going to help.

I think (not from experience! ) it could work. I can imagine this the following way:

1. Bought a new, slightly smaller disk
2. Setup a smaller swap partition size
3. The data partition can start with a smaller LBA

Sent from my mobile phone

 

[Chris Moore]

I think (not from experience! ) it could work. I can imagine this the following way:

1. Bought a new, slightly smaller disk
2. Setup a smaller swap partition size
3. The data partition can start with a smaller LBA

Sent from my mobile phone

Certainly true, if you were partitioning the drive and manually initiating the replacement at the command line. If you want to do that from the WEB UI in FreeNAS, I don't think it will work but I have not been able to find modern drives that differ in size to test with. Historically, in the 90s, it was not unusual to find small differences in drive capacity even from the same manufacturer. I have not seen that in the TB range drives over the last few years. Probably because the drives only report the amount of free space that their firmware is programmed to report, regardless of the amount of mechanical space on disk. All modern drives are manufactured with spare space that is reserved to the drive for reallocating sectors and other management functions that we are not privileged to know. If I received a drive that did not match the space it was supposed to have, I would send it back as being defective.
I honestly don't expect to see that problem. Consistency in the reported capacity is much greater now than it has ever been before, in my experience.

 

[Ender117]

Certainly true, if you were partitioning the drive and manually initiating the replacement at the command line.

I think that would be good enough, mismatch are rare these days like you said. Of course if one have hundreds of drives like you then the space saving (or lack thereof) would be more significant and they should be keeping spares that has been verified to work anyway.