SWAP file and partitions on disks

[Johnny Fartpants]

Hi All,

I've been trawling through lots of threads both inside and outside of the freenas forums to get my answer but Im still none the wiser.

Everything I've learned about ZFS tells me that you shouldn't use partitions and you should give it whole disks. However by default freenas creates a 2GB swap file/partition on every disk added to the pool. There also seems to be some concern about having your swap file spread across all the disks in the pool although Im not 100% sure if these concerns are valid?

The reason I ask is I like the idea of using glabel to name all my disks before I create a pool but freenas doesn't allow you to do this. The only way is to create a pool from the CLI after naming your disks and then exporting it to then import it via freenas UI. However this doesn't create the swap partition and gives ZFS the whole disk.

I'd be interested to hear your thoughts on this and if anyone has had any practical experience of creating a pool this way and running it without a swap file. Most of the storage systems I use have 64GB of RAM or more so do I really need a swap?

Appreciate your thoughts.

 

[Ericloewe]

The reason I ask is I like the idea of using glabel to name all my disks before I create a pool

The question is, what do you hope to accomplish?

 

[Spearfoot]

Hi All,

I've been trawling through lots of threads both inside and outside of the freenas forums to get my answer but Im still none the wiser.

Everything I've learned about ZFS tells me that you shouldn't use partitions and you should give it whole disks. However by default freenas creates a 2GB swap file/partition on every disk added to the pool. There also seems to be some concern about having your swap file spread across all the disks in the pool although Im not 100% sure if these concerns are valid?

The reason I ask is I like the idea of using glabel to name all my disks before I create a pool but freenas doesn't allow you to do this. The only way is to create a pool from the CLI after naming your disks and then exporting it to then import it via freenas UI. However this doesn't create the swap partition and gives ZFS the whole disk.

I'd be interested to hear your thoughts on this and if anyone has had any practical experience of creating a pool this way and running it without a swap file. Most of the storage systems I use have 64GB of RAM or more so do I really need a swap?

Appreciate your thoughts.

As you pointed out, you can configure FreeNAS so that it doesn't create a swap partition on new disks, but I strongly recommend that you set swap up on your system. I boot from an SSD and have plenty of space on the FreeNAS boot volume, so I created a swap file on it and specify it in a startup script. And I set the System->Advanced->Swap size on each drive in GiB, affects new disks only. Setting this to 0 disables swap creation completely (STRONGLY DISCOURAGED). value to zero before creating my pool.

 

[HoneyBadger]

Everything I've learned about ZFS tells me that you shouldn't use partitions and you should give it whole disks.

This is correct for illumos where ZFS will not force-enable write cache if given a partition, in case an additional partition on the drive is using another filesystem (eg: UFS.)

On the FreeBSD implementation all ZFS pools have whole_disk set to true and will always have write-cache enabled.

Partitions are perfectly safe. On the note of swap though, I'd suggest leaving per-drive swap creation enabled in case of mismatched LBAs on disk replacement (you can adjust the swap size manually to make it work), but relocating swap as suggested by @Spearfoot is advisable simply because if you happen to hit swap (if you did, you messed up) and a disk fails, part of your memory could be swapped to the failed disk and then Bad Stuff Happens.

One of multiple reasons why I've started going with mirrored SSD for installs when I really care about uptime.

Reference: OpenZFS Wiki
(http://open-zfs.org/wiki/Performance_tuning#Whole_Disks_versus_Partitions)

 

[Ericloewe]

One of multiple reasons why I've started going with mirrored SSD for installs when I really care about uptime.

That doesn't change anything. Boot devices don't have swap by default and swap in 9.x is not protected by a GEOM RAID1, as it will be in 10.

Are you applying this manually?

 

[HoneyBadger]

Are you applying this manually?

Of course.

https://www.freebsd.org/doc/handbook/adding-swap-space.html

 

[pjc]

That doesn't change anything. Boot devices don't have swap by default and swap in 9.x is not protected by a GEOM RAID1, as it will be in 10.

Sorry to resurrect an old thread, but I'm just now beginning a migration from 9.x up to 11, and I'm now realizing I didn't grok how FreeNAS was handling swap before. It appears it uses the swap present on any drive that's attached, is that right?

I haven't had a drive fail yet, but it seems that this arrangement means that a single drive failure would likely cause a machine crash in 9.x, no? Would it handle things gracefully if it's not using any swap space currently?

Is this single-point-of-failure no longer the case in FreeNAS 9.3.10 (10.x FreeBSD) and 11?

 

[Ericloewe]

FreeNAS 11.1 introduced mirrored swap, so this is no longer a serious concern.

 

[pjc]

How does the mirrored swap work? For example, if I have a 5-drive RAID-Z2, that'll be 10GB of raw swap space across 5 drives. Will it treat that as 4GB of mirrored (2x2)?

 

[Ericloewe]

Yes.

 

[pjc]

So half the usable space is still enough swap? Or is FreeNAS 11.1 now allocating 4GB per drive to compensate?

Separately, I'm going to be expanding a pool via zpool attach/detach from the CLI. Does "zpool detach" cleanly remove that drive from swap usage? Or is there something in the GUI that's doing something clever when detaching pools/devices?

 

[Ericloewe]

So half the usable space is still enough swap?

Yes. Ideally, zero swap is enough swap.

from the CLI

Why?

Does "zpool detach" cleanly remove that drive from swap usage?

No.

Or is there something in the GUI that's doing something clever when detaching pools/devices?

Yes, it has to go and change the swap settings.

 

[pjc]

Why?

Because I want to add both additional mirrors before detaching the old drives. I didn't see any way to do that in the GUI?

If I reboot, will it simply set up swap based on the currently available drives, or will it yell about missing swap drives if I've pulled the old drives while the system is off?

And, finally, in the new mirrored configuration, if a drive fails, does it operate swap in degraded mode, or does it use the opportunity to gracefully decommission the non-redundant portion of swap?

 

[Ericloewe]

Because I want to add both additional mirrors before detaching the old drives. I didn't see any way to do that in the GUI?

You just replace disks without first disconnecting the old ones: http://doc.freenas.org/11/storage.html#replacing-drives-to-grow-a-zfs-pool

 

[pjc]

Right, but that would go from 2-disk mirror to 2-disk+resilvering and then immediately offline & remove the old drive, leaving it as a 2-disk mirror. I want to go from a 2-disk mirror to a 4-disk mirror, which there seems to be no way to do from the GUI.

Eventually I plan to remove the old 2 drives. But until I'm ready, I want to be able to back out of the planned expansion easily (by detaching the new drives, without requiring a full resilver of the original drives).

 

[Ericloewe]

I want to go from a 2-disk mirror to a 4-disk mirror,

Yeah, you'd need to do it manually. That's been a feature request for years now, which got buried with the whole Corral fiasco...

If I reboot, will it simply set up swap based on the currently available drives, or will it yell about missing swap drives if I've pulled the old drives while the system is off?

It should set things up correctly.

And, finally, in the new mirrored configuration, if a drive fails, does it operate swap in degraded mode, or does it use the opportunity to gracefully decommission the non-redundant portion of swap?

Good question. Fill us in if you find out.

 

[danb35]

I want to go from a 2-disk mirror to a 4-disk mirror

Do you want to move to a four-way mirror, or do you want to create two mirrored sets, striped together? If the latter, you can easily do that through the GUI. If the former, you're right that you'll need to do it through the CLI.

 

[pjc]

Definitely a 4-disk mirror. I'll eventually drop the original two, at which point I want the remaining pair to be a redundant mirror.

I can also double-check the swap behavior from these manual changes by rebooting and seeing if FreeNAS automatically recognizes and uses the swap space on the new (manually attached) drives. If it does, then I think I can offline and detach the original pair via the GUI?

 

[Ericloewe]

As an aside, this is the first time I've ever seen anyone who wants a four-way mirror. Give us some zpool status outputs of that, for the historical record.

 

[Stux]

Fairly certain FreeNAS rebuilds the swap tables with each disk insertion event. That’s how it worked in 11.0 ;)

swapon/swapoff/swapinfo

Simplest thing is just to do

swapoff -a before removing drives. Then swapon -a to re-enable swap.