How to relocate swap to an SSD or other partition

[Stux]

Stux submitted a new resource:

How to relocate swap to an SSD or other partition - creating a swap device, and replacing pool swap with a Post-Init command

FreeNAS by default will allocate 2GiB on each pool HD and use this as swap. Unfortunately, if a disk fails, your system may crash.

If you have an SSD, or another device/partition for use as swap it can be beneficial to relocate swap to that device.

At the same time it is beneficial to leave the default pool swap where it is because it allows you the ability to replace...

Read more about this resource...

 

[Stux]

@Scharbag you might be interested in this...

 

[MrToddsFriends]

creating a swap device, and replacing pool swap with a Post-Init command

Thanks for this guidance. Hmmm, still not sure if I should use a dedicated swap device as suggested or if I should wait for mirrored swap partitions in 11.1.

 

[Stux]

Thanks for this guidance. Hmmm, still not sure if I should use a dedicated swap device as suggested or if I should wait for mirrored swap partitions in 11.1.

I've been using a lot ESXi recently, and I can simply pass in a virtual disk for swap. Alternatively, I can use part of my SLOG device as swap too

 

[tahoward]

As of 11.1 this does not appear to work:

Added the following as post init command in FreeNAS:

Code:

swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "/dev/gptid/55da6c0b-feda-11e7-8a1d-001b21be7fae.eli none swap sw 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -a

Rebooted and ran swapinfo:

Code:

Device		  1K-blocks	 Used	Avail Capacity
/dev/mirror/swap0.eli   2097152		0  2097152	 0%
/dev/mirror/swap1.eli   2097152		0  2097152	 0%
/dev/mirror/swap2.eli   2097152		0  2097152	 0%
/dev/mirror/swap3.eli   2097152		0  2097152	 0%
/dev/mirror/swap4.eli   2097152		0  2097152	 0%
/dev/gptid/55da6c0b-feda-11e7-8  16777216		0 16777216	 0%
Total			27262976		0 27262976	 0%

Seems the vdev swap space is still persisting.

Edit: There should be a /dev/mirror/swap5.eli in there. Perhaps this only replaces the last swap device?
Edit Again: I disabled the init script and rebooted and swapinfo still only showed 5 vdev swap devices even though my pool consists of 6 vdevs. Not sure if this is normal.

 

[svtkobra7]

@Stux ... hoping you can lend a friendly hand here and let me know your thoughts on this ...

Supposition:

• Incorrect former swap configuration causing issues?
• Encrypted volumel causing issues?

Old Config (Not your instructions)

• Followed this guide: https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/
• System > Advanced > Swap size | changed to 0
• Via Shell:

Code:

dd if=/dev/zero of=/usr/swap0 bs=1m count=4096
chmod 0600 /usr/swap0

• Added Post Init Command:
  Code:

  echo "md99 none swap sw,file=/usr/swap0,late 0 0" >> /etc/fstab && swapon -aL

Going In Position / Swap

• Removed post init command
• Via SSH:
  Code:

  swapoff -a

• Shutdown

Going In Position / Swap Drive

• Created new 16GB vDisk, Thick provisioned, eagerly zeroed
• Created new NVMe Controller
• Assigned new vDisk to new NVMe Controller

New Swap Config

Code:

root@FreeNAS:~ # swapinfo
Device		  1K-blocks	 Used	Avail Capacity

• Via Shell:

Code:

gpart create -s gpt nvd1
   nvd1 created

Code:

gpart show nvd1
=>	  40  33554352  nvd1  GPT  (16G)
		40  33554352		- free -  (16G)

• Via Shell:

Code:

gpart add -i 1 -b 128 -t freebsd-swap nvd1
   nvdp1 added

Code:

root@FreeNAS:~ # gpart show nvd1
=>	  40  33554352  nvd1  GPT  (16G)
		40		88		- free -  (44K)
	   128  33554264	 1  freebsd-swap  (16G)

Code:

root@FreeNAS:~ # glabel status | grep nvd1
gptid/6bf4c238-801e-11e8-8dd5-000c2999bfe2	 N/A  nvd1p1

Code:

root@FreeNAS:~ # gpart list nvd1
Geom name: nvd1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 33554391
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: nvd1p1
   Mediasize: 17179783168 (16G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   rawuuid: 6bf4c238-801e-11e8-8dd5-000c2999bfe2
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 17179783168
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 33554391
   start: 128
Consumers:
1. Name: nvd1
   Mediasize: 17179869184 (16G)
   Sectorsize: 512
   Mode: r1w1e2

• Added Post Init Command:
  Code:

  swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "/dev/gptid/6bf4c238-801e-11e8-8dd5-000c2999bfe2 none swap sw 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -a

Reboot Time!

• No swap

Code:

root@FreeNAS:~ # swapinfo
Device		  1K-blocks	 Used	Avail Capacity

• Unlock Volume
• Swap appears?

Code:

root@FreeNAS:~ # swapinfo
Device		  1K-blocks	 Used	Avail Capacity
/dev/nvd1p1.eli  16777132		0 16777132	 0%

• Run the post init manually:

Code:

root@FreeNAS:~ # swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "/dev/gptid/6bf4c238-801e-11e8-8dd5-000c2999bfe2.eli none swap sw 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -a
swapoff: removing /dev/nvd1p1.eli as swap device
swapon: /dev/gptid/6bf4c238-801e-11e8-8dd5-000c2999bfe2.eli: Invalid parameters

• Change gptid to /dev/nvd1p1.eli and run command:
  Code:

  root@FreeNAS:~ # swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "/dev/gptid/6bf4c238-801e-11e8-8dd5-000c2999bfe2.eli none swap sw 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -a

• Change post init command
• Reboot
• System reports swap before and after encrypted volume unlock.
• I do see a message in the footer that savecore: /dev/nvd1p1: Operation not permitted

Any clue what is going on here? Do I have swap set up correctly or no? Thanks in advance!

 

[Stux]

As of 11.1 this does not appear to work:

Added the following as post init command in FreeNAS:

Code:

swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "/dev/gptid/55da6c0b-feda-11e7-8a1d-001b21be7fae.eli none swap sw 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -a

Rebooted and ran swapinfo:

Code:

Device		  1K-blocks	 Used	Avail Capacity
/dev/mirror/swap0.eli   2097152		0  2097152	 0%
/dev/mirror/swap1.eli   2097152		0  2097152	 0%
/dev/mirror/swap2.eli   2097152		0  2097152	 0%
/dev/mirror/swap3.eli   2097152		0  2097152	 0%
/dev/mirror/swap4.eli   2097152		0  2097152	 0%
/dev/gptid/55da6c0b-feda-11e7-8  16777216		0 16777216	 0%
Total			27262976		0 27262976	 0%

Seems the vdev swap space is still persisting.

Edit: There should be a /dev/mirror/swap5.eli in there. Perhaps this only replaces the last swap device?
Edit Again: I disabled the init script and rebooted and swapinfo still only showed 5 vdev swap devices even though my pool consists of 6 vdevs. Not sure if this is normal.

post a copy of your /etc/fstab file (in code tags)

 

[Stux]

@Stux ... hoping you can lend a friendly hand here and let me know your thoughts on this ...

Supposition:

• Incorrect former swap configuration causing issues?
• Encrypted volumel causing issues?

Old Config (Not your instructions)

• Followed this guide: https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/
• System > Advanced > Swap size | changed to 0
• Via Shell:

Code:

dd if=/dev/zero of=/usr/swap0 bs=1m count=4096
chmod 0600 /usr/swap0

• Added Post Init Command:
  Code:

  echo "md99 none swap sw,file=/usr/swap0,late 0 0" >> /etc/fstab && swapon -aL

Going In Position / Swap

• Removed post init command
• Via SSH:
  Code:

  swapoff -a

• Shutdown

Going In Position / Swap Drive

• Created new 16GB vDisk, Thick provisioned, eagerly zeroed
• Created new NVMe Controller
• Assigned new vDisk to new NVMe Controller

New Swap Config

Code:

root@FreeNAS:~ # swapinfo
Device		  1K-blocks	 Used	Avail Capacity

• Via Shell:

Code:

gpart create -s gpt nvd1
   nvd1 created

Code:

gpart show nvd1
=>	  40  33554352  nvd1  GPT  (16G)
		40  33554352		- free -  (16G)

• Via Shell:

Code:

gpart add -i 1 -b 128 -t freebsd-swap nvd1
   nvdp1 added

Code:

root@FreeNAS:~ # gpart show nvd1
=>	  40  33554352  nvd1  GPT  (16G)
		40		88		- free -  (44K)
	   128  33554264	 1  freebsd-swap  (16G)

Code:

root@FreeNAS:~ # glabel status | grep nvd1
gptid/6bf4c238-801e-11e8-8dd5-000c2999bfe2	 N/A  nvd1p1

Code:

root@FreeNAS:~ # gpart list nvd1
Geom name: nvd1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 33554391
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: nvd1p1
   Mediasize: 17179783168 (16G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r1w1e1
   rawuuid: 6bf4c238-801e-11e8-8dd5-000c2999bfe2
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 17179783168
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 33554391
   start: 128
Consumers:
1. Name: nvd1
   Mediasize: 17179869184 (16G)
   Sectorsize: 512
   Mode: r1w1e2

• Added Post Init Command:
  Code:

  swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "/dev/gptid/6bf4c238-801e-11e8-8dd5-000c2999bfe2 none swap sw 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -a

Reboot Time!

• No swap

Code:

root@FreeNAS:~ # swapinfo
Device		  1K-blocks	 Used	Avail Capacity

• Unlock Volume
• Swap appears?

Code:

root@FreeNAS:~ # swapinfo
Device		  1K-blocks	 Used	Avail Capacity
/dev/nvd1p1.eli  16777132		0 16777132	 0%

• Run the post init manually:

Code:

root@FreeNAS:~ # swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "/dev/gptid/6bf4c238-801e-11e8-8dd5-000c2999bfe2.eli none swap sw 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -a
swapoff: removing /dev/nvd1p1.eli as swap device
swapon: /dev/gptid/6bf4c238-801e-11e8-8dd5-000c2999bfe2.eli: Invalid parameters

• Change gptid to /dev/nvd1p1.eli and run command:
  Code:

  root@FreeNAS:~ # swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "/dev/gptid/6bf4c238-801e-11e8-8dd5-000c2999bfe2.eli none swap sw 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -a

• Change post init command
• Reboot
• System reports swap before and after encrypted volume unlock.
• I do see a message in the footer that savecore: /dev/nvd1p1: Operation not permitted

Any clue what is going on here? Do I have swap set up correctly or no? Thanks in advance!

You removed the swap partitions from your disks right?

Not sure what side-effects that has on the orginal swap process that FreeNAS sets up. Its possible it skips it... and then there's not a lot for the post-init script to do.

What you need to do is disable the script... then rebooot... then put a copy of your /etc/fstab in code quotes... both before and after unlocking your encrypted media.

 

[svtkobra7]

Thanks, as always, for your kind help.

You removed the swap partitions from your disks right?

• Perhaps I too blindly followed that guide, but I never had issues before.
• I'm not sure where that script creates the md99 device, but I used that approach since day 1, so I'm not sure I ever had swap on HDDs (possible)? Heck, I've actually never even seen reported swap utilization (Reporting > Memory > Swap utilization).
• Wouldn't gpart show easily reveal any/all swap partitions? If so, the ONLY disk I see with "FreeBSD-swap" = vDisk created as other than boot they are all "FreeBSD-ZFS."
  
  Code:

  																				
  =>		 40  11721045088  da14  GPT  (5.5T)								   
  		   40		   88		- free -  (44K)							   
  		  128  11721044992	 1  freebsd-zfs  (5.5T)						   
    11721045120			8		- free -  (4.0K)							  
  

Not sure what side-effects that has on the orginal swap process that FreeNAS sets up. Its possible it skips it... and then there's not a lot for the post-init script to do.

• To eliminate any/all side effects, would a clean install / default config restore + upload of config unwind any ill effects from that prior command?
• I really think it has something to do with the fact that my Pool is encrypted … isn't it too coincidental that the gptid is destroyed as soon as I unlock my pool the first time? Not sure of why a geli attach to 4 disks would result in a geli destroy of encrypted swap, but just seems like its too coincidental.

What you need to do is disable the script... then rebooot... then put a copy of your /etc/fstab in code quotes... both before and after unlocking your encrypted media.

• post init command disabled.
• Pre geli attach

Code:

[root@FreNAS ~]# cat /etc/fstab											  
freenas-boot/grub	   /boot/grub	  zfs	 rw,noatime	  1	   0	  
fdescfs /dev/fd fdescfs rw	  0 0											
/dev/nvd1p1.eli none	swap	sw	  0	   0																					 

• Post geli attach

Code:

[root@FreeNAS ~]# cat /etc/fstab											  
freenas-boot/grub	   /boot/grub	  zfs	 rw,noatime	  1	   0	  
fdescfs /dev/fd fdescfs rw	  0 0											
/dev/nvd1p1.eli none	swap	sw	  0	   0							 
​

Code:

[root@FreeNAS ~]# swapinfo													 
Device		  512-blocks	 Used	Avail Capacity						   
/dev/nvd1p1.eli   33554264		0 33554264	 0%		

I believe if I execute swapoff -a and then reboot I don't see the swap via swap info …

Any thoughts?​

 

[AVSION]

@Stux hope you can help, i have the same issues when with the swap after reboot it wont use the 900p partition unless i run it manually after reboot, see below all the information. any idea why the swap not using the 900p partition after reboot? all the system info is in my signiture

Thank you

Code:

root@freenas[~]# glabel status | grep nvd0
gptid/6ff84a55-c0a1-11e8-9f75-0cc47adb3218	 N/A  nvd0p2
gptid/76d01ebd-c0a1-11e8-9f75-0cc47adb3218	 N/A  nvd0p3
gptid/6762c737-c0a1-11e8-9f75-0cc47adb3218	 N/A  nvd0p1

Code used

Code:

root@freenas[~]# swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "/dev/gptid/6762c737-c0a1-11e8-9f75-0cc47adb3218.eli none swap sw 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -a

before reboot

Code:

root@freenas[~]# swapinfo	  
Device		  1K-blocks	 Used	Avail Capacity
/dev/gptid/6762c737-c0a1-11e8-9  16777216		0 16777216	 0%

Code:

root@freenas[~]# cat /etc/fstab
fdescfs /dev/fd fdescfs rw	  0 0
/dev/gptid/6762c737-c0a1-11e8-9f75-0cc47adb3218.eli none swap sw 0 0

after reboot

Code:

root@freenas[~]# cat /etc/fstab
fdescfs /dev/fd fdescfs rw	  0 0
/dev/mirror/swap0.eli   none	swap	sw	  0	   0
/dev/mirror/swap1.eli   none	swap	sw	  0	   0
/dev/mirror/swap2.eli   none	swap	sw	  0	   0
/dev/mirror/swap3.eli   none	swap	sw	  0	   0

Top After Reboot

Code:

Welcome to Secure Shell App version 0.8.45.
Answers to frequently asked questions: https://goo.gl/muppJj (ctrl+click on links to open)
ChangeLog/release notes: https://goo.gl/YnmXOs
last pid:  3857;  load averages:  0.38,  0.19,  0.08																															up 0+00:04:15  12:39:50
59 processes:  1 running, 58 sleeping
CPU:  0.0% user,  0.0% nice,  0.1% system,  0.0% interrupt, 99.9% idle
Mem: 514M Active, 136M Inact, 759M Wired, 14G Free
ARC: 312M Total, 70M MFU, 225M MRU, 66K Anon, 2774K Header, 14M Other
	81M Compressed, 245M Uncompressed, 3.00:1 Ratio
Swap: 8192M Total, 8192M Free

Permissions

Code:

ls -al /dev/
dr-xr-xr-x   2 root  wheel	  512 Sep 26 12:35 fd
crw-------   1 root  wheel	 0x27 Sep 26 12:35 fido
crw-rw-rw-   1 root  wheel	  0xe Sep 26 12:35 full
crw-r-----   1 root  operator   0x4 Sep 26 12:35 geom.ctl
dr-xr-xr-x   2 root  wheel	  512 Sep 26 12:35 gpt
dr-xr-xr-x   2 root  wheel	  512 Sep 26 12:35 gptid

Code:

root@freenas[~]# ls -al /dev/gptid
total 1
dr-xr-xr-x   2 root  wheel	  512 Sep 26 12:35 .
dr-xr-xr-x  12 root  wheel	  512 Sep 26 12:35 ..
crw-r-----   1 root  operator  0xab Sep 26 12:35 243c617b-7de8-4bee-837b-af6374bd0469
crw-r-----   1 root  operator  0x9a Sep 26 12:38 6762c737-c0a1-11e8-9f75-0cc47adb3218
crw-r-----   1 root  operator  0xbc Sep 26 12:43 6762c737-c0a1-11e8-9f75-0cc47adb3218.eli
crw-r-----   1 root  operator  0xac Sep 26 12:35 6e36fb44-758a-43de-9d2b-79d9e4618ca5
crw-r-----   1 root  operator  0x4c Sep 26 12:35 6ff84a55-c0a1-11e8-9f75-0cc47adb3218
crw-r-----   1 root  operator  0x4d Sep 26 12:35 76d01ebd-c0a1-11e8-9f75-0cc47adb3218
crw-r-----   1 root  operator  0xa3 Sep 26 12:35 b5b40bb4-4e99-11e7-9cba-0cc47adb3218
crw-r-----   1 root  operator  0x9d Sep 26 12:35 b69faf01-4e99-11e7-9cba-0cc47adb3218
crw-r-----   1 root  operator  0x9b Sep 26 12:35 b792e7b8-4e99-11e7-9cba-0cc47adb3218
crw-r-----   1 root  operator  0x9f Sep 26 12:35 b87cfa92-4e99-11e7-9cba-0cc47adb3218
crw-r-----   1 root  operator  0xa1 Sep 26 12:35 b969428a-4e99-11e7-9cba-0cc47adb3218
crw-r-----   1 root  operator  0xa5 Sep 26 12:35 ba5cf8b5-4e99-11e7-9cba-0cc47adb3218
crw-r-----   1 root  operator  0xa7 Sep 26 12:35 bb4ae668-4e99-11e7-9cba-0cc47adb3218
crw-r-----   1 root  operator  0x86 Sep 26 12:35 bbc10cb7-c123-11e8-b380-000c29782429
crw-r-----   1 root  operator  0xa9 Sep 26 12:35 bc3d4265-4e99-11e7-9cba-0cc47adb3218

EDIT: I fixed it by doing the following Removing the post init script, restart, run the swap code manually, add the init code save and restart.
i did try it before and it didn't work i thing you have to do it in this order that was the difference.

 

[kaboompow]

As of 11.1 this does not appear to work:

Added the following as post init command in FreeNAS:

Code:

swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "/dev/gptid/55da6c0b-feda-11e7-8a1d-001b21be7fae.eli none swap sw 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -a

Rebooted and ran swapinfo:

Code:

Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap0.eli   2097152        0  2097152     0%
/dev/mirror/swap1.eli   2097152        0  2097152     0%
/dev/mirror/swap2.eli   2097152        0  2097152     0%
/dev/mirror/swap3.eli   2097152        0  2097152     0%
/dev/mirror/swap4.eli   2097152        0  2097152     0%
/dev/gptid/55da6c0b-feda-11e7-8  16777216        0 16777216     0%
Total            27262976        0 27262976     0%

Seems the vdev swap space is still persisting.

Edit: There should be a /dev/mirror/swap5.eli in there. Perhaps this only replaces the last swap device?
Edit Again: I disabled the init script and rebooted and swapinfo still only showed 5 vdev swap devices even though my pool consists of 6 vdevs. Not sure if this is normal.

Has anyone been able to get this working in 11.x? I'm also running into the same issues.

 

[Mastakilla]

Has anyone been able to get this working in 11.x? I'm also running into the same issues.

How to relocate swap to the boot drive

I noticed the release notes for 11.1 showed 'Support for mirrored swap partitions'. Does this occur automatically (and is there way to verify/determine) or is this something that has to be configured through the gui/cli? I did not find a reference in the manual, and I'm curious what 'support'...

www.ixsystems.com

Seems like /etc/fstab is no longer storing the swap devices and also "swapoff -a" doesn't seem to disable the mirrored swap on your pool, so I'm afraid the post-init command is no longer valid...

Not sure what the proper way to do this is either. Or if it even is still advised to do this.

 

[fasavard]

Hi,
Do you know a way it could be posible to apply this to a jail.

Right know i did everything and insinde Freenas,all is good.
Once i boot a jail (Resilio sync) i get the SSDswap, but the regular swap is also there.

 

[GBillR]

Not sure what the proper way to do this is either. Or if it even is still advised to do this.

Hate to revive a thread this old, but I do not see an answer to this anywhere, and I am seeing the same behavior.

Is relocating the swap like this possible anymore with 11.x or higher?

 

[Mastakilla]

I'm currently using the default mirrored swap devices on the pool