Resilvering: Out of swap space -> processes getting killed

[Chris Moore]

Of course if one have hundreds of drives like you then the space saving (or lack thereof) would be more significant

In case it wasn't clear. Those hundreds of drives are at work, not my personal systems. We do keep spares. For the servers with over 60 drives, we have a minimum of five drives on hand and for the server that has 124 drives we have ten spares on hand.
I normally only keep two spares of each size for my systems at home.

 

[danb35]

If you want to do that from the WEB UI in FreeNAS, I don't think it will work

Sure it will, but...

I have not been able to find modern drives that differ in size to test with.

...because disk capacities are pretty much standardized now. But yes, the FreeNAS method will work through the GUI:

• Somehow get a replacement disk that's smaller than the rest of the disks in your vdev by less than 2 GB
• Set swap partition size to 0
• Replace failed/failing/removed disk with the new one
• FreeNAS will create a partition table, no swap partition, freebsd-zfs partition using the remaining space
• Because the freebsd-zfs partition is big enough, replacement works without issues

There's much less need for this now than there was 20 years ago, but I guess the capability's still good to have. And the system should have some swap anyway, though the devs clearly don't feel we need very much of it.

 

[Chris Moore]

the system should have some swap anyway, though the devs clearly don't feel we need very much of it.

I don't remember where I read this, but I was reading that TrueNAS doesn't do swap at all, but it is a much more tightly controlled configuration.

 

[HoneyBadger]

I don't remember where I read this, but I was reading that TrueNAS doesn't do swap at all, but it is a much more tightly controlled configuration.

As I recall, it has swap, but it's tuned to avoid ever hitting it, and if a TrueNAS appliance does hit swap, it's considered a Very Bad Thing and they will tune to avoid it. ;)

 

[Ender117]

Sure it will, but...

...because disk capacities are pretty much standardized now. But yes, the FreeNAS method will work through the GUI:

• Somehow get a replacement disk that's smaller than the rest of the disks in your vdev by less than 2 GB
• Set swap partition size to 0
• Replace failed/failing/removed disk with the new one
• FreeNAS will create a partition table, no swap partition, freebsd-zfs partition using the remaining space
• Because the freebsd-zfs partition is big enough, replacement works without issues

There's much less need for this now than there was 20 years ago, but I guess the capability's still good to have. And the system should have some swap anyway, though the devs clearly don't feel we need very much of it.

There is a potential problem that future replacement disk, even with the right size won't work unless you set swap partition=0 again. I think it's still best practice to do some calculation and make sure ZFS partition exactly matches.

As I recall, it has swap, but it's tuned to avoid ever hitting it, and if a TrueNAS appliance does hit swap, it's considered a Very Bad Thing and they will tune to avoid it. ;)

In my opinion all modern system should not ever hit swap. It's like safety belt, you must have it, but you should operate in a way that you would never need it.

 

[danb35]

There is a potential problem that future replacement disk, even with the right size won't work unless you set swap partition=0 again

Not at all. Let's say you have a pool consisting of 6 x 6000 GB disks in RAIDZ2. You need to replace one, but inadvertently buy a 5999 GB disk instead.

• Your current disks have a 2 GB swap partition, and a 5998 GB data partition
• You can't directly replace the 6000 GB disk with the 5999 GB disk, and you don't bother doing the math--you just set swap to 0 and then do the replacement
• The pool now consists of 5 x 5998 GB and 1 x 5999 GB partitions in RAIDZ2
• You set swap back to 2 GB in the GUI
  
• Your weird disk fails, so you need to replace it. You get a normal 6000 GB disk this time
• The normal replacement process works just fine--the data partition on the new disk will be the same size as the data partitions on all the other disks.

 

[HoneyBadger]

The pool now consists of 5 x 5998 GB and 1 x 5999 GB partitions in RAIDZ2

I don't think this is what happens. IIRC FreeNAS will check the existing partitions being used as vdevs and create one of matching size when swap=0.

Will see if I can test this out later.

 

[danb35]

IIRC FreeNAS will check the existing partitions being used as vdevs and create one of matching size when swap=0.

I didn't think so, but let's try. I've got a VM with 11.1-U6, so I just added three 100 GB virtual disks and one 99 GB one. Created a RAIDZ pool on the three, resulting in:

Code:

root@freenas:~ # gpart show
=>	  40  33554352  da0  GPT  (16G)
		40	  1024	1  bios-boot  (512K)
	  1064  33553320	2  freebsd-zfs  (16G)
  33554384		 8	   - free -  (4.0K)

(snip)

=>	   40  209715120  da2  GPT  (100G)
		 40		 88	   - free -  (44K)
		128	4194304	1  freebsd-swap  (2.0G)
	4194432  205520720	2  freebsd-zfs  (98G)
  209715152		  8	   - free -  (4.0K)

=>	   40  209715120  da3  GPT  (100G)
		 40		 88	   - free -  (44K)
		128	4194304	1  freebsd-swap  (2.0G)
	4194432  205520720	2  freebsd-zfs  (98G)
  209715152		  8	   - free -  (4.0K)

=>	   40  209715120  da4  GPT  (100G)
		 40		 88	   - free -  (44K)
		128	4194304	1  freebsd-swap  (2.0G)
	4194432  205520720	2  freebsd-zfs  (98G)
  209715152		  8	   - free -  (4.0K)

Try to replace da2 with da5 through the GUI, which initially resulted in a traceback (seems the GUI should handle that error condition better). Set the swap to 0 and tried again, which worked. Result:

Code:

root@freenas:~ # gpart show
=>	  40  33554352  da0  GPT  (16G)
		40	  1024	1  bios-boot  (512K)
	  1064  33553320	2  freebsd-zfs  (16G)
  33554384		 8	   - free -  (4.0K)

(snip)

=>	   40  209715120  da2  GPT  (100G)
		 40		 88	   - free -  (44K)
		128	4194304	1  freebsd-swap  (2.0G)
	4194432  205520720	2  freebsd-zfs  (98G)
  209715152		  8	   - free -  (4.0K)

=>	   40  209715120  da3  GPT  (100G)
		 40		 88	   - free -  (44K)
		128	4194304	1  freebsd-swap  (2.0G)
	4194432  205520720	2  freebsd-zfs  (98G)
  209715152		  8	   - free -  (4.0K)

=>	   40  209715120  da4  GPT  (100G)
		 40		 88	   - free -  (44K)
		128	4194304	1  freebsd-swap  (2.0G)
	4194432  205520720	2  freebsd-zfs  (98G)
  209715152		  8	   - free -  (4.0K)

=>	   40  207617968  da5  GPT  (99G)
		 40		 88	   - free -  (44K)
		128  207617872	1  freebsd-zfs  (99G)
  207618000		  8	   - free -  (4.0K)

Note that the freebsd-zfs partition on da5 is 99G, not 98G like the rest of them.

 

[HoneyBadger]

I didn't think so, but let's try.
...
Note that the freebsd-zfs partition on da5 is 99G, not 98G like the rest of them.

Well, that's just silly, in my opinion.

 

[Chris Moore]

Well, that's just silly, in my opinion.

Agreed, very unusual behavior.

 

[danb35]

Well, that's just silly, in my opinion.

Curious why you say this--it's exactly the behavior I expected. If you make a pool (or vdev) of mismatched disks, FreeNAS doesn't partition them all to the size of the smallest one; why would you expect it to do so in this case?

 

[HoneyBadger]

In my opinion, a REPLACE of an existing failed disk should do its best to recreate the data partitions as closely as possible, and FreeNAS having the layer of abstraction (and GUI) over top of ZFS should put up a dialog similar to:

"You appear to be replacing a failed disk. Are you currently attempting to grow your pool? (Y/N)"

If you say "No" or it's a hotspare, it should try to match the size of the existing partitions so that you don't have that little bit of weirdness.

 

[pro lamer]

Well, that's just silly, in my opinion.

Agreed, very unusual behavior.

From the manual (some older versions) :

Check and verify that the autoexpand property is enabled beforeattempting to grow the pool. If it is not, the pool will not recognize that the disk capacity has increased. By default, this property is enabled in FreeNAS® versions 8.3.1 and higher.

Then (edit)

Note that the freebsd-zfs partition on da5 is 99G, not 98G like the rest of them.

Then the swap space should not be set to zero in such cases but to the value of one.

Such cases = replacing with a smaller drive

Otherwise one wouldn't be able to replace it with any smaller one in future.

Imagine:
100GB drive / 2 GB swap
Then
99 GB drive / 1 GB swap
Then
98 GB drive / 0 GB swap
Then
97 GB drive... Ouch ;)

Or disable autoexpand temporarily.

should put up a dialog similar to:

"You appear to be replacing a failed disk. Are you currently attempting to grow your pool? (Y/N)"

Sounds like a good candidate for a feature request... (+1 for this ;) FWIW)

Edit: That could be done without tampering the ZFS - just GUI driven autoexpand param change there...

Sent from my mobile phone