Encryption seems to be interfering a bit with unencrypted system

[jgreco]

So I'm doing some qualification of a new 12 x 4TB pool being built here, and so I threw the latest FreeNAS on the 2011-based ESXi system on the bench.

Code:

Build 	FreeNAS-8.3.1-RELEASE-p2-x64 (r12686+b770da6_dirty)
Platform 	Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz

Only having seven of the drives in inventory right now, I thought it'd be fun to play and see what sort of performance, etc. Well, almost right away we had an infant mortality, and while looking at that, I noticed that the encryption framework appears to be getting loaded and activated even though no encryption has been specified. When I did the disk replacement and resilvered, afterwards the new disk device had an updated /dev/da3p1.eli entry and there were kernel messages indicating the configuration of the device.

Code:

[root@freenas] /mnt/test# ls -l /dev/*eli
crw-r-----  1 root  operator    0, 152 May 26 12:45 /dev/da3p1.eli
crw-r-----  1 root  operator    0, 129 May 24 21:28 /dev/da4p1.eli
crw-r-----  1 root  operator    0, 131 May 24 21:28 /dev/da5p1.eli
crw-r-----  1 root  operator    0, 133 May 24 21:28 /dev/da6p1.eli
crw-r-----  1 root  operator    0, 135 May 24 21:28 /dev/da7p1.eli
crw-r-----  1 root  operator    0, 137 May 24 21:28 /dev/da8p1.eli
crw-r-----  1 root  operator    0, 139 May 24 21:28 /dev/da9p1.eli

I was also seeing what appeared to me to be unusually high system and CPU loading while doing the resilver, and now that the system is idle, something in kernel-land is stuck eating one of the two assigned cores.

Code:

last pid: 26781;  load averages:  1.02,  1.11,  1.64                                             up 1+16:05:12  13:32:57
26 processes:  1 running, 25 sleeping
CPU:  0.0% user,  0.0% nice, 50.0% system,  0.0% interrupt, 50.0% idle
Mem: 152M Active, 7620K Inact, 4465M Wired, 12K Cache, 142M Buf, 3280M Free
ARC: 4181M Total, 282M MFU, 3879M MRU, 18K Anon, 13M Header, 7422K Other
Swap: 16G Total, 340K Used, 16G Free

Destroying the pool doesn't seem to fix that but does cause geli to destroy all the devices, but this is a necessary step because I must go back and make sure that this isn't an "insufficient coffee" mistake like maybe I accidentally clicked "use encryption" instead of "use 4096" and it's just that none of the encryption option boxes showed up in the GUI.

So recreating the pool, for-sure with encryption NOT checked, and it still gets created apparently using the geli devices. And then panics when I reboot it. But at least after I reset it, it isn't chowing 100% of one of its cores anymore, and the CPU loading appears more reasonable.

Rolling it back to 8.3.0, the pool imports fine, and so it isn't clear what the encryption module is doing, or why it is loaded for each disk in the pool if it isn't being used. Just a heads-up to BE CAREFUL.

 

[titan_rw]

I didn't think there was a way to disable encryption of swap devices?

Or are the p1 partitions the zpool? If swap size is set to 0, then p1's should be the pool right? If so, I hope it's not simply geli-ing all p1's regardless of whether they're swap or not.

I tried it in a vm running the latest 9.1, and I get this:

Code:

GEOM_ELI: Device da1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Detached da1p1.eli on last close.
GEOM_ELI: Detached da2p1.eli on last close.
GEOM_NOP: Device da1p1.nop created.
GEOM_NOP: Device da2p1.nop created.
GEOM_NOP: Device da1p1.nop removed.
GEOM_NOP: Device da2p1.nop removed.

So it seems like it creates and then immediately destroys the .eli devices. Then creates the .nop devices for pool creation?

After the pool is created, there is no .eli devices in /dev.

 

[jgreco]

Ah, yes, that's it. I just installed a fresh image and it defaults to swap and it's encrypting the swap. That seems unnecessary and potentially unwise. I just wanted to make a record of what I was seeing because I was fighting several simultaneous battles all at once. I still haven't figured out what was causing it to eat a core, suspect something to do with the disk replacement.

Been fun, everything's fighting me, IRQ storms, failed drives, ESXi not supporting 10gigE, MSI support for the M1015's, etc. But boy FreeNAS is really cool if you start throwing resources at it.

 

[titan_rw]

True.

At some point I'd like to make a decent 10gigE freenas box running a socket 2011 xeon of some kind and start at 64 gigs of ram.

A bit overkill for home use, but it would be a fun project. And with 10gig coming down, it's cheaper and cheaper to move to.