Encryption performance with Core-i3 AES-NI

[klayman]

Hi there,

I just started with FreeNAS and have a question about write performance on my freshly built system. First of all: we're talking about hobby / private use here. I'm storing a large amount of data for photography on the device, some office stuff, and of course media files. Photos and office are backed up regularly on external disks, media files are not (don't care). I do want to have encryption as there are some confidential files as well. My idea thus was building a NAS server from the following components:

- Intel Core i3-4130 (with AES-NI support)
- Asus H87I-Plus mITX Board
- 8 GB RAM
- 5 x 4TB Seagate NAS HDD in RaidZ1

The objective was hardware support for encryption (because all those QNAP/Syno devices do really bad at encryption) and low power consumption. Now, set up and running I was testing the system for its performance and was a bit surprised: I only get about 35-40MByte write performance over a gigabit link, but around 100-110 Mbyte read performance. During writes CPU load is around 99%.
Then I set up an unencrypted RaidZ1 and - surprise, surprise - write performance peaked around 90-100 MByte. CPU load then was around 40-50 with short peaks of 80%. Soooo, big question here: is there anything I can do about encryption performance?? I always thought that AES-NI would more or less solve the problem by significantly reducing CPU load, but maybe I did get that wrong.

best regards,
Klayman

 

[warri]

This sounds weird, I'm sure an AES-NI enabled CPU should do a lot better. Can you check if AES-NI is enabled? Run dmesg | grep AESNI from the CLI.

 

[cyberjock]

Few comments:

1. Your CPU is a 2 core 4 thread. AES-NI doesn't do HT, so you really have 2 cores. Not the most ideal, but we can figure out where the bottleneck is.
2. You can run some benchmarks to figure out how fast your CPU is capable of running AES-NI code by checking out http://forums.freenas.org/threads/encryption-performance-benchmarks.12157/ Please add your CPU benchmarks to the thread so others can see how your CPU does.
3. AES-NI is supposed to improve your encryption performance, but other factors also can affect performance. ZFS uses some CPU resources and CIFS can use a lot too. You might not have enough to do everything with your CPU. Let's see what the CPU benchmarks say before we start speculating as to the cause.
4. You have 8GB of RAM with about 20TB of disk space. That's quite a bit short. If I were designing a system for you with 20TB of disk space I would have gone with 16GB of RAM minimum. Using encryption does require geli to use RAM, so you could be putting enough pressure on your system's RAM to cause ZFS performance to be poor with encryption but be fine without that added pressure.

Let's see what warri's request says and what your benchmark values are.

 

[Dusan]

2. You can run some benchmarks to figure out how fast your CPU is capable of running AES-NI code by checking out http://forums.freenas.org/threads/encryption-performance-benchmarks.12157/ Please add your CPU benchmarks to the thread so others can see how your CPU does.

You can also run openssl speed aes-256-cbc to test raw single core AES-256 throughput (openssl will use AES-NI if available). You can even run openssl speed aes-256-cbc -multi <number_of_threads> to check multi core performance. The last number (8196 bytes block) will be very similar to the one you get via the geli test, just this test is much simpler to execute.

 

[cyberjock]

AES-256 isn't what the encrypted pools use, so its not as good of a benchmark as the thread I made and linked above.

 

[Dusan]

AES-256 isn't what the encrypted pools use, so its not as good of a benchmark as the thread I made and linked above.

Encrypted pool partitions use AES-256 in XTS mode, swap uses AES-128 in XTS mode. CBC (Cipher-block chaining) mode is a bit faster than XTS mode (Xor-encrypt-xor based tweaked-codebook mode with ciphertext stealing), however it is still a good benchmark to compare AES performance. A CPU that's twice as fast doing AES-CBC will also basically be twice as fast doing AES-XTS. I still think that you would get much more responses in the benchmark thread if it only required to run a single command vs. several steps that include booting a FreeBSD image.

 

[cyberjock]

Encrypted pool partitions use AES-256 in XTS mode, swap uses AES-128 in XTS mode. CBC (Cipher-block chaining) mode is a bit faster than XTS mode (Xor-encrypt-xor based tweaked-codebook mode with ciphertext stealing), however it is still a good benchmark to compare AES performance. A CPU that's twice as fast doing AES-CBC will also basically be twice as fast doing AES-XTS. I still think that you would get much more responses in the benchmark thread if it only required to run a single command vs. several steps that include booting a FreeBSD image.

I agre that if I could have run a single command that would have been preferred. But I was in very uncharted waters at the time and alot of people wanted to test benchmark speeds without upgrading FreeNAS. The simplest solution was to use FreeBSD itself. There was also some problems with running geli from the CLI and it was possible to kernel panic the box(as I did a few times on my test box before I chose to go with FreeBSD). Even now, my experience with geli is very light(But I'm fairly knowledgable with AES and the differencea between XTS and CBC) and I really can't pick someone's brain to answer my questions. So a lot of it is PFM(Pure F*$#* magic) to me.

Pretty sure encrypted pools are 128bit and swap is 256bit. I've had this discussion before, so unless its changed recently I'm 99% sure that's what it is.

Edit: Ok, I just tested with my system. Despite it seeming to be backwards my swapspace is AES-XTS 256 and my zpool is AES-XTS-128 for 100% certainty. :)

When I had this discussion before the short answer is that 128 is faster than 256(duh) but isn't that much more secure in the big picture. Yes, it is technically, but there's no attack vector that makes 128 so weak that 256 would be a more secure solution anytime within our lifespans.

 

[Dusan]

Pretty sure encrypted pools are 128bit and swap is 256bit. I've had this discussion before, so unless its changed recently I'm 99% sure that's what it is.

Ah, you are right I wasn't paying attention when looking at geli list. It's 256 bits for swap and 128 for data. So, openssl speed aes-128-cbc then.

 

[cyberjock]

You are too fast.. just edited my post. ;P

 

[klayman]

ok, back we are :)

dmesg | grep aes gives me "<AES-CBC,AES-XTS> on motherboard", but only when booted into FreeNAS. When using FreeBSD live CD 9.1 (loaded from USB image) I don't get anything. The commands posted in cyberjock's performance threat result in below output. All was done with 16GB RAM. Once confirmed I did everything right I'll also update the performance threat.

regards,
Klayman

Edit: I guess the dmesg command didn't find anything because the module wasn't loaded...

root@:/root # kldload geom_eli
cryptosoft0: <software crypto> on motherboard
root@:/root # kldload geom_zero
root@:/root # sysctl kern.geom.zero.clear=0
kern.geom.zero.clear: 1 -> 0
root@:/root # geli onetime -s 4096 -l 256 -e aes-xts gzero
GEOM_ELI: Device gzero.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: software
root@:/root # dd if=/dev/gzero.eli of=/dev/null bs=1m count=4096
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 177.652049 secs (24176289 bytes/sec)
root@:/root # geli kill gzero
GEOM_ELI: gzero has been killed
GEOM_ELI: Device gzero.eli destroyed
root@:/root # kldunload geom_zero
root@:/root # kldunload geom_eli
cryptosoft0: detached

root@:/root # kldload aesni
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
root@:/root # kldload geom_zero
root@:/root # sysctl kern.geom.zero.clear=0
kern.geom.zero.clear: 1 -> 0
root@:/root # geli onetime -s 4096 -l 256 -e aes-xts gzero
GEOM_ELI: Device gzero.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: hardware
root@:/root # dd if=/dev/gzero.eli of=/dev/null bs=1m count=4096
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 111.532367 secs (38508707 bytes/sec)
root@:/root # geli kill gzero
GEOM_ELI: gzero has been killed
GEOM_ELI: Device gzero.eli destroyed

 

[Dusan]

Something is definitely wrong. Only 24MB/s without AES-NI and only 39MB/s with AES-NI. The benchmark thread shows that even the low power 1.6GHz AMD E-350 can do 30MB/s without AES-NI. You should be seeing 140+MB/s (AES-NI) with your CPU. Is it possible that your CPU is being throttled down for some reason? Overheating?

 

[cyberjock]

Is powerd enabled? If so disable it and reboot.

Check your CPU temps with sysctl -a |egrep -E "cpu\.[0-9]+\.temp" I believe.


I don't think this can be your problem, but can you check the BIOS settings and make sure its enabled if there is an option to enable/disable it?

 

[klayman]

I'll reload the BIOS with the defaults, disable all power saving options and do the test again. It is a normal boxed CPU cooler, nothing fancy.

 

[klayman]

That did the trick!! I now got the 4GB transferred in 26 seconds, resulting in roughly 156 MB/s. In my quest for low power consumption I set the BIOS to hand over the system to the OS in "Max Battery Mode". Apparently FreeNAS doesn't by default do any power management.

thanks & best regards,
Klayman

 

[cyberjock]

Apparently FreeNAS doesn't by default do any power management.

No,its that the BIOS overrides most power management controls in OSes.

FreeNAS does have its own power management options. It's called powerd.[/quote]

 

[cyberjock]

Actually, thanks to the code change I think that its safe to say that the openssl test isn't as good as actually doing a geli device test. Unfortunately I have to figure out how to do a device on a RAMdisk or equivalent for benchmark testing of the 9.2.0 RC, using FreeNAS and not FreeBSD, with ZFS. Glad I have 20GB of RAM. Just have to come up with the actual commands now. If you know how to do this Dusan I'm all ears. You seen to know more about this stuff than me.

 

[Dusan]

"mdconfig -a -t malloc -o nocompress -s 1g" will create an 1G ramdisk device. It will print the name of the device to console. In FreeNAS, the first you create will be /dev/md3 as md0~2 are already used by the system. I checked the mdconfig.c source and it seems it compresses the ramdisk by default so I added the nocompress option. You may want to test what performance impact it has. You can use /dev/md3 as dd target. To destroy it run "mdconfig -d -u 3". You can also add geli on top of the ramdisk. Running "geli onetime /dev/md3" will create /dev/md3.eli. You can then use that to create a zpool: "zpool create -m /mnt/tank tank /dev/md3.eli"

 

[jyavenard]

using 9,2-RC

$ sudo dd if=/dev/gzero.eli of=/dev/null bs=1m count=4096
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 3.311322 secs (1297055173 bytes/sec)

1.3GB/s... pretty impressive...

That's a E3-1220V3 processor (quad-core 3.1GHz)

 

[cyberjock]

My spare system aka POS (e5606, 2.13Ghz socket 1366) went from 110MB/sec to 750MB/sec for 1 thread. That's just 1 device. Theoretically you can multiply that by the number of threads(4 in my case) and get 3GB/sec(before zfs overhead, etc etc etc). That's about what I was getting in Windows with truecrypt so that makes more sense.

I don't think there is any AES-NI compatible processor that is worse than the one I have. So I think it's safe to say that nobody will ever be bottle-necked with encryption if their CPU is AES-NI compatible ever again. I'll post more info in my encryption benchmarks thread and probably close the thread as there's no point to continue to run any benchmarks for encryption. It's already pretty well known that you either use AES-NI with encryption or you don't use encryption. Performance is just horrible without it.

 

[Durandal]

That did the trick!! I now got the 4GB transferred in 26 seconds, resulting in roughly 156 MB/s. In my quest for low power consumption I set the BIOS to hand over the system to the OS in "Max Battery Mode". Apparently FreeNAS doesn't by default do any power management.

thanks & best regards,
Klayman

What motherboard do you have? I'm just going to build a new FreeNAS machine with a Core i3 4130T CPU and a SuperMicro X10 motherboard. Do you know if the "Max battery mode" disables all power saving features or just some?