Encryption performance benchmarks

Status
Not open for further replies.

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Edit: Please go down and read the last post on this page. Starting with 9.2.0 a fix was made that made AES-NI so fast that pool performance will not be significantly affected by encryption unless you plan to run 10Gb network hardware and have a pool that should be doing multi-GB/sec performance. Basically if you have an AES-NI compatible CPU you won't notice the performance impact of encryption. If you don't have an AES-NI compatible CPU performance will be significantly impacted(aka don't do it).

This thread is to discuss and compare various CPUs and how they perform with and without AES-NI CPUs. This discussion is intended to provide users with a comparison of potential performance numbers if encryption is used and what CPUs can provide what performance since 8.3.1 has been released and some users are curious as to what kind of performance penalty can be expected from using encryption.

You can check if your CPU supports AES-NI by running the command # dmesg | grep aes from FreeNAS 8.3.1(If you run this command from anything before 8.3.1 you will see the features line if AES-NI is supported but the aesni0 device will not exist). If you do not get an output that looks like what is below your CPU doesn't support AES-NI.

Code:
# dmesg | grep aes
Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>
aesni0: <AES-CBC,AES-XTS> on motherboard
 
The following instructions were performed using a FreeBSD 9.1 64-bit CD in "Live CD" mode. There shouldn't be a significant difference in performance between 32-bit and 64-bit for encryption. However, zpools aren't recommended on a 32-bit OS because of the 4GB RAM limitations(remember that the manual recommends no less than 6GB of RAM to use ZFS). These commands do not work in FreeNAS and if you try to run them you will crash the system.

Instructions for testing:

1. Boot the FreeBSD 9.1 x64 CD.
2. Choose "Live CD" when asked if you want to install or use the Live CD.
3. Enter the username "root" when the login prompt is displayed. No password is required.

Perform the following steps to test a NON-AES-NI compatible CPU or to test an AES-NI compatible CPU with AES-NI disabled:

(The bold will be the commands and the non-bold should be roughly the expected output)

Code:
# kldload geom_eli
cryptosoft0: <software crypto> on motherboard
# kldload geom_zero
# sysctl kern.geom.zero.clear=0
kern.geom.zero.clear: 1 -> 0
# geli onetime -s 4096 -l 256 -e aes-xts gzero
GEOM_ELI: Device gzero.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software
# dd if=/dev/gzero.eli of=/dev/null bs=1m count=4096
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 76.077181 secs (56455395 bytes/sec)
# geli kill gzero
GEOM_ELI: gzero has been killed.
GEOM_ELI: Device gzero.eli destroyed.
# kldunload geom_zero
# kldunload geom_eli


Perform the following steps to test an AES-NI compatible CPU with AES-NI enabled:

Code:
# kldload aesni
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
# kldload geom_zero
# sysctl kern.geom.zero.clear=0
kern.geom.zero.clear: 1 -> 0
# geli onetime -s 4096 -l 256 -e aes-xts gzero
GEOM_ELI: Device gzero.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
# dd if=/dev/gzero.eli of=/dev/null bs=1m count=4096
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 41.142665 secs (104392054 bytes/sec)
# geli kill gzero
GEOM_ELI: gzero has been killed.
GEOM_ELI: Device gzero.eli destroyed.
# kldunload geom_zero
# kldunload geom_eli
# kldunload aesni


To keep things organized if you provide your values please include the following in red so that future posters can sift through and separate conversation from actual raw data:

CPU model and frequency: Intel Xeon E5606 @ 2.13Ghz (don't forget to correct the speed if you overclock/underclock)
CPU maximum number of cores/threads: 4/4 (included for HT/non-HT CPUs)
AES-NI speeds: 104MB/sec (If not supported please include this line with 'N/A')
NON-AES-NI speeds: 56MB/sec

So what do these values tell me?

First, they tell me that I can encrypt about twice the data per second with AES-NI versus non-AES-NI. AES-NI is enabled by default if your CPU supports AES-NI.

Second, if I had a zpool with 4 or more drives(since my CPU supports only 4 threads) and there was no service or zpool overhead and the hard drives/zpool weren't the bottleneck then I could expect a maximum of 416MB/sec(104MB/sec x 4). Unfortunately, depending on your services, total system RAM, zpool type and size, etc these values will only go down. The 416MB/sec is theoretical and should not be taken as values you could actually attain with your system and typical sharing services enabled and in use. Additionally, any zpool scrub in progress on encrypted hard drives would seriously impact the performance of the file sharing services.

My zpool currently is a RAIDZ3 of 18x2TB drives. DD tests bench my zpool at 900-1000MB/sec. I'd expect that if I were to use encryption on my zpool I'd get about 200-300MB/sec maximum. Since I use two 1Gb LAN ports I would certainly be hurting my LAN speeds if both were heavily used. Also noteworthy is that my scrubs would take significantly longer to perform.

If you have a system that you'd like to provide values for, please include them in this thread for comparison. If we get a lot of data on various CPUs(especially the commonly requested CPUs such as Atoms and low powered AMDs) I'll use the second post to provide a list of all of the data for easy access.

Thanks,

-Cyberjock

Notes:

1. The benchmarks provided only provide the maximum theoretical performance of encryption. In a production environment you can expect different numbers because of the increased overhead of the file sharing services used, ZFS overhead, and added latency from the disks themselves.
2. Geli supports 1 thread per "device". A device would typically mean a hard disk. The following benchmark values will only give you the performance of a single CPU core/thread. If you want to max out your potential performance you should have the same number of hard drives in the zpool or UFS raid as you do cores. For example, if you only have 3 hard drives but your CPU supports up to 8 simultaneous threads you will only have 3 threads available for encryption. This does have the advantage of providing the other 5 "cores/threads" for all of the other services the system uses. If you chose to use a hardware RAID(which isn't recommended for ZFS), then the entire RAID would count as one device(read: very poor performance). If you use RAIDZ2 you must include the total number of hard drives in the zpool and not the number of drives excluding redundancy.
3. AES-NI is a relatively new technology. Only certain models of Intel CPUs released in Q1'11 or newer and some AMD CPUs from Q3'11 and newer support AES-NI. A list of the Intel CPUs that support AES-NI can be found here.

Found this benchmark on the internet: http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071608.html

CPU model and frequency: Intel Atom D525 @ 1.8Ghz
CPU maximum number of cores/threads: 2/4
AES-NI speeds: N/A
NON-AES-NI speeds: 14.7MB/sec

Here's another one of my machines:

CPU model and frequency: Intel Atom D2500 @ 1.86Ghz
CPU maximum number of cores/threads: 2/2
AES-NI speeds: N/A
NON-AES-NI speeds: 16.2MB/sec
 

titan_rw

Guru
Joined
Sep 1, 2012
Messages
586
I tested two of my machines.

The first one was the laptop I tested the live usb flash drive on. Booted it up to make sure the flash drive was ok. Figured I might as well run the test.

CPU: Intel i5-430m @ 2.26 ghz (stock). (no aes-ni support, software only)
Cores: 2, Threads: 4.

software aes:
65638967 bytes / sec, or 62.6 mb/sec.


Second machine tested is my primary freenas box.

CPU: Intel i5-3570k @ 3.4 ghz (stock). (aes-ni supported)
Cores: 4, Threads 4.

software aes:
98506237 bytes / sec or 94 mb/sec.

hardware aes:
152860570 bytes / sec or 145.8 mb/sec.


I can write to the pool of this machine (10 drive z3) at about 1,000 mb/sec. So encryption would definitely slow it down. I imagine gigabit ethernet wouldn't be a problem, but scrubbing would definitely take longer. It currently scrubs at about 1,100 mb/sec. But it uses upwards of 75% of the cpu during scrub, no encryption. So I imagine scrubbing would be significantly slower.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
Well I saw this posting and thought I'd run a little testing and share the results...

My Main Computer
CPU model and frequency: Intel i7-950 @ 3.07Ghz
CPU maximum number of cores/threads: 4/8 (HT is enabled)
AES-NI speeds: N/A
NON-AES-NI speeds: 68.5MB/sec
DDR3 1600 24GB

My FreeNAS Computer
CPU model and frequency: Intel E8500 @ 3.16Ghz
CPU maximum number of cores/threads: 2/2 (Not a HT CPU)
AES-NI speeds: N/A
NON-AES-NI speeds: 80.9MB/sec
DDR3 1066 16GB

So what intrigues me is the fact that an older CPU outperformed the newer i7 CPU. I suspect the HT is the cause for the lower than expected performance so maybe if I turn it off in the BIOS it will improve the performance but I don't like to mess around when it comes to my main computer so I won't be changing just to test this out. I also added the RAM just in case later down the road someone wants to factor that into something since I'd think RAM speed is a variable in the calculation, maybe a minor one though.

Also, I ran the test by placing a bootable image of FreeBSD 9.1 onto an 8GB Flash Drive and booted that since I have since disconnected my CD-RW drive from my FreeNAS box.
 

titan_rw

Guru
Joined
Sep 1, 2012
Messages
586
Just tried two more computers.


My main desktop:

Intel i7-920 (2.67ghz is stock I think), Overclocked to 3.6 ghz.
Cores: 4, Threads: 8.
software aes: 93.5 mb/sec.
hardware aes: N/A.




My backup Freenas box:

Intel core2 quad q9550 @ 2.83 ghz (stock, cpu not oc'd).
Cores: 4, Threads: 4.
software aes: 64 mb/sec.
hardware aes: N/A.



I have used truecrypt on my main desktop, but it's multi-threaded encryption, even from / to a single device. It shows 550 megs / sec aes, but that's using all 8 cores.
 

titan_rw

Guru
Joined
Sep 1, 2012
Messages
586
Another MB I had lying around:

AMD Athlon X2 6000+ (3.0 ghz, stock, not oc).
Cores: 2, Threads: 2.
software aes: 67.5 mb/sec.
hardware aes: N/A.
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
CPU model and frequency: Intel Xeon E3-1220L V2 @ 2.30GHz
CPU maximum number of cores/threads: 2/4
AES-NI speeds: 96MB/sec
NON-AES-NI speeds: 62MB/sec

(consumes 30W with 5 spun down WD RED drives)
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
Starting powerd (which enables Intel Turbo Boost) before running the tests I get these numbers:

AES-NI speeds: 140MB/sec
NON-AES-NI speeds: 86MB/sec

The numbers stay the same even for much longer test runs (i.e. count = 10000+)
 

macer

Cadet
Joined
May 12, 2013
Messages
1
AMD A8-5500 (Bulldozer) @ 3.4GHz (No OC) Cores/Threads 4/4
AES-NI speeds: 110.050759MB/s
NON AES-NI speeds: 72.7835093MB/s

Outputs:
4294967296 bytes transferred in 37.219189 secs (115396585 bytes/sec)
4294967296 bytes transferred in 56.276484 secs (76319041 bytes/sec)
 

DrDoom

Cadet
Joined
May 21, 2013
Messages
1
AMD Phenom II X4 965 @ 3.4 GHz Cores = 4 Threads = 4
32 GB 1333 CL9
NO AES-NI

aes-xts 256 = 66MByte/s

Intel Core i5-2520M @ 2.50GHz Cores = 2 Threads = 4 (Sandy-Bridge)
8 GB 1333 CL9

AES-NI = 122 MByte/s
NO AES-NI = 70 MByte/s

the truecrypt-benchmark on Windows 7 gets 1,3 GByte/s with the AES-NI of the i5. But why is there such a big difference?

my new A8-5600K @ 3,6 GHz Cores = 4 Threads = 4
32 GB 1333 CL9

AES-NI = 126 MByte/s
NO AES-NI = 83 MByte/s
 
Joined
Aug 2, 2013
Messages
1
CPU model and frequency: AMD Dual Core E-350 Processor (1.6 GHz)
CPU maximum number of cores/threads: 2/2
AES-NI speeds: n/a
NON-AES-NI speeds: 30MB/sec
 

mael

Dabbler
Joined
Jun 27, 2013
Messages
20
CPU model and frequency: Intel E3-1230 v3 @ 3.3GHz
CPU maximum number of cores/threads: 4/8
AES-NI speeds: 149720202 bytes/sec, ~142.78MB/sec
NON-AES-NI speeds: 99753510 bytes/sec, ~95.13MB/sec
 

pmb

Cadet
Joined
Nov 5, 2013
Messages
3
CPU model and frequency: AMD Dual Core E-450 Processor (1.65 GHz)
CPU maximum number of cores/threads: 2/2
AES-NI speeds: N/A
NON-AES-NI speeds: 31450393 bytes/sec (30MB/sec)
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
So, this will make the last post on this topic as you basically will choose to use encryption with AES-NI or performance will be horrible. Starting with FreeNAS 9.2.0-release candidate a patch has been applied that has dramatically increased performance of encryption.

My CPU is one of the crappiest Xeons made by Intel with AES-NI support. I now get over 750MB/sec per core(3GB/sec total). Here's a chart comparing 9.1.1 versus 9.2.0 with AESNI:

These benchmarks were done with an 8GB RAMdrive which was then encrypted with geli 128-bit AES-XTS encryption, the followed up with creating a pool.

Lastly, I did gzero tests because I could. They are valuable in the sense that they are theoretical speeds, but the zpool speeds would be more useful.

Note that these speeds are on a per disk and per core. You are limited to whichever is a smaller number. So if you have an 8 thread machine with 5 disks in your pool you will be limited to 5x the below numbers(in particular the zpool speeds), but also potentiall limited by the pool's ability to read and write.

Code:
Test                      9.1.1            9.2.0
 
RAMDRIVE raw 128k write   1.3GB/sec        1.3GB/sec
RAMDRIVE raw 4k write     438MB/sec        438MB/sec
RAMDRIVE raw 128k read    2.1GB/sec        2.1GB/sec
RAMDRIVE raw 4k read      422MB/sec        421MB/sec
Enc drive 128k write      106MB/sec        521MB/sec
Enc drive 32k write       105MB/sec        486MB/sec
Enc drive 4k write        74MB/sec         200MB/sec
Enc drive 128k read       108MB/sec        615MB/sec
Enc drive 32k read        105MB/sec        573MB/sec
Enc drive 4k read         80MB/sec         208MB/sec
zpool 128k write          100MB/sec        416MB/sec
zpool 32k write           88MB/sec         378MB/sec
zpool 4k write            83MB/sec         275MB/sec
zpool 128k read           4.9GB/sec*       5.1GB/sec*
zpool 32k read            4.5GB/sec*       3.8GB/sec*
zpool 4k read             944MB/sec*       969MB/sec*
 
gzero device write 128k   109MB/sec        746MB/sec
gzero device write 32k    105MB/sec        617MB/sec
gzero device write 4k     77MB/sec         243MB/sec
gzero device read 128k    108MB/sec        745MB/sec
gzero device read 32k     107MB/sec        616MB/sec
gzero device read 4k      78MB/sec         242MB/sec

* - These values are zpool reads with an 8GB RAM disk, but the ZFS cache boosted the speed to obviously outrageous speeds because it bypassed the encryption since the system has 20GB of RAM. Good example of how RAM can make a big difference with ZFS.

So in conclusion, the bottom line is if you have an AESNI enabled CPU, your speeds are about to make a MAJOR jump starting with 9.2.0.

In essence, encryption isn't going to significantly affect your performance anymore. For that reason I'm going to close this thread with these numbers provided. If we need to ever provide an update, we can always reopen it.

Thanks to everyone that helped provide benchmarking values for the non AES-NI supported CPUs!

Thread locked!
 
Status
Not open for further replies.
Top