Edit: Please go down and read the last post on this page. Starting with 9.2.0 a fix was made that made AES-NI so fast that pool performance will not be significantly affected by encryption unless you plan to run 10Gb network hardware and have a pool that should be doing multi-GB/sec performance. Basically if you have an AES-NI compatible CPU you won't notice the performance impact of encryption. If you don't have an AES-NI compatible CPU performance will be significantly impacted(aka don't do it).
This thread is to discuss and compare various CPUs and how they perform with and without AES-NI CPUs. This discussion is intended to provide users with a comparison of potential performance numbers if encryption is used and what CPUs can provide what performance since 8.3.1 has been released and some users are curious as to what kind of performance penalty can be expected from using encryption.
You can check if your CPU supports AES-NI by running the command # dmesg | grep aes from FreeNAS 8.3.1(If you run this command from anything before 8.3.1 you will see the features line if AES-NI is supported but the aesni0 device will not exist). If you do not get an output that looks like what is below your CPU doesn't support AES-NI.
The following instructions were performed using a FreeBSD 9.1 64-bit CD in "Live CD" mode. There shouldn't be a significant difference in performance between 32-bit and 64-bit for encryption. However, zpools aren't recommended on a 32-bit OS because of the 4GB RAM limitations(remember that the manual recommends no less than 6GB of RAM to use ZFS). These commands do not work in FreeNAS and if you try to run them you will crash the system.
Instructions for testing:
1. Boot the FreeBSD 9.1 x64 CD.
2. Choose "Live CD" when asked if you want to install or use the Live CD.
3. Enter the username "root" when the login prompt is displayed. No password is required.
Perform the following steps to test a NON-AES-NI compatible CPU or to test an AES-NI compatible CPU with AES-NI disabled:
(The bold will be the commands and the non-bold should be roughly the expected output)
Perform the following steps to test an AES-NI compatible CPU with AES-NI enabled:
To keep things organized if you provide your values please include the following in red so that future posters can sift through and separate conversation from actual raw data:
CPU model and frequency: Intel Xeon E5606 @ 2.13Ghz (don't forget to correct the speed if you overclock/underclock)
CPU maximum number of cores/threads: 4/4 (included for HT/non-HT CPUs)
AES-NI speeds: 104MB/sec (If not supported please include this line with 'N/A')
NON-AES-NI speeds: 56MB/sec
So what do these values tell me?
First, they tell me that I can encrypt about twice the data per second with AES-NI versus non-AES-NI. AES-NI is enabled by default if your CPU supports AES-NI.
Second, if I had a zpool with 4 or more drives(since my CPU supports only 4 threads) and there was no service or zpool overhead and the hard drives/zpool weren't the bottleneck then I could expect a maximum of 416MB/sec(104MB/sec x 4). Unfortunately, depending on your services, total system RAM, zpool type and size, etc these values will only go down. The 416MB/sec is theoretical and should not be taken as values you could actually attain with your system and typical sharing services enabled and in use. Additionally, any zpool scrub in progress on encrypted hard drives would seriously impact the performance of the file sharing services.
My zpool currently is a RAIDZ3 of 18x2TB drives. DD tests bench my zpool at 900-1000MB/sec. I'd expect that if I were to use encryption on my zpool I'd get about 200-300MB/sec maximum. Since I use two 1Gb LAN ports I would certainly be hurting my LAN speeds if both were heavily used. Also noteworthy is that my scrubs would take significantly longer to perform.
If you have a system that you'd like to provide values for, please include them in this thread for comparison. If we get a lot of data on various CPUs(especially the commonly requested CPUs such as Atoms and low powered AMDs) I'll use the second post to provide a list of all of the data for easy access.
Thanks,
-Cyberjock
Notes:
1. The benchmarks provided only provide the maximum theoretical performance of encryption. In a production environment you can expect different numbers because of the increased overhead of the file sharing services used, ZFS overhead, and added latency from the disks themselves.
2. Geli supports 1 thread per "device". A device would typically mean a hard disk. The following benchmark values will only give you the performance of a single CPU core/thread. If you want to max out your potential performance you should have the same number of hard drives in the zpool or UFS raid as you do cores. For example, if you only have 3 hard drives but your CPU supports up to 8 simultaneous threads you will only have 3 threads available for encryption. This does have the advantage of providing the other 5 "cores/threads" for all of the other services the system uses. If you chose to use a hardware RAID(which isn't recommended for ZFS), then the entire RAID would count as one device(read: very poor performance). If you use RAIDZ2 you must include the total number of hard drives in the zpool and not the number of drives excluding redundancy.
3. AES-NI is a relatively new technology. Only certain models of Intel CPUs released in Q1'11 or newer and some AMD CPUs from Q3'11 and newer support AES-NI. A list of the Intel CPUs that support AES-NI can be found here.
Found this benchmark on the internet: http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071608.html
CPU model and frequency: Intel Atom D525 @ 1.8Ghz
CPU maximum number of cores/threads: 2/4
AES-NI speeds: N/A
NON-AES-NI speeds: 14.7MB/sec
Here's another one of my machines:
CPU model and frequency: Intel Atom D2500 @ 1.86Ghz
CPU maximum number of cores/threads: 2/2
AES-NI speeds: N/A
NON-AES-NI speeds: 16.2MB/sec
This thread is to discuss and compare various CPUs and how they perform with and without AES-NI CPUs. This discussion is intended to provide users with a comparison of potential performance numbers if encryption is used and what CPUs can provide what performance since 8.3.1 has been released and some users are curious as to what kind of performance penalty can be expected from using encryption.
You can check if your CPU supports AES-NI by running the command # dmesg | grep aes from FreeNAS 8.3.1(If you run this command from anything before 8.3.1 you will see the features line if AES-NI is supported but the aesni0 device will not exist). If you do not get an output that looks like what is below your CPU doesn't support AES-NI.
Code:
# dmesg | grep aes Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI> aesni0: <AES-CBC,AES-XTS> on motherboard
Instructions for testing:
1. Boot the FreeBSD 9.1 x64 CD.
2. Choose "Live CD" when asked if you want to install or use the Live CD.
3. Enter the username "root" when the login prompt is displayed. No password is required.
Perform the following steps to test a NON-AES-NI compatible CPU or to test an AES-NI compatible CPU with AES-NI disabled:
(The bold will be the commands and the non-bold should be roughly the expected output)
Code:
# kldload geom_eli cryptosoft0: <software crypto> on motherboard # kldload geom_zero # sysctl kern.geom.zero.clear=0 kern.geom.zero.clear: 1 -> 0 # geli onetime -s 4096 -l 256 -e aes-xts gzero GEOM_ELI: Device gzero.eli created. GEOM_ELI: Encryption: AES-XTS 256 GEOM_ELI: Crypto: software # dd if=/dev/gzero.eli of=/dev/null bs=1m count=4096 4096+0 records in 4096+0 records out 4294967296 bytes transferred in 76.077181 secs (56455395 bytes/sec) # geli kill gzero GEOM_ELI: gzero has been killed. GEOM_ELI: Device gzero.eli destroyed. # kldunload geom_zero # kldunload geom_eli
Perform the following steps to test an AES-NI compatible CPU with AES-NI enabled:
Code:
# kldload aesni cryptosoft0: <software crypto> on motherboard aesni0: <AES-CBC,AES-XTS> on motherboard # kldload geom_zero # sysctl kern.geom.zero.clear=0 kern.geom.zero.clear: 1 -> 0 # geli onetime -s 4096 -l 256 -e aes-xts gzero GEOM_ELI: Device gzero.eli created. GEOM_ELI: Encryption: AES-XTS 256 GEOM_ELI: Crypto: hardware # dd if=/dev/gzero.eli of=/dev/null bs=1m count=4096 4096+0 records in 4096+0 records out 4294967296 bytes transferred in 41.142665 secs (104392054 bytes/sec) # geli kill gzero GEOM_ELI: gzero has been killed. GEOM_ELI: Device gzero.eli destroyed. # kldunload geom_zero # kldunload geom_eli # kldunload aesni
To keep things organized if you provide your values please include the following in red so that future posters can sift through and separate conversation from actual raw data:
CPU model and frequency: Intel Xeon E5606 @ 2.13Ghz (don't forget to correct the speed if you overclock/underclock)
CPU maximum number of cores/threads: 4/4 (included for HT/non-HT CPUs)
AES-NI speeds: 104MB/sec (If not supported please include this line with 'N/A')
NON-AES-NI speeds: 56MB/sec
So what do these values tell me?
First, they tell me that I can encrypt about twice the data per second with AES-NI versus non-AES-NI. AES-NI is enabled by default if your CPU supports AES-NI.
Second, if I had a zpool with 4 or more drives(since my CPU supports only 4 threads) and there was no service or zpool overhead and the hard drives/zpool weren't the bottleneck then I could expect a maximum of 416MB/sec(104MB/sec x 4). Unfortunately, depending on your services, total system RAM, zpool type and size, etc these values will only go down. The 416MB/sec is theoretical and should not be taken as values you could actually attain with your system and typical sharing services enabled and in use. Additionally, any zpool scrub in progress on encrypted hard drives would seriously impact the performance of the file sharing services.
My zpool currently is a RAIDZ3 of 18x2TB drives. DD tests bench my zpool at 900-1000MB/sec. I'd expect that if I were to use encryption on my zpool I'd get about 200-300MB/sec maximum. Since I use two 1Gb LAN ports I would certainly be hurting my LAN speeds if both were heavily used. Also noteworthy is that my scrubs would take significantly longer to perform.
If you have a system that you'd like to provide values for, please include them in this thread for comparison. If we get a lot of data on various CPUs(especially the commonly requested CPUs such as Atoms and low powered AMDs) I'll use the second post to provide a list of all of the data for easy access.
Thanks,
-Cyberjock
Notes:
1. The benchmarks provided only provide the maximum theoretical performance of encryption. In a production environment you can expect different numbers because of the increased overhead of the file sharing services used, ZFS overhead, and added latency from the disks themselves.
2. Geli supports 1 thread per "device". A device would typically mean a hard disk. The following benchmark values will only give you the performance of a single CPU core/thread. If you want to max out your potential performance you should have the same number of hard drives in the zpool or UFS raid as you do cores. For example, if you only have 3 hard drives but your CPU supports up to 8 simultaneous threads you will only have 3 threads available for encryption. This does have the advantage of providing the other 5 "cores/threads" for all of the other services the system uses. If you chose to use a hardware RAID(which isn't recommended for ZFS), then the entire RAID would count as one device(read: very poor performance). If you use RAIDZ2 you must include the total number of hard drives in the zpool and not the number of drives excluding redundancy.
3. AES-NI is a relatively new technology. Only certain models of Intel CPUs released in Q1'11 or newer and some AMD CPUs from Q3'11 and newer support AES-NI. A list of the Intel CPUs that support AES-NI can be found here.
Found this benchmark on the internet: http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071608.html
CPU model and frequency: Intel Atom D525 @ 1.8Ghz
CPU maximum number of cores/threads: 2/4
AES-NI speeds: N/A
NON-AES-NI speeds: 14.7MB/sec
Here's another one of my machines:
CPU model and frequency: Intel Atom D2500 @ 1.86Ghz
CPU maximum number of cores/threads: 2/2
AES-NI speeds: N/A
NON-AES-NI speeds: 16.2MB/sec
