Encryption improvements in 9.2.0

[jyavenard]

I think what's getting cyberjock's underwear all twisty is the fact that the usage of random data is completely unnecessary here. As he points out, encrypting a block of zeros and a block of random data will take the *same* amount of computational time. The same. Whether it's AES-CBC or AES-XTS at work (and 128 or 256 bit AES for that matter), the overhead of the algorithm is always in cycles/byte. Not in cycles/type-of-byte. :) All AESNI does is reduce the number of cycles per byte.

I understand the math behind encryption...

And I normally would have taken this further because of what logic dictates. But curiosity got the best of me, and I ran a little experiment:
In 9.1; writing 100GB of zeros on an unencrypted pool takes the same amount of time as writing random data: good so far

Yet, doing the same on an encrypted pool, I always found that writing zeros is slower on average than writing random data, and that by over 10%.
And I get greatly varying times: over 5 runs, I had a variance of 6200, while writing random the results were always consistent with one another.

I'm at loss why, it makes no sense: maybe it encounters a corner case in the encryption code used.

In 9.2: it makes no difference as it should.
That seems to contradict what you wrote earlier @jkh: in 9.1 encrypting zeros either isn't as quick as other data, or if it's not at the encryption level, there's something else at play.

 

[cyberjock]

You suck! I saw...

FreeNAS-9.2.0-RELEASE-6eccc24-x64 (as tagged on github)

And I about jizzed in my pants!

 

[jkh]

I understand the math behind encryption...

[difference in time it takes to encrypt zeros vs random data is puzzling jyavenard]

I'm at loss why, it makes no sense: maybe it encounters a corner case in the encryption code used.

There are a lot of variables here, not least of which are the AESNI instructions themselves and how well they perform with certain types of data. Yes, in theory the cycles/byte should remain constant for any type of data, but in practice it's also possible that the algorithm (particularly the silicon implementation thereof) can exhibit slightly different performance characteristics with different types of data - remember that the cycles/byte number is just an *average*, it's not a guarantee that you will see *exactly* that number of cycles for absolutely any kind of data.

In any case, the testing methodology is also a bit flawed here - if you really want to do scientific measurements, you need to decouple encryption from I/O in your measurements since you're obviously also taking different code paths in each case. Try a simple encryption test and time that on the same data but on a non-encrypted pool, for example.

 

[Dusan]

Yet, doing the same on an encrypted pool, I always found that writing zeros is slower on average than writing random data, and that by over 10%.
And I get greatly varying times: over 5 runs, I had a variance of 6200, while writing random the results were always consistent with one another.

I must agree with jkh that this seems to point to a problem with the testing method. A properly implemented encryption algorithm must make sure that the encryption time of same sized data is the same (or as close as possible). Any variation opens a side-channel -- an attacker that is able to time such encryption is then able to infer information about the clear text (https://en.wikipedia.org/wiki/Timing_attack). Such a huge variance would mean that the FreeBSD encryption code is horribly broken. I don't find this very plausible, so the other option is that there is some additional factor/influence you did not fully control in your tests.

 

[jyavenard]

I must agree with jkh that this seems to point to a problem with the testing method.

it may not be testing just encryption, but under all tests, the conditions and commands ran are the same: as such, this annihilate any "errors" as the error or bias are the same.
While I haven't proven encryption in 9.1 of zeros were slower, nor have I proven than writing zeros is slower.
What it does show however, is that no matter what it is; the entire cycle (encryption+writing+zfs) is slower in one case than in another.

 

[jyavenard]

Any particular reasons AES-XTS 256 isn't used when encrypting data?

It is for swap partition.
I amended an existing ticket: https://bugs.freenas.org/issues/2393 ; but it would be nice to let the user chose, or up the default. Especially as there's now no speed impact in doing so...

 

[jyavenard]

For info regarding speed:
AES-XTS 256 vs AES-XTS 128:

128:

# geli onetime -s 4096 -l 128 -e aes-xts gzero
[root@freenas-surrey1] /mnt/pool/.zfs/snapshot# dd if=/dev/gzero.eli of=/dev/null bs=1m count=4096
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 3.075693 secs (1396422567 bytes/sec)

256:
# dd if=/dev/gzero.eli of=/dev/null bs=1m count=4096
4096+0 records in
4096+0 records out

4294967296 bytes transferred in 3.434808 secs (1250424267 bytes/sec)

a speed impact of only 11%... And still well above anything that would actually impact actual disk transfer speed.

 

[Dusan]

And still well above anything that would actually impact actual disk transfer speed.

That actually depends on how many drives you have. A typical 5400 RPM drive transfers at ~150MB/s (WD RED). (1250424267 bytes/sec)/(150MB/s) = 8. So, with your CPU it would start to limit you at about 8 drives, probably sooner as ZFS also has its overhead.

 

[jyavenard]

But that's not much dependent on AES 256 vs AES 128.
Either encryption level will give you similar limitation.

this was the point of my post.

Additionally, the number above was for one core only.

My understanding is that with hardware acceleration you automatically get as many core as available in geli

Also, read (or write) are never directly proportional to how many disks you have. Would love to get GB/a for any raid array using mechanical drives !

 

[Dusan]

Additionally, the number above was for one core only.

My understanding is that with hardware acceleration you automatically get as many core as available in geli

As I understand it, it used one core per one provider in 9.1.1 -- you had to have at least as many drives as cores to use your CPU to the max. However, after this change: https://bugs.freenas.org/projects/f...ions/b6e9abdd5936f233014b35a4a1f35f3f15b6e164 (which is also included in 9.2.0) it seems you should get all cores even in your test (single geli provider).
EDIT: you can easily verify this by re-running your gzero test with sysctl kern.geom.eli.threads=1.

 

[cyberjock]

So FreeNAS isn't a 1 core per provider design with geli anymore? I was unaware of that change. I'll have to go test this on my test system. :p

 

[jyavenard]

EDIT: you can easily verify this by re-running your gzero test with sysctl kern.geom.eli.threads=1.

from the documentation:

kern.geom.eli.threads: 0
Specifies how many kernel threads should be used for doing soft-
ware cryptography. Its purpose is to increase performance on SMP
systems. If hardware acceleration is available, only one thread
will be started. If set to 0, CPU-bound thread will be started
for every active CPU.

http://www.freebsd.org/cgi/man.cgi?query=geli&apropos=0&sektion=0&manpath=FreeBSD 9.0-RELEASE&format=html

so the default should be 1, yet when I changed the value, I saw:

kern.geom.eli.threads: 4 -> 1

Edit: never mind that, the man page predates that commit

I'm seeing the same value for testing:

1202515314 bytes/sec

(I have my encrypted zpool resilvering, and I'm seeing 281MB/s via zpool iostat)

I doubt a device driver like gzero would be multi-threaded ; but I'm not familiar with the inner details but changing kern.geom.eli.threads has no effect on the dd test

 

[cyberjock]

So with 9.2.0-RC I just did the following:

Added the sysctl kern.geom.eli.threads=1 and rebooted.

After reboot:

[root@freenas] ~# sysctl -a | grep eli
kern.geom.eli.batch: 0
kern.geom.eli.threads: 1
kern.geom.eli.overwrites: 5
kern.geom.eli.visible_passphrase: 0
kern.geom.eli.tries: 3
kern.geom.eli.debug: 0
kern.geom.eli.version: 6
kern.geom.eli.key_cache_misses: 0
kern.geom.eli.key_cache_hits: 0
kern.geom.eli.key_cache_limit: 8192
[root@freenas] ~# kldload geom_zero
[root@freenas] ~# sysctl kern.geom.zero.clear=0
kern.geom.zero.clear: 1 -> 0
[root@freenas] ~# geli onetime -s 4096 -l 128 -e aes-xts gzero
[root@freenas] ~# dd if=/dev/gzero.eli of=/dev/null bs=1m count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 13.946064 secs (751879527 bytes/sec)
[root@freenas] ~#

My CPU goes to 26% usage(4 thread CPU). With 9.1.1 I couldn't get over 110MB/sec. I changed the sysctl kern.geom.eli.threads=0 but I still get the same speeds. So either 1 of 2 things are happening. gzero is atually our limiting factor with the AES-NI speedups and we can't even see the speed difference between one thread and the CPU limit or we're still 1 thread per provider. I'm not sure which.

And to be honest, I'm not sure if I care.

If we're one thread per provider, 750MB/sec is far more than most people's "care" factor.

If it really is multiple threads and we're not able to see the difference because gzero is "too slow" then the performance is far more than 750MB/sec, so again, I don't care.

Intel claimed that AES-NI was supposed to cause a performance increase of 3-10x over software implementations. I'm at about 14x the software implementation on the same hardware based on my tests from January with FreeBSD 9.2 liveCD(was 56MB/sec with gzero and AESNI disabled).

I'm no encryption guru, but I thought I had read somewhere that AES-XTS can't be parallelized among one provider because if how the encryption works. So it could be that its multithreaded if we chose a different method. But as far as I know AES-XTS is the only viable option for disk encryption.

Edit: And jyavenard pretty much confirmed my results just minutes before me. So more knowledge is necessary. Time to hit the books!

 

[cyberjock]

But that's not much dependent on AES 256 vs AES 128.
Either encryption level will give you similar limitation.

this was the point of my post.

Additionally, the number above was for one core only.

My understanding is that with hardware acceleration you automatically get as many core as available in geli

Also, read (or write) are never directly proportional to how many disks you have. Would love to get GB/a for any raid array using mechanical drives !

I read somewhere a couple of years ago that AES 256 really isn't that much more secure than AES 128. There's ways to factor down and do other things to get AES 256 down to only 2 bits more difficulty(and potentially less because AES 256 was conceived as being AES 128 just with more bits but didn't work out quite as planned hence the minimal increase in difficulty). Some people surmise that AES-256 may be less secure than AES 128 because of how it is implemented and that 128 should be preferred because even if you could do some factoring to help with a brute force attack the attacker couldn't reasonable exclude enough combinations to make cracking it in a lifetime feasible.

Of course, there's LOTS of right and wrong info on encryption. Lots of people questions how secure AES is since the head of the NSA had a hand in AES being ratified(and what better idea to help yourself have an "in" if you know how to crack it but it appears uncrackable and you can get it approved for "secure encryption").

I was talking to someone yesterday about this whole encryption thing and someone basically accused the iX developers of being complete buffoons for implementing geli so poorly that you could only get 100MB/sec with AESNI enabled with gzero. Ok, so it might not have been implemented in the best way, but I don't think that's a fair comparison as iX isn't in the business of selling security software. It's quite possible that iX has completely failed to implement the encryption properly and its actually quite insecure. What I'd love to see is some of the geli programmers do some auditing of the security model FreeNAS uses for encryption. They might have good ideas of things that might be wrong or could use improvement.

 

[cyberjock]

Here's the info on 128 versus 256 I was looking for if you like to read details:

http://crypto.stackexchange.com/questions/5118/is-aes-256-weaker-than-192-and-128-bit-versions

What I took from that whole thing is that 256 is better than 128 if quantum computing ever becomes a problem for brute forcing AES. It's technically possible that 256 might be weaker than AES128 on standard computers, someday. But, even if 256 is less secure, it's still more than sufficiently secure to keep your data safe for all standard computers.

So, does it matter? I'm thinking "probably" not in the big picture. But, at the same time, there's no reason NOT to go to AES-256.

In short....

There's no reason to avoid AES-128 but no reason not to go to AES-256.

 

[Dusan]

I was talking to someone yesterday about this whole encryption thing and someone basically accused the iX developers of being complete buffoons for implementing geli so poorly that you could only get 100MB/sec with AESNI enabled with gzero. Ok, so it might not have been implemented in the best way, but I don't think that's a fair comparison as iX isn't in the business of selling security software. It's quite possible that iX has completely failed to implement the encryption properly and its actually quite insecure. What I'd love to see is some of the geli programmers do some auditing of the security model FreeNAS uses for encryption. They might have good ideas of things that might be wrong or could use improvement.

Does that "somebody" actually understand geli and FreeNAS? All the FreeNAS "geli implementation" does is run "geli init -s 4096". If you installed FreeBSD 9.2 and followed the geli manual (http://www.freebsd.org/doc/handbook/disks-encrypting.html) you would get exactly the same security and suprise! lower performance. iX developers are actually cherry picking patches from "future" FreeBSD releases to improve the performance. For example, the 10x speedup patch is not in FreeBSD 9.2 and the last time I checked it was not even in the stable/9 codeline (it was only in stable/10). Also, the bug tracker shows that iX is in contact with the authors of the code: https://bugs.freenas.org/issues/3539#note-7 (jmg is author of the speedup patches). So, I don't think iX is doing random changes/implementations without understanding them. They actually work hard to get us the best custom FreeBSD build in areas where it matters (ZFS, geli, ...).

 

[cyberjock]

Does that "somebody" actually understand geli and FreeNAS? All the FreeNAS "geli implementation" does is run "geli init -s 4096". If you installed FreeBSD 9.2 and followed the geli manual (http://www.freebsd.org/doc/handbook/disks-encrypting.html) you would get exactly the same security and suprise! lower performance. iX developers are actually cherry picking patches from "future" FreeBSD releases to improve the performance. For example, the 10x speedup patch is not in FreeBSD 9.2 and the last time I checked it was not even in the stable/9 codeline (it was only in stable/10). Also, the bug tracker shows that iX is in contact with the authors of the code: https://bugs.freenas.org/issues/3539#note-7 (jmg is author of the speedup patches). So, I don't think iX is doing random changes/implementations without understanding them. They actually work hard to get us the best custom FreeBSD build in areas where it matters (ZFS, geli, ...).

Honestly, I'm not sure how much he knows about the FreeNAS implementation. He knows its basically geli devices being presented to ZFS but that's probably about it. I am definitely appreciative of what iX is doing and I did realize it was code that wasn't in FreeBSD Stable. Honestly, I find it amazing that FreeNAS is as stable as it is despite the customization of the build that we have. Before iXsystems reinvented FreeNAS I dismissed the old project because I thought that trying to run a database of settings in a GUI that is nothing more but an interpreter for command line stuff is dangerous and not something I'd ever trust my data with. Hey, I haven't lost my data yet and I don't think I've seen anyone lose their data that was found to be a direct result of the software. Sure, people have used bad/inappropriate hardware with FreeNAS and people have used the software in ways it wasn't designed. But I consider those to be user error and not FreeNAS' fault.

Personally, I didn't have much of a problem with the performance as it was in the prior releases. I think WAY too many people fret over benchmarks and numbers and don't realize that the 1Gb LAN port you have is going to limit you to 120MB/sec or so and that's it. Do not pass go and do not collection $200. The main reason I went with encryption was so I could validate for others that your data isn't going to be eaten(we've had a few nasty bugs, but I haven't had to do a restore from backup yet nor do I know anyone that has).

Overall, I'm a bit excited to put 9.2.0 on my server, but always cautious as you never know what bug might bite. But I have no reason to think that the iXsystems developers don't have their head in the game.

What I do find interesting is that FreeBSD is just getting this encryption thing "right" with this new speed boost. But why/how did FreeBSD get it wrong for so long? I mean, Truecrypt ha been blisteringly fast for me when I first tested it in 2009 for a friend. Do new features normally take 3+ years to mature? Just asking questions because I don't know.

 

[jyavenard]

Does that "somebody" actually understand geli and FreeNAS? All the FreeNAS "geli implementation" does is run "geli init -s 4096". If you installed FreeBSD 9.2 and followed the geli manual (http://www.freebsd.org/doc/handbook/disks-encrypting.html) you would get exactly the same security and suprise! lower performance. iX developers are actually cherry picking patches from "future" FreeBSD releases to improve the performance.

exactly... there's nothing fancy about FreeNAS use of ZFS over geli; it's pretty much the common approach in FreeBSD land.

there are concerns to have in using zfs over geli, after all zfs has no direct access to the hardware disk anymore and access it via another layer: that zfs needs access to the hardware directly has been the #1 argument presented here over and over when it came to choosing a disk controller card: it's pretty difficult to understand why in one case it would be okay, but not in another

 

[cyberjock]

there are concerns to have in using zfs over geli, after all zfs has no direct access to the hardware disk anymore and access it via another layer: that zfs needs access to the hardware directly has been the #1 argument presented here over and over when it came to choosing a disk controller card: it's pretty difficult to understand why in one case it would be okay, but not in another

I had that discussion with a developer about a year ago when encryption hit public. The answer was "if ZFS starts having problems you've got SMART reporting and other mechanisms in FreeNAS that will still detect and report the errors".

I can vouch that in the event a disk starts having problems you do get reports and emails up the wazoo. I got a whole bunch within about 10 minutes when the disk started throwing errors. It got so bad I offlined the disk in less than an hour so I wouldn't have to delete 1000 emails.