Encryption improvements in 9.2.0

[jpaetzel]

We've merged in a number of changes from FreeBSD HEAD that provide a drastic speedup to GELI, which is the subsystem underlying the ZFS volume encryption in FreeNAS.

If you have a CPU with the AESNI feature expect to see native disk speed performance for volumes with as many as a dozen drives in them.

That's right, there should be virtually no impact to performance when volume encryption is enabled.

 

[jyavenard]

This is an awesome news...

A tad too late for me, I would have gone a different path when setting my pool.

What would be the easiest way to enable encryption on a pool and keep the content?

could zpool export/import be used so you keep all the structure and content of an existing pool and import it within an encrypted pool ?

never done encrypted pool before...

 

[cyberjock]

This is an awesome news...

A tad too late for me, I would have gone a different path when setting my pool.

What would be the easiest way to enable encryption on a pool and keep the content?

could zpool export/import be used so you keep all the structure and content of an existing pool and import it within an encrypted pool ?

never done encrypted pool before...

Pools can only be encrypted when they are created at present. So you'd have to destroy and recreate your pool. :(

 

[jyavenard]

I understand that (unfortunately).
Recreating all zfs dataset is a bit of a pain; I would also lose all the snapshots ever made if I was just to copy the files across.

I was hoping I could do something like: zpool export (from non-encrypted pool) --> zpool import (and import on an encrypted pool).

That way all content of the pool/zfs is preserved.

Probably won't work, as zfs import/export just send the pool content as-is.. But one could dream :)

 

[cyberjock]

Well, if you have the ability to destroy and recreate your pool and your concern is preserving the contents read up on ZFS replication. Problem solved!

 

[Dusan]

Probably won't work, as zfs import/export just send the pool content as-is.. But one could dream :)

That's not how zfs export/import works. Those commands do not "send the pool" anywhere. However, there are commands that do exactly that: zfs send/receive. The FreeNAS replication system is using this mechanism. You can do a snapshot and then zfs send from the old pool and zfs receive the stream to the new pool. The -R option is quite useful here:
[PANEL]-R Generate a replication stream package, which will replicate the specified filesystem, and all descendent file systems, up to the named snapshot. When received, all properties, snapshots, descendent file systems, and clones are preserved.[/PANEL]
Btw.: There is a hacky way to encrypt a pool in place. You would need to basically reverse this: How to remove encryption from a ZFS volume (while keeping the data). However, it requires good knowledge of FreeNAS internals and I'd not recommend it anyway as you are deliberately degrading your pool during the process.

 

[jyavenard]

That's not how zfs export/import works. Those commands do not "send the pool" anywhere. However, there are commands that do exactly that: zfs send/receive. The FreeNAS replication system is using this mechanism. You can do a snapshot and then zfs send from the old pool and zfs receive the stream to the new pool. The -R option is quite useful here:

I've been using zfs export / import since zfs was first introduced in FreeBSD in order to create a mirror of our data server... Was doing so over ssh, but here I would have simply put all disks in the same chassis and simply pipe one into the other...

But that would still have taken an awful long time; my new nas hasn't entered production yet; I've only migrated all the data of various multiple machines into one and I have a cron over each machine updating that data.. but there's still over 8TB of data transferred. Transferring one into the other, and back again (so both machines use encrypted pool) is going to take me days.

Btw.: There is a hacky way to encrypt a pool in place. You would need to basically reverse this: How to remove encryption from a ZFS volume (while keeping the data). However, it requires good knowledge of FreeNAS internals and I'd not recommend it anyway as you are deliberately degrading your pool during the process.

but now that sounds like a great thing to try... I'm not concerned about losing anything (not yet)

 

[Dusan]

but now that sounds like a great thing to try... I'm not concerned about losing anything (not yet)

Agree, it could be a fun way to learn more about FreeNAS internals. Start by looking at function __encrypt_device in notifier.py. Also get familiar with the storage_encrypteddisk and storage_volume tables in the config DB (gui/storage/models.py).

 

[jyavenard]

Mostly (as I mentioned before, but that part seems to have been skipped) it sounds like it's going to take far less time than transferring all the content across twice.

Will see. Also interested in benchmarking the pool while encrypted. Very curious on how that go.

 

[cyberjock]

I will tell you that encryption REALLY hurts pool performance right now. Of course, being 250MB/sec instead of 500MB/sec doesn't matter to me since I'm using only Gb LAN anyway.

All that encryption really means is that scrubs take MUCH longer than you'd expect.

 

[jyavenard]

Is that with the latest changes and AES improvements? What happened to the "native" disk performance?

My last scrubs took 36 hours to complete, so I'm not sure adding a few hours to the work will make a much of a difference :)

 

[cyberjock]

For me the scrub time is about double what I would have expected for an unencrypted pool in the same situation.

I don't run the test builds on my FreeNAS machine so I can't vouch for how much the "native speed" patch will change the speed. Database configurations have ruined the config file for users when upgrading in the past, so I never do anything except RELEASE versions on my production system. When 9.2.0 hits RELEASE I'll definitely be providing numbers within a few days. :)

 

[jyavenard]

We've merged in a number of changes from FreeBSD HEAD that provide a drastic speedup to GELI, which is the subsystem underlying the ZFS volume encryption in FreeNAS.

If you have a CPU with the AESNI feature expect to see native disk speed performance for volumes with as many as a dozen drives in them.

That's right, there should be virtually no impact to performance when volume encryption is enabled.

which commit introduce that feature ??

I'd like to run a comparison with before and after that change.

 

[cyberjock]

I'm not sure, but I know the release candidate(due VERY soon) has it in. I know it's been added in the last 3 days, so no public build I'm aware of has the update... yet.

 

[Dusan]

which commit introduce that feature ??

This is the relevant bug: https://bugs.freenas.org/issues/3539
There are several commits and all are in TrueOS. This is the main patch: https://github.com/trueos/trueos/commit/657e6d25b219893f637ce6c65e73e7ab5e06936d
This I believe is that last one: https://github.com/trueos/trueos/commit/debc19b0628596a2f7d2befc89ccf4e509fd6707

 

[cyberjock]

I do have a spare system that has AES-NI support. It's about the crappiest 1366 that supports AES-NI, so that would be a good CPU to test(its an e5606). Maybe I'll bust out the release candidate and have a go at it for fun when its released. I just need to figure out how to do a good test with a 14GB RAMdrive or something. I'll have to bust out my CLI-fu and figure it out again.

 

[jyavenard]

hmmm...

something weird here:
pool is unencrypted
pool2 is encrypted.

both pool are identical otherwise: 6 RAIDZ2 WD RED 4TB


dd if=/dev/zero of=/mnt/pool/testing/test.big bs=1m count=10000: 19.251569s (constant runs after runs)
dd if=/dev/zero of=/mnt/pool2/testing/test.big bs=1m count=10000
try #1: 27.719961 secs
try #2: 37.758803
try #3: 25.820013 secs
try #4: 39.633492

system is 99.7% idle

I wrote a little utility that write random (almost perfect, it can't be compressed) and work in a similar fashion to dd.
for comparison
./dd2 -b 1048576 -n 10000 -o /dev/null : wrote 10485760000 bytes in 4.254453s: 2464655104.00 bytes/sec


./dd2 -b 1048576 -n 10000 -o /mnt/pool/testing/test.big : 20.130564s (constant runs after runs)
./dd2 -b 1048576 -n 10000 -o /mnt/pool2/testing/test.big :
try #1: 23.434783s
try #2: 28.374720s
try #3: 26.818596s
try #4: 25.124928s
so it fluctuates quite a bit, not as much as writing zeros though... how crazy is that?!

going to copy the entire pool to another non-encrypted pool... post the result tomorrow

 

[cyberjock]

So I'm confused.

I understand your first test(normal test). I understand your group that gives the results between 25 and 39 seconds.

What is the -n 10000 do? What is this dd2? Is this a script or an executable? What is supposed to make dd2 better/different than dd for testing?

But where does the random come in? As I discussed before using random devices is pointless(its not even a good test if you are using compression) because there's no fast way to generate large amounts of data for low CPU usage. As soon as you start causing CPU usage to increase that isn't directly related to the testing criteria you invalidate the test. If you really need to test a compressed pool you need to create a dataset that is uncompressed and use /dev/zero to go to the dataset. Read this sticky: http://forums.freenas.org/threads/notes-on-performance-benchmarks-and-cache.981/ .

Not sure what your configuration is, but if you are running FreeNAS under ESXi and have more vCPUs allocated than you have physical cores you can expect weird performance results. I know this has confused quite a few people.

I just did a dd if=/dev/zero of=/mnt/encrypted/testfile bs=1m count=100000 and I got a consistent value within 2 seconds of each other over 5 tests(with ESXi but no other VMs running).

 

[jyavenard]

so the result of:

time zfs send -Rv pool@clone1 | zfs receive -F -d pool2

unencrypted -> encrypted

real 571m43.088s
user 0m0.784s
sys 104m46.892s

unencrypted -> unencrypted

real 482m20.667s
user 0m0.713s
sys 101m34.148s

so a 15.5% penalty...

 

[jyavenard]

So I'm confused.

I understand your first test(normal test). I understand your group that gives the results between 25 and 39 seconds.

all the answers to your questions are in my previous post...

What is the -n 10000 do? What is this dd2? Is this a script or an executable? What is supposed to make dd2 better/different than dd for testing?

as written: "I wrote a little utility that write random (almost perfect, it can't be compressed) and work in a similar fashion to dd."

I changed the name of arguments
dd / dd2
bs -> b
count -> n
if -> -i
of -> -o

The difference is that dd2 can generate incompressible random data at a very fast rate (2.4GB/s on this machine, or 6.03GB/s using the SSE accelerated code, but it's less random: but I'm fairly certain it doesn't matter as far as zfs realtime compression goes)

which is why I provided as a reference:

./dd2 -b 1048576 -n 10000 -o /dev/null : wrote 10485760000 bytes in 4.254453s: 2464655104.00 bytes/sec

which shows the speed at which random numbers are generated: 2464655104.00 bytes/sec -> 2.46GB/s (or 2.29GiB/s as per IETF) and how long it took: 4.2s to generate 1TB of data. So 4.2s is to be removed from the results here to get the purely disk related data.

But where does the random come in? As I discussed before using random devices is pointless(its not even a good test if you are using compression) because there's no fast way to generate large amounts of data for low CPU usage. As soon as you start causing CPU usage to increase that isn't directly related to the testing criteria you invalidate the test. If you really need to test a compressed pool you need to create a dataset that is uncompressed and use /dev/zero to go to the dataset. Read this sticky: http://forums.freenas.org/threads/notes-on-performance-benchmarks-and-cache.981/ .

No, what it means is that you can't use /dev/random (and I *never* said you could). Which I'm *not* using. And yes you can generate large amount of random data with relatively low-cpu usage, faster than any mechanical drive could sustain. The quality of the randomness isn't as good as /dev/random obviously, but it is sufficient enough as far as speed testing goes and ensuring the data is incompressible (which is all we care about).

2.4GB/s is more than fast enough to test the performance here. The SSE2-accelerated random generated can reach 6GB/s on that machine. Testing very fast SSDs in stripe could be an issue: but that's not the case here...

The aim was that you wouldn't have to worry about compression or caching... It's not going to help, quite the opposite.

If you want the source code of dd2, let me know...

Not sure what your configuration is, but if you are running FreeNAS under ESXi and have more vCPUs allocated than you have physical cores you can expect weird performance results. I know this has confused quite a few people.

I'm not using ESXi; plain FreeNAS. System is a E3-1220V3, on a X10SL7-F motherboard, 32GB of RAM. 8 disks are on the LSI adapters, 4 are on the intel (2 SATA3 and 2 SATA2)

I just did a dd if=/dev/zero of=/mnt/encrypted/testfile bs=1m count=100000 and I got a consistent value within 2 seconds of each other over 5 tests(with ESXi but no other VMs running).

And that was precisely the point of my post: with this new encryption code: results fluctuate greatly...

I'm going to try the beta which I believe doesn't have the new code... and see if that great speed fluctuation is still there[/quote][/quote]