Encryption improvements in 9.2.0

[jkh]

I've been using zfs export / import since zfs was first introduced in FreeBSD in order to create a mirror of our data server... Was doing so over ssh, but here I would have simply put all disks in the same chassis and simply pipe one into the other...

No you haven't. You've been using zfs send / receive to do this. Entirely different verbs.

 

[jyavenard]

No you haven't. You've been using zfs send / receive to do this. Entirely different verbs. :)

yes... I certainly did mixed those!

Could you confirm that the beta release doesn't have the new encryption improvement ?

(ps: what happened to the nighlies?)

 

[jkh]

I don't believe that BETA had those improvements, no, but you could be absolutely sure with 9.1.1. :)

The 9.2.0-RC should be out later this evening (really, honestly this time for SURE!) - I retired the nightlies because they were all pre-RC. Once the RC is out, I'll restart them leading up to -RELEASE.

 

[jyavenard]

I don't believe that BETA had those improvements, no, but you could be absolutely sure with 9.1.1. :)

but how will 9.1.1 work with a system that has been upgraded to 9.2 ?

 

[jkh]

but how will 9.1.1 work with a system that has been upgraded to 9.2 ?

Well, obviously you'd have to downgrade that system again first. :)

 

[jyavenard]

Well, obviously you'd have to downgrade that system again first. :)

don't want to drag this thread off-topic with those questions, but is there an "official" method on how to downgrade a system?
Or is flashing 9.1.1 on the USB a sufficient method of "downgrade" ? :)

I don't recall if I did a backup of the config prior upgrading to 9.2 :(
Edit: oh, I did do a backup of the config...

 

[cyberjock]

No, there's no way to officially downgrade unfortunately. I'm planning to do some RAM disk testing on my test box using 9.1.1 and the 9.2.0-RC as soon as jkh gets off his butt and makes us an actual working RC. ;) (this is an inside joke because twice he's compiled the RC and something went wrong..hopefully tomorrow)

If you are comfortable with posting the dd2 source I'd like to take a look(and put it to the test on my test system when I do 9.1.1 and 9.2.0). ;)

 

[jyavenard]

If you are comfortable with posting the dd2 source I'd like to take a look(and put it to the test on my test system when I do 9.1.1 and 9.2.0). ;)

there:
http://jyavariousideas.blogspot.com.au/2013/12/how-to-write-random-data-to-disk-fast.html

Currently using 9.1.1; and speed varies greatly too... Having said that, I generate a file that is 9GB in size, and I have 32GB of RAM...

 

[cyberjock]

Not to dismiss your dd2 tool, but especially for benchmarking and troubleshooting I really try to stick with tried and true tools. I'm really thinking the variance is going to be from your dd2 tool. But curiosity is getting the best of me with this dd2 you have.

I really don't understand your comment on the blog:

To me, writing zero serves little purpose, it is highly compressible to start with. Aside writing data linearly, is hardly a method to benchmark your disk speed!

If you aren't using compression it doesn't matter if its all zeros or all ones or random. What IS important is that CPU usage isn't significantly affected and that the source device can dump out large quantities of data easily(something /dev/zero DEFINITELY does). That's why /dev/zero has been the source of choice for so long. My machine can do almost 32GB/sec, more than an order of magnitude faster than your dd2. So clearly a better source for source data.

And I really don't get why you think that writing with dd2 versus dd with /dev/zero is different regarding your comment "writing data linearly". ZFS will still allocate the data exactly the same...based on how ZFS uses free space via the metaslabs.

They both sound like gross concept errors to me. I explained that all to you before, and you never really explained how any of my explanation is wrong. But whatever.

 

[jyavenard]

Not to dismiss your dd2 tool, but especially for benchmarking and troubleshooting I really try to stick with tried and true tools. I'm really thinking the variance is going to be from your dd2 tool. But curiosity is getting the best of me with this dd2 you have.

you make too many assumptions. I'm seeing exactly the same variations using dd.

dd2 write on disks the same way dd does... using count blocks.
so if you read with bs=4k; it reads 4k, and write 4k at once...

how this will be handled by the system is... up to the system.

If you aren't using compression it doesn't matter if its all zeros or all ones or random. What IS important is that CPU usage isn't significantly affected and that the source device can dump out large quantities of data easily(something /dev/zero DEFINITELY does). That's why /dev/zero has been the source of choice for so long. My machine can do almost 32GB/sec, more than an order of magnitude faster than your dd2. So clearly a better source for source data.

And I really don't get why you think that writing with dd2 versus dd with /dev/zero is different regarding your comment "writing data linearly". ZFS will still allocate the data exactly the same...based on how ZFS uses free space via the metaslabs.

uh??? there's nothing more linear than writes using dd !
linear vs random read or writes... I thought those were pretty accepted definitions, so didn't see the need to expand on those

They both sound like gross concept errors to me. I explained that all to you before, and you never really explained how any of my explanation is wrong. But whatever.

I didn't say they were wrong... I said that you make too many assumptions. Both on what you know, and what you think the other person knows.
as far how dd writes data, including how ZFS cache it that makes the speed returned grossly over-estimated, I went over that already... Don't want to expand on it again and turn it into a flame war

 

[jkh]

Every time you two start talking to one another, I get nervous. :)

 

[cyberjock]

uh??? there's nothing more linear than writes using dd !
linear vs random read or writes... I thought those were pretty accepted definitions, so didn't see the need to expand on those.

And somehow dd2 is supposed to be better? dd is designed to test maximum possible throughput. If you want I/O testing you use other tools. Didn't dad ever teach you to use the right tool for the right job?

I didn't say they were wrong... I said that you make too many assumptions. Both on what you know, and what you think the other person knows.
as far how dd writes data, including how ZFS cache it that makes the speed returned grossly over-estimated, I went over that already... Don't want to expand on it again and turn it into a flame war

Pardon me for listening to everyone else on the planet that uses dd for diagnostic testing and analysis.. with ZFS. We don't have stickies from 2011 that have worked for us without exception since then. Clearly they need to be schooled by the mighty jyavenard.

Thank god you are hear to save us from eternal damnation.

 

[jkh]

... aaand there it is. The reason for my nervousness, now become manifest! :)

I think it's probably best if we stay away from benchmark discussions in the forums unless those doing the benchmarks are willing to use more complex benchmarks like iozone, bonnie or, for actual network filesystem tests, SPEC (and remember for SPEC to be meaningful, you need 10 client machines - that's what our rig at iX looks like).

I say this because benchmarking discussions generally result in more heat than light, and not just when cyberjock and jyavenard are involved. :) I still remember the great lmbench war of 1993. It wasn't pretty.

 

[jyavenard]

And somehow dd2 is supposed to be better? dd is designed to test maximum possible throughput. If you want I/O testing you use other tools. Didn't dad ever teach you to use the right tool for the right job?

dd2 works like dd, but allows to write random data, without the randomisation impacting the speed test.

dd2 can generate random data at a rate of several gigabytes per second.

It's identical to doing: dd if=/dev/random ....
without the downside of reading /dev/random being too slow...

that's what it was designed to do: no more, no less...

 

[jyavenard]

I think it's probably best if we stay away from benchmark discussions in the forums unless those doing the benchmarks are willing to use more complex benchmarks like iozone, bonnie or, for actual network filesystem tests, SPEC (and remember for SPEC to be meaningful, you need 10 client machines - that's what our rig at iX looks like).

Sorry, but that's exactly the tools I used in my benchmark thread ! iozone + bonnie... It doesn't matter what data I present. The end result (e.g. criticisms is always the same).

I'm not intending to benchmark per say... What I'm trying to achieve here is two fold:
1- Compare write speed between encrypted vs unencryped
2- Compare write speed between encrypted 9.1.1 and 9.2 with new code in...

I hope I'll achieve that soon

 

[jkh]

Well, criticism and benchmarking tend to go hand in hand, which is why I'm generally so averse to the topic. Someone posts numbers, then someone else says the testing was obviously flawed because X, and then the original poster comes back and says "No, my testing was fine, you just didn't understand what was being tested" and then someone else cites 30 years of industry experience which categorically proves the numbers are totally bogus and everyone in the thread are morons, and then things get really personal. :) I'm not just talking about this thread, believe me. I wish it were that easy. I'm talking about every single benchmarking thread, against almost every single possible thing it's possible to benchmark (syscall performance, context switching, memory allocation, I/O, network latency, you name it) I've ever seen. :-/

 

[cyberjock]

I'm not intending to benchmark per say... What I'm trying to achieve here is two fold:
1- Compare write speed between encrypted vs unencryped
2- Compare write speed between encrypted 9.1.1 and 9.2 with new code in...

And somehow using random data versus dd is going to matter? There's no distinction between encrypted and unencrypted with regards to /dev/zero versus any other source. So why would you go and use a program that isn't industry standard, then use the results as if they mean something? The whole reason why industry standard tests are used is because they've been heavily reviewed, fixed of errors, and validated to be usable for an intended function. Encryption doesn't care if its full fledged /dev/random or /dev/zero. The processing power required is still the same.

That's my WHOLE point.

That is also when I plan to do my benchmarking with the 9.2.0-RC I plan to use a RAM drive. I don't want latency or any kind of disk issue to contaminate the results. I want to know the real penalty and the real limits to what this new code can do. I don't need to find out my pool is 10% faster. I want to know the actual theoretical throughput of the system and its ability to encrypt and store data without losses from the storage subsystem. Then you end up with a very pure product.

Your scientific method is very broken.

 

[jkh]

dd2 works like dd, but allows to write random data, without the randomisation impacting the speed test.
dd2 can generate random data at a rate of several gigabytes per second.

I think what's getting cyberjock's underwear all twisty is the fact that the usage of random data is completely unnecessary here. As he points out, encrypting a block of zeros and a block of random data will take the *same* amount of computational time. The same. Whether it's AES-CBC or AES-XTS at work (and 128 or 256 bit AES for that matter), the overhead of the algorithm is always in cycles/byte. Not in cycles/type-of-byte. All AESNI does is reduce the number of cycles per byte.

That's just not how crypto works. If you (or anyone else) want to prove it to yourself, try a simpler test:

dd if=/dev/random of=/tmp/randoms bs=4k count=10000
dd if=/dev/zero of=/tmp/zeros bs=4k count=10000

Then:

time openssl aes-256-cbc -k thisisareallyfinepassphrase -in /tmp/randoms -out /tmp/xxx
time openssl aes-256-cbc -k thisisareallyfinepassphrase -in /tmp/zeros -out /tmp/xxx

See?

 

[jyavenard]

I think what's getting cyberjock's underwear all twisty is the fact that the usage of random data is completely unnecessary here. As he points out, encrypting a block of zeros and a block of random data will take the *same* amount of computational time. The same. Whether it's AES-CBC or AES-XTS at work (and 128 or 256 bit AES for that matter), the overhead of the algorithm is always in cycles/byte. Not in cycles/type-of-byte. :) All AESNI does is reduce the number of cycles per byte.

hold on a second...

You have to look at things in their context...

In only talked about writing random data in this post:
http://forums.freenas.org/threads/encryption-improvements-in-9-2-0.16609/#post-86229

I know writing zeros or random in this instance should have made no difference. Yet, I was seeing massive variation when writing zeros, but not when writing random data.
I didn't claim that it was going to happen, it was just what I was experiencing.

So my conclusion was:

it fluctuates quite a bit, not as much as writing zeros though... how crazy is that?!

I never said anything about the need to write random data vs writing zeros.

I only mentioned this particular idiosyncrasy under that context.

This introduced that little dd2 utility that I wrote.

To which I got questioned about (dd2) (and I replied, maybe I should have done what you earlier suggested: just ignore all his posts) and got as an answer:

there's no fast way to generate large amounts of data for low CPU usage.

to which I replied that this was very incorrect, there's plenty of way to do so (my code contains 8 different methods doing just that) and at a very fast rate. And it got off on a side-tangent talking about compression.

Following which, once again a strawman argument was built about testing compression vs uncompressed in the context of speed test; or that the differences I was seeing were due to the use of the tool dd2....

If you go back, you'll see that I've *never* said such thing.... I didn't even imply it.

It's all started when I mentioned on how surprised I was to see a difference (when there should be none) when writing zeros (speed varies greatly) vs writing random (speed don't vary as much)

This may be worth reading to put things into context... this explains really what's going on and how arguments are being build up here.

Hope that shed enough light, for you to clearly see through the mud being thrown around

 

[jyavenard]

Well, I can only assume the RC build I was trying earlier didn't have the encryption improvement...

It is indeed *significant* improvement over 9.1.

Copying using zfs send | zfs receive: 8.17T as reported per zfs send

unencrypted -> unencrypted:
482m20.667s

unencrypted -> encrypted
FreeNAS-9.2.0-RC-d129882 : 571m 43.088s
FreeNAS-9.2.0-RELEASE-6eccc24-x64 (as tagged on github): 476m 44.13s

You read well, writing to the encrypted pool is now even faster... Impossible you say, and so do I... Must be due to other factors, after all the results are only 1.3% of one. Within normal variation...

*but*...
I ran the following command:

repeat 3 dd if=/dev/zero of=/mnt/pool/testing/test.big bs=1048576 count=100000

followed by:

repeat 3 dd if=/dev/zero of=/mnt/pool2/testing/test.big bs=1048576 count=100000

pool is unencrypted, pool2 is encrypted (pool2 was created under 9.2-RC latest, and it's good to see that it can still be used with 9.1)

this creates a 100GB file, three times in a row.

What I found over several times re-running the scenario above:
1- I always end up with writing to the encrypted pool being slightly faster than writing to the unencrypted pool.
2- 9.1 was always slightly faster than 9.2 writing to the unencrypted pool (around 3% difference)

For 1-, It could be that the LSI SATA adapter (used by pool1) isn't as quick than the intel adapters that is used for pool2.

Still, the important result: as mentioned earlier: there's now no penalty in using an encrypted pool vs unencrypted.

Congratulations to FreeNAS and FreeBSD teams... that's just nothing short of amazing...