FreeNas 9.2 vs FreeNas 9.3

[Jano]

Hi
I made some comparation of FN9.2 (9.2.1.5) and latest 9.3 about performance.

As test platform VMWare IO Analyzer was used.

Results are strange, specially on 9.2.

Could you please take a look for this ? Add some comments ? Answers to questions on the last page ?


Edited:
New version of document (v2) has been added.

 

[L]

So it would be nice to understand what you are trying to see? I would typically want to run benchmarks with systems that are identically configured. The diff's between your 9.2 and 9.3 systems are significant.

 

[jkh]

Hi
I made some comparation of FN9.2 (9.2.1.5) and latest 9.3 about performance.

Yes, at first glance this seems like a nice performance report - you documented your setup, you graphed the results, neat! However, as Linda says, you also broke the first rule of performance testing: You changed multiple variables at the same time, significant variables in fact, such that the results you came up with were essentially meaningless.

Here's what you do: Go back and do this again, except this time, keep the VMWare / Switch topology the same and also use the same client (same physical machine, same memory, everything) but just switch the software in between test runs. Graph those results. We will certainly be interested and look at them. If you are testing 9.2.1.x, you should also try the tests with experimental iSCSI target on and off, since you will be testing two different iSCSI software stacks there (9.3 has only one). You should, at the end of the test runs, have 3 different (and significant) points of comparison with just one variable changed for each.

Thanks.

 

[Jano]

First intention was to measure performance of RAID 10 (over 2, 4, 6 WD BLACK and 2 WD RED) with FN9.3 to estimate performance with RAID10 (over 14x 2TB WD RED) with FN9.3 after migration (just estimate).

After collecting data I tried to understand this results and...
FN9.3 works as expected.
FN9.2 far away for expectation - but if I take into account fact that gstat reports all disk %busy < 20 and network saturation far away from full 4 GBit (even not 50% of it) I started to think that the same methodology of test working with FN9.3 is not working with FN9.2.

ESXi generate random access IO, so I think using dd for tests does not give real results.

Just for test I made dd few minutes ago and I had results are 200-600MB/s (of course some data were in cachce so to have real results there is necessary to eliminate this).

 

[depasseg]

I understand that there were significant changes/enhancements to ISCSI in 9.3. I think your tests might confirm that.

 

[jkh]

First intention was to measure performance of RAID 10 (over 2, 4, 6 WD BLACK and 2 WD RED) with FN9.3 to estimate performance with RAID10 (over 14x 2TB WD RED) with FN9.3 after migration (just estimate).

Yes, but ZFS uses memory *very* aggressively for caching and one of your two systems has 2X the memory of the other, which invalidates the tests. It's fine to just try to get raw numbers from a single system as an "acceptance test", but there's no point in *comparing* those numbers unless the test configurations are as identical as you can make them, that's the only point.

It's also somewhat pointless to worry about 9.2.1.x performance at this point as that branch is dead. 9.2.1.9 was the last release, and there won't be any more, so even if you found some terrible performance bug in 9.2.1.x, we'd just tell you to upgrade to 9.3 since the time to have pointed this out would have been about a year ago, when 9.2.1 was released and the problem could have been addressed somewhere in the following 9 point releases. :)

 

[L]

Love to see your results. What I would do is create several usb flash drives for os(on 9.2) use the same hardware config for testing. On 9.3 you can create multiple boot environments so you can make modifications.

 

[Jano]

Yes, at first glance this seems like a nice performance report - you documented your setup, you graphed the results, neat! However, as Linda says, you also broke the first rule of performance testing: You changed multiple variables at the same time, significant variables in fact, such that the results you came up with were essentially meaningless.

Here's what you do: Go back and do this again, except this time, keep the VMWare / Switch topology the same and also use the same client (same physical machine, same memory, everything) but just switch the software in between test runs. Graph those results. We will certainly be interested and look at them. If you are testing 9.2.1.x, you should also try the tests with experimental iSCSI target on and off, since you will be testing two different iSCSI software stacks there (9.3 has only one). You should, at the end of the test runs, have 3 different (and significant) points of comparison with just one variable changed for each.

Thanks.

That would be nice but not possible with hardware I have.
That's fully right that many factors has been changed but all of them refer to server of FN.
At first glance FN9.2 server looks more powerful that test machine so for estimation tests, even if many factors were changed general intention should be still correct - have data showing something significantly better with FN9.2 and FN9.3.

Even if I will not compre FN9.3 and FN9.2 later you agree with me that performance registered ith FN9.2 is strange.

 

[Jano]

Yes, but ZFS uses memory *very* aggressively for caching and one of your two systems has 2X the memory of the other, which invalidates the tests. It's fine to just try to get raw numbers from a single system as an "acceptance test", but there's no point in *comparing* those numbers unless the test configurations are as identical as you can make them, that's the only point.

It's also somewhat pointless to worry about 9.2.1.x performance at this point as that branch is dead. 9.2.1.9 was the last release, and there won't be any more, so even if you found some terrible performance bug in 9.2.1.x, we'd just tell you to upgrade to 9.3 since the time to have pointed this out would have been about a year ago, when 9.2.1 was released and the problem could have been addressed somewhere in the following 9 point releases. :)

Point is I do not believe I found terrible performance problem rather methodology of tests were not ok in case of FN9.2
Also point is I want to migrate to 9.3 as I like new features added and believe CTL is right way of evolution.

Just objective it to understand, to learn why

 

[L]

Actually I spent the time to really look at your results and wow. The differences between your FN9.2#1 and #2 seem amazing. Am I seeing this corrrectly that the only difference is iscsi config? It looks to me like they are identical configs with just changes in
FirstBurstLength = 256KB MaxBurstLength = 256KB MaxRecvDataSeqLength = 128KB

It looks like you got a 2x perf improvement.

 

[L]

The other thing I find very interesting is that you are not getting linear performance increase on the different block sizes especially on the 9.2. Do you set the recordsize = to the test io size? Halfing the io size should double the number per second. Odd that they are flat on 9.2 #2.

 

[Jano]

No, as I said 9.2 is my production solution and I was able to use it for limited tests.
Just I play with iSCSI settings.

 

[L]

I have yet to find the docs or methods for setting iscsi settings in 9.3 are you setting these in the config file and restarting?

 

[Jano]

In case of 9.3 settings of iSCSI are default as well as BS of ZVOL.
In case of 9.2 after every change system was restarted, then warm up by 30 minutes sequencial load (512b blocks) till memory allocation will grow till 80...90%

In fact one of my question is where to find doc of iSCSI settings (more than simplest like chap, extends etc) in 9.3 - these settings were removed from UI and in fact did not find information they are hardcoded and not possible to change (except sources) or are available by another way.

 

[Jano]

Document has been updated with 6 new test cases (allowing compare) - what do you think now about results and conclusions ?

 

[zambanini]

comparing that setup with nexenta would be nice.

 

[jkh]

Document has been updated with 6 new test cases (allowing compare) - what do you think now about results and conclusions ?

That is MUCH better, thank you! Very thorough, and takes care to segregate the numbers out into categories that can be more easily compared and understood!

As to the conclusions, I guess all I can say is that this kind of performance increase is what we were going for with the CTL work and all of the workload optimization that has gone into it (in some scenarios, it is even possible to break a million IOPS, with the right SSD HW and controllers)!

Your results with RAID10 are also not too surprising - there is only so much optimization that additional spindles can bring you with SATA before you run into other limitations at the controller bottleneck, and SATA drives in general are not known to be speed demons. Choosing an RAID LSI 9260-16i configuration was also kind of strange - that is not generally recommended, and most folks who buy the LSI IR cards (with SAS) flash them immediately to IT firmware to get the redundant, and performance-impacting, RAID firmware completely out of the way. You would do far better with a good SAS controller, a decent SAS expander and SAS drives if performance for VMWare is your goal (that's all we sell in configurations other than the FreeNAS Mini, which only has 4 drives max and is tailored for a different market).

In short, I think the SW (9.3) has taken you about as far as you can go, and it's time for a better HW configuration if you want to increase your IOPs. If you can go with SAS SSD drives, that will drive the numbers even higher (quite a bit)!

 

[Jano]

Well so finally I migrate production server to 9.3 and bellow you can find last version of document with updated data. Really 9.3 works nice and significantly better that 9.2.

Hope this analyse will help other persons.

Enjoy.