Recommendations for benchmarking before and after RAID card swap?

[oRAirwolf]

I am going to be upgrading my RAID card in my FreeNAS server next week. I am going from a Dell H310 in IT mode (LSI SAS 2008) to an IBM M1215 in IT mode (LSI SAS 3008). I was hoping to do some performance comparisons to see if moving over to a faster controller will make any difference in performance. Do you guys have any suggestions on benchmarks that can be run on FreeNAS to test things like sequential performance over time or random IOPS at various queue depths? I have a fairly simple setup of 8 x 8TB WD Red 5400 RPM drives and 48GB of RAM in a single RAIDZ1 vdev. I can also test from Windows, but I am limited to the performance of a 10Gbe link.

 

[Johnnie Black]

There won't be any difference, H310 is good for at least 2500MB/s and you can't get anywhere close to that with 8 WD Reds, basically unless you're using SSDs or a lot of disks connected to an expander it will be the same.

 

[Arwen]

In rare cases, upgrading a controller card, (Storage, Network, etc...), can improve performance. But, as @Johnnie Black said, without using SSDs or a SAS expander, it's unlikely. (But, one of the rare cases I have seen, is that the newer cards have faster control logic, and reduced latency. Generally only a few percentage points of performance improvement if any, below the easy test measurement threshold level.)


You might want to use HBA instead of RAID, for the places where you said "RAID card". We get kinda nervous when people mention "RAID card"s, (though you were both clear and concise in stating they were in IT mode).

Brought to you by the Nit Picky Police, (say that 10 times fast, without errors and you get a twinkie!).

 

[joeschmuck]

I was hoping to do some performance comparisons to see if moving over to a faster controller will make any difference in performance. Do you guys have any suggestions on benchmarks that can be run on FreeNAS to test things like sequential performance over time or random IOPS at various queue depths?

To answer your question... Look in the resources section of the forum and also look into "dd" testing as well. I'm assuming that you did not do a Google search for "freenas benchmark testing". I did some benchmark testing many years ago and they would still be very valid, do a search for "freenas realtek nic joeschmuck benchmark" and you will find it. While this was primarily testing in NIC performance, it still has some good testing you can do for FreeNAS internal performance.

 

[oRAirwolf]

@Johnnie Black @Arwen @joeschmuck

I thought you might be interested to hear an update. I didn't do anything too crazy, but I did some tests with Crystal Disk Mark 6.0.1 over my 10 Gbe link (Aquantia AQC107 RJ-45 -> Netgear S3300-28X Switch -> Mellanox ConnectX-3 EN SFP+) to my file server before swapping out the H310 for the M1215. I'm not saying these are definitive results, but I was pretty happy with the performance increase. Especially with random read performance. All tests were done 5 times with a 32GB dataset.

Dell PERC H310 SAS 2008

Code:

Run 1

   Sequential Read (Q= 32,T= 1) :   980.828 MB/s
  Sequential Write (Q= 32,T= 1) :   588.256 MB/s
  Random Read 4KiB (Q=  8,T= 8) :	11.760 MB/s [   2871.1 IOPS]
 Random Write 4KiB (Q=  8,T= 8) :	 7.919 MB/s [   1933.3 IOPS]
  Random Read 4KiB (Q= 32,T= 1) :	13.232 MB/s [   3230.5 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :	 7.173 MB/s [   1751.2 IOPS]
  Random Read 4KiB (Q=  1,T= 1) :	 6.824 MB/s [   1666.0 IOPS]
 Random Write 4KiB (Q=  1,T= 1) :	 6.909 MB/s [   1686.8 IOPS]

  Test : 32768 MiB [Z: 32.7% (15827.1/48433.4 GiB)] (x5)  [Interval=5 sec]
  Date : 2018/10/28 18:31:44
	OS : Windows 10  [10.0 Build 17134] (x64)
	
-----------------------------------------------------------------------

Run 2

   Sequential Read (Q= 32,T= 1) :   883.328 MB/s
  Sequential Write (Q= 32,T= 1) :   609.082 MB/s
  Random Read 4KiB (Q=  8,T= 8) :	 3.415 MB/s [	833.7 IOPS]
 Random Write 4KiB (Q=  8,T= 8) :	 2.535 MB/s [	618.9 IOPS]
  Random Read 4KiB (Q= 32,T= 1) :	 3.761 MB/s [	918.2 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :	 2.785 MB/s [	679.9 IOPS]
  Random Read 4KiB (Q=  1,T= 1) :	 5.015 MB/s [   1224.4 IOPS]
 Random Write 4KiB (Q=  1,T= 1) :	 3.825 MB/s [	933.8 IOPS]

  Test : 32768 MiB [Z: 32.7% (15827.1/48433.4 GiB)] (x5)  [Interval=5 sec]
  Date : 2018/10/28 18:54:08
	OS : Windows 10  [10.0 Build 17134] (x64)

IBM M1215 SAS 3008

Code:

Run 1

   Sequential Read (Q= 32,T= 1) :  1222.577 MB/s
  Sequential Write (Q= 32,T= 1) :   595.054 MB/s
  Random Read 4KiB (Q=  8,T= 8) :	73.566 MB/s [  17960.4 IOPS]
 Random Write 4KiB (Q=  8,T= 8) :	 3.872 MB/s [	945.3 IOPS]
  Random Read 4KiB (Q= 32,T= 1) :	71.973 MB/s [  17571.5 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :	 4.740 MB/s [   1157.2 IOPS]
  Random Read 4KiB (Q=  1,T= 1) :	 6.833 MB/s [   1668.2 IOPS]
 Random Write 4KiB (Q=  1,T= 1) :	 5.651 MB/s [   1379.6 IOPS]

  Test : 32768 MiB [Z: 32.7% (15827.1/48433.4 GiB)] (x5)  [Interval=5 sec]
  Date : 2018/10/28 20:29:29
	OS : Windows 10  [10.0 Build 17134] (x64)

-----------------------------------------------------------------------

Run 2

   Sequential Read (Q= 32,T= 1) :  1151.499 MB/s
  Sequential Write (Q= 32,T= 1) :   595.249 MB/s
  Random Read 4KiB (Q=  8,T= 8) :	73.534 MB/s [  17952.6 IOPS]
 Random Write 4KiB (Q=  8,T= 8) :	 7.725 MB/s [   1886.0 IOPS]
  Random Read 4KiB (Q= 32,T= 1) :	71.999 MB/s [  17577.9 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :	12.545 MB/s [   3062.7 IOPS]
  Random Read 4KiB (Q=  1,T= 1) :	 6.739 MB/s [   1645.3 IOPS]
 Random Write 4KiB (Q=  1,T= 1) :	 6.795 MB/s [   1658.9 IOPS]

  Test : 32768 MiB [Z: 32.7% (15827.1/48433.4 GiB)] (x5)  [Interval=5 sec]
  Date : 2018/10/28 20:35:53
	OS : Windows 10  [10.0 Build 17134] (x64)

I'm sure these results could be picked apart, but the upgrade to a SAS 3008 over a SAS 2008 may be worth looking into to squeeze some extra performance out of a ZFS array.

 

[Johnnie Black]

There could be some performance increase, not because of the extra bandwidth but possibly because the HBA chip should be faster and with less latency, as for the random read performance it does appear to be better, still there's such a big difference in the in the first 2 runs with the H310 that puts some doubts on the results consistency, but any performance increase is good news.

 

[joeschmuck]

I would have to say that the testing looks like a good step in the right directions however if you wanted to do a more subjective test I'd make sure you setup the dataset with no compression, remove your RAM down to 16GB and then try your testing again because 32GB of test data can fit in 48GB of RAM (part of the beauty of having so much RAM). You want to removal all potential caching or compression advantages to test for true throughput results. Crystal Disk is a nice tool and I use it as well but to test the difference between the two cards I would run some internal throughput testing vice 10Gb interface testing.

I'm not trying to poke holes in your testing, please don't take it that way, I'm pretty impressed by the results and I think it would be cool to wire my home to 10Gb too but I have more important things to do with my money such as rebuilding my truck rear differential, not gonna be cheap.

I personally don't care for the H310 but that is just me, so I think you made a good choice in the change and I'm glad that it is meeting your expectations. If you do more performance testing, post the setup of the dataset and your testing program/application as this data would tell us if you are getting results that are skewed.