Multiple controllers X bandwidth ( including PCIe ) ( SaS6/iR vs 2 SaS5/iR)

[Mauricio Silveira]

Hi all.

I'm new to FreeNAS, but I'm an experienced sys admin, and I try to improve my knowledge from time to time.

FreeNAS research led me to other topics I never really dived into before such as SATA bandwidth saturation calculations...

I'll give FreeNAS a shot very soon, but I'd like to to have some theory calrified:
From what I've just learned, in a 12-drive array, using an on-board intel 6 port SATAII controller is not enough, so should I better use one SaS6/iR or 2 SaS5/iR controllers to balance the drives over the total 2/3 controllers?
From my POV, theoretically, I should go with onboard intel + 2 x SaS5/iR, amd I right? Or only benchmarking should tell the truth?

I know 1Gbit NIC would be my bottleneck for a single connection, but in a read intense environment, 4x1Gbit LACP for a dozen Virtualization Hosts, my guess is that I or any customer of mine will be happy with the results.

BTW, I was searching for SaS5/iR at local ebay ( mercadolivre in Brazil ), but I only found SaS6/iR and they're cheap, second hand! So, I'd use 2 SaS6/iR instead of 2 SaS5/iR.

 

[tvsjr]

You would only hit this limit in extreme edge cases. As you point out, your NICs are far more a limiting factor than a single SATA2 interface. In a virtualization environment, you're also far more concerned with IOPS than with aggregate bandwidth. In short, you're adding complexity where you won't see benefit.

Make sure you're picking a card that can be put in pure HBA mode.

If you're planning to use LACP to add bandwidth, understand that this only happens in aggregate. The connection from FreeNAS to a specific host will be limited to 1Gbps. Make sure you choose your storage IPs smartly to ensure the traffic is properly spread across the NICs. Or, go 10Gbps, which is a better option.

 

[Chris Moore]

I happen to have two FreeNAS installations, other than the ones listed in my signature, that I am putting through testing right now and the information I have gained is exactly what you are looking at.
One system has two Dell PERC H310 cards (flashed with the IT firmware, see this link: http://www.vladan.fr/flash-dell-perc-h310-with-it-firmware/).
The second system has two Dell SAS 6/iR cards ( no flashing of the ROM needed).

In the system with the PERC H310 cards (they run at SATA-3 speed) I am able to saturate a 10 GB network interface all the time with capacity to spare as the total potential of the connection inside the server is 1200 where a 10 GB NIC actually tops out about 900.

In the system with the SAS 6/iR interface cards (they run at SATA-2 speed) the best I was able to get was around 800, with the average being closer to 600 and this falls well below the capability of the 10 GB network interface.

The PERC H310 cards are around $55 on eBay and the SAS 6/iR cards are around $10 on eBay. So, it is all down to the performance you are looking to obtain.

In my testing, I had 12 enterprise class SATA-3 drives and connected 6 to each of the two controllers. In FreeNAS, I created two RAID-Z2 vdevs striped together into one pool. I tried a couple of other configurations but this was the one that gave the best combination of performance and capacity.

I am still testing these two servers prior to putting them into production and I have already ordered two more of the PERC H310 cards to upgrade the second server so they will be equal in performance.

If anyone has questions, I will be happy to give further observational data.

 

[Mirfster]

So, I'd use 2 SaS6/iR instead of 2 SaS5/iR.

Just to note that the SAS6/iR cannot see over 2.2TB of any one drive. So any drives you attach to it will top out at 2.2TB each; regardless if they are 3TB, 4TB, etc...

 

[Chris Moore]

Just to note that the SAS6/iR cannot see over 2.2TB or any one drive. So any drives you attach to it will top out at 2.2TB each; regardless if they are 3TB, 4TB, etc...

True, I didn't consider that. I did my test with all 2 TB drives, so this never came up, but it would be better to have the flexibility to go to a larger drive in the future.