SATA port considerations

[joeinaz]

I don't know if this has been discussed before but here goes: I am finalizing a new FreeNAS build and the last consideration is drive connection. I want to get thoughts on the following options:

1. HBA (M1015 in IT mode for example)
2. Onboard SATA ports (My motherboard has 8 SATA ports)
3. SATA ports on a PCIe card (I am looking at a 4 port Vantec card)

Other questions:

3Gb/s vs 6Gb/s how much of a difference?
SATA ports from different sources; can/should you create a FreeNAS array from same speed SATA ports from say a mixture of motherboard and PCIe card ports?

I current use an M1015 but want to look at other things for my new (up to) 12 disk project.

 

[Maelos]

I will give it a go, but more experienced users will likely chime in. If you are building a "bare metal" FreeNAS build where there is no hypervisor beneath it, no RAID, or other program/platform between FreeNAS and the disks, you will be OK with either onboard SATA ports or an HBA. I would, however, suggest avoiding SATA port PCIe cards as most people here use an SAS card (HBA flashed to IT mode) with mini SAS to SATA cables. Let me see if I can find a few of the guides that have this info for you...

https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/ <- Much of the advice is already there.

Common mistakes
The following are common mistakes which should be avoided at all costs:
• Using any SATA controller card. These are universally crap.
• Using SATA port multipliers. Intel SATA controllers do not support these, they are built down to a price and are generally unreliable.
• Using any sort of hardware RAID o Exporting individual drives as RAID0 volumes is not a good solution o The vast majority of hardware RAID controllers is not appropriate for FreeNAS

https://forums.freenas.org/index.php?resources/freenas®-quick-hardware-guide.7/ <- A slimmed down version of the above

 

[joeinaz]

I understand the basic recommendations. I have been using FreeNAS for years both off of the motherboard and using an HBA. I am looking for detail reasons why something may or may not work, perform better and or be more reliable. In case of a 12 disk array, there are numerous ways to connect and I am considering options.

 

[Nick2253]

Typically, an onboard controller is very reliable (assuming, of course, that we're actually talking about one of the recommended motherboards). Unless you need extra drives, or you are virtualizing FreeNAS, there's no good reason to spend the money on an extra SAS HBA.

PCI/PCIe SATA cards, are a different story. The chipsets/controllers on these cards are typically not well supported in FreeBSD (same is true for many consumer mobo onboard controllers). This can lead to issues with mysterious data loss, crashes, and general instability that ultimately puts your data at risk. Understand that this isn't just a FreeNAS problem: outside of Windows, most of these PCIe/PCI cards are not well supported by quality drivers, and their low cost preempts the manufacturer from spending lots of dev effort to change it.

3Gbps is sufficient for pretty much every HDD made. There are a few high-speed SAS drives that can just exceed 3Gbps, but I don't know a single SATA HDD that can. SSDs are a different story. Many SSDs will benefit from 6Gbps. So, it really depends on what your use case is. Mixing and matching ports isn't a concern: the beauty of FreeNAS is that it will aggregate across many different controllers. In my opinion, you'll need to be mindful of these speed or latency differences only when using SSDs.

 

[joeinaz]

Thanks Nick2253, this is the kind of data I am looking for...

 

[kdragon75]

Another argument for 6Gbps controllers its if you are using a backplane with an expander. You may still benefit from the extra bandwidth because it's being shared across many drives.

 

[Chris Moore]

I suggest using a SAS expander.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[joeinaz]

I suggest using a SAS expander.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

I just received my expander (HP) in the mail yesterday! Now I am working on the optimal configuration; I can go 10 - 12 disks with 2 or more SSDs for performance. I also may integrate an LTO4 tape for backup. I was thinking of running Windows Server 2016 and Exchange server using FreeNAS virtualization, or simply using the FreeNAS as a iSCSI target for an external server. The key is getting the best performance.

 

[Chris Moore]

I just received my expander (HP) in the mail yesterday!

The key is getting the best performance.

What model SAS expander did you get?

 

[joeinaz]

What model SAS expander did you get?

HP 487738 24 port 3Gb/s expander.

 

[kdragon75]

I just received my expander (HP) in the mail yesterday! Now I am working on the optimal configuration; I can go 10 - 12 disks with 2 or more SSDs for performance. I also may integrate an LTO4 tape for backup. I was thinking of running Windows Server 2016 and Exchange server using FreeNAS virtualization, or simply using the FreeNAS as a iSCSI target for an external server. The key is getting the best performance.

If this a business application I would strongly suggest using ESXi or hyper-v as your hypervisor or if your up to the task and are a Linux/BSD expert, KVM. bhyve on FreeNAS is way to much in flux to be used in a business production environment.

Also if you only have one VM host ESXi Free works great and point point Fibre Channel is dirt cheap. While the GUI is not built for it, it's not hard to set up. Most of the config is identical to iSCSI as thats 99% just a front end for CAM Target Layer, just a different driver. I know most qLogic cards are well supported. Im planning to use the qle2564 with two ports to each of my two hosts.

 

[joeinaz]

If this a business application I would strongly suggest using ESXi or hyper-v as your hypervisor or if your up to the task and are a Linux/BSD expert, KVM. bhyve on FreeNAS is way to much in flux to be used in a business production environment.

Also if you only have one VM host ESXi Free works great and point point Fibre Channel is dirt cheap. While the GUI is not built for it, it's not hard to set up. Most of the config is identical to iSCSI as thats 99% just a front end for CAM Target Layer, just a different driver. I know most qLogic cards are well supported. Im planning to use the qle2564 with two ports to each of my two hosts.

I am very familiar with VMware and have it running on one of many Blade servers we have. Fibre channel is an excellent choice if I could find an inexpensive pair of fibre channel switches for my Blade chassis. I have some converged network cards (basically 10GbE) available but I am not sure how compatible they are with FreeNAS. I was going to get started with Gigabit Ethernet. One option is to use bhyve to run a single Linux VM to run tape backup and have the FreeNAS system act as a storage and backup appliance to a number of blades. The other option is to have ESXi as the base operating environment (using hardware pass-through for FreeNAS) and virtualize both my FreeNAS and Windows Server installations. I am running a 10 core E5-2650 with 32 - 48GB of RAM as a starting point. This would allow me to separate my company email installation from the Blade chassis which is currently more of a test/dev environment.

 

[Chris Moore]

HP 487738 24 port 3Gb/s expander.

So, with that, you will want to run a single link from the SAS controller to the SAS expander. All the regular HDDs will connect to the SAS expander because 3Gb/s is fast enough for spinning disks. the other port on the SAS controller can use a regular breakout cable to connect up to 4 SATA SSDs and still get the 6Gb/s speed to them.
I am a believer in SAS as the best answer for connecting drives. SATA is fine for a few drives, but when you have more drives, SAS really scales well. Instead of needing a bunch of cards, you can just add an expander.
The system I am using now for my main FreeNAS has two 24 slot expander backplanes connected to one SAS controller and it is faster now than when I was using two SAS controllers with half the drives on each controller.

3Gb/s vs 6Gb/s how much of a difference?

For spinning disks, no difference because the fastest spinning drive I am aware of (an HGST 10TB unit) delivers data to the controller at a max of 238MB/s (based on their specs) and that isn't fast enough to fully saturate 3Gb/s SATA/SAS

SATA ports from different sources; can/should you create a FreeNAS array from same speed SATA ports from say a mixture of motherboard and PCIe card ports?

It has already been said, but the add on card SATA interfaces are (in my experience) prone to problems that make them undesirable in a server. I stick to SAS controllers for all my data drives. I have two boot drives on the integrated SATA controller and don't use it for anything else.

I current use an M1015 but want to look at other things for my new (up to) 12 disk project.

The M1015 is a fine card. I run the equivalent Dell H310 SAS controller in all my FreeNAS systems and in my main one each of the connectors is connected to a 24port SAS expander to give me the potential of connecting 48 drives. The card is good up to over 200 drives because you can cascade SAS expanders.

 

[Chris Moore]

The other option is to have ESXi as the base operating environment (using hardware pass-through for FreeNAS) and virtualize both my FreeNAS and Windows Server installations.

Did you look at the build log by @Stux where he documents his ESXi / FreeNAS all in one build?

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

 

[Johnnie Black]

Like Chris mentioned 3Gbps is fine for spinners, but a single SAS1 link for the expander might be a bottleneck, depending on how many disks you're using, you'll have around 1100MB/s usable for all disks, also check that the disks are linking @ 3Gbps, as those expanders with earlier firmwares can only link @ 1.5Gbps with SATA disks.