HBA suggestion for a 2U 20-30TB build

[George Kyriazis]

Hello FreeNAS folks,

I am contemplating doing a built with the following characteristics (this is for a work environment, mostly datasets):
- Intel R2312WFTZS system (dual Skylake with 12 sas/sata bays) (https://www.intel.com/content/www/u.../server-board-s2600wf-systems/r2312wftzs.html)
- need about 20-30 TB storage space currently, with more in the future.

I was thinking of using 4-6 12TB drives in raidz2. From what I've read and from prior experience, that should be a good layout choice. Are 12TB drives (thinking about Exos He12) robust those days? How about SAS vs SATA?

I have some questions regarding the HBAs. Between the following 2 options, which one has better support on FreeNAS:
- Intel RMS3AC160 (using Broadcom SAS 3316 chipset) (sata/sas)
- Intel RSP3TD160F (using Avago SAS 3516 chipset) (sata/sas/nvme)

Both of them are RAID-type chipsets, but do they support JBOD?

 

[jgreco]

Hello FreeNAS folks,

I am contemplating doing a built with the following characteristics (this is for a work environment, mostly datasets):
- Intel R2312WFTZS system (dual Skylake with 12 sas/sata bays) (https://www.intel.com/content/www/u.../server-board-s2600wf-systems/r2312wftzs.html)
- need about 20-30 TB storage space currently, with more in the future.

I was thinking of using 4-6 12TB drives in raidz2. From what I've read and from prior experience, that should be a good layout choice. Are 12TB drives (thinking about Exos He12) robust those days? How about SAS vs SATA?

I have some questions regarding the HBAs. Between the following 2 options, which one has better support on FreeNAS:
- Intel RMS3AC160 (using Broadcom SAS 3316 chipset) (sata/sas)
- Intel RSP3TD160F (using Avago SAS 3516 chipset) (sata/sas/nvme)

Both of them are RAID-type chipsets, but do they support JBOD?

It isn't a question of if they support JBOD. They do. They are not, however, HBA's. They roughly correspond to LSI 9360-8i and 9460-8i. They are RAID controllers. Period. Stop. Not HBA's.

The most robust drivers for storage, by far, for FreeNAS are the Intel SATA and LSI 9211-class HBA. These have billions of problem-free hours under their belt and work flawlessly. If you really need a 12Gbps HBA (which won't be meaningful or useful with hard drives, so you really don't need it) you want to go with something like the LSI 9300-8i. But there aren't a lot of people who are using this.

Unless you're intending to upgrade to flash at a later date, the smarter move is to use an LSI 9211-8i. This has been the case for many years and they work flawlessly. You can get any of the OEM equivalents and crossflash them into IT mode.

https://www.ixsystems.com/community/threads/confused-about-that-lsi-card-join-the-crowd.11901/

 

[Chris Moore]

If you really need a 12Gbps HBA (which won't be meaningful or useful with hard drives, so you really don't need it) you want to go with something like the LSI 9300-8i. But there aren't a lot of people who are using this.

No argument with any of that, but I was specifying parts for a new server recently and there was no option to go with a "lesser" model than the LSI 9300-8i, it works.

I have some questions regarding the HBAs. Between the following 2 options, which one has better support on FreeNAS:
- Intel RMS3AC160 (using Broadcom SAS 3316 chipset) (sata/sas)
- Intel RSP3TD160F (using Avago SAS 3516 chipset) (sata/sas/nvme)

If these are the only two options the vendor has, you are looking at the wrong vendor.

 

[George Kyriazis]

Thanks guys. So, bottom line both 9211 or 9300 will work fine.

Let's assume I get a 9211. Since 9211 does only 6Gbps, is there a real-world benefit for getting SAS drives vs SATA? I know the SCSI protocol is better, but in real-life scenarios does it make a difference?

 

[Chris Moore]

If you are not using multipath and redundant controllers, SATA drives are probably just fine. I have hundreds of them in the servers I manage at work and I like them. If you want a redundant path from the system to the drive, then you must use SAS drives, but the speed of the drives is a mechanical limit more than a matter of the interface type.

 

[jgreco]

Thanks guys. So, bottom line both 9211 or 9300 will work fine.

We expect so. The 9300 just isn't a common choice because it's much more expensive. Many of the posters here are enthusiast/hobbyist class users, so there's attractiveness in a $30 solution that is also a preferred solution. Those of us who do this professionally and are here are probably cheapskates, because if we had unlimited budget, we'd just buy some big commercial product... like TrueNAS.

Let's assume I get a 9211. Since 9211 does only 6Gbps, is there a real-world benefit for getting SAS drives vs SATA? I know the SCSI protocol is better, but in real-life scenarios does it make a difference?

Your hard drive WON'T break 3Gbps and probably only vaguely exceeds 1.5Gbps. You can run 24 drives off a four-lane SAS cable with only modest oversubscription in practice.

The SAS drives are generally somewhat better quality. SAS also has the opportunity to do multipathing, and you can run external cables to an external chassis for additional expansion. You don't need to use SAS drives for that, just a SAS expander. Head to head, a SAS vs a SATA drive of the same model family will feel similar. There are differentiators under stress, but you're better mitigating that at the ZFS level with L2ARC etc. If we do not require "same model family", then the SATA will almost always offer a larger capacity for an equivalent price.

As an example, rather than buying two 12TB SAS 7200rpm WD Ultrastar DC HC520 drives at around $554/each which are indeed sweet drives, some of us stocked up on WD Easystore 8TB SATA 5400rpm external drives for $130/each and shucked them. Four of these will run $520 and in mirrors would provide 16TB of space, at less than half the cost of the 12TB mirror. But you get no warranty. But you can get lots more of them. Shrug.

 

[George Kyriazis]

Thanks!

Talking about expanders.. While reading some articles, I came across references to RES2SV240 expanders. Here's what I don't understand however: In pictures (https://www.amazon.com/Intel-RAID-Expander-Card-RES2SV240/dp/B0042NLUVE) and in documentation (https://www.intel.com/content/dam/s...boards/server/sb/e93121003_res2sv240_hwug.pdf) it looks like you just plug it in to a pci-e slot and you go; ie. no HBA as a prerequisite. That kinda conflicts what I understand an expander to be (ie. there are wires coming in from an HBA and more wires going to to the drives). Can someone explain / elaborate on that?

Yes, I am planning on using L2ARC and ZIL, so probably not too much stress on the drives. I was thinking of getting 12TB Exos (https://www.amazon.com/Seagate-256MB-Cache-Enterprise-ST12000NM0007/dp/B0759Q9FXZ) @$365, which is cheaper than the $554 Ultrastar, but yes, more expensive than the $130 8TB one. Still, they are for work, so I'm not paying the bill.

 

[jgreco]

Thanks!

Talking about expanders.. While reading some articles, I came across references to RES2SV240 expanders. Here's what I don't understand however: In pictures (https://www.amazon.com/Intel-RAID-Expander-Card-RES2SV240/dp/B0042NLUVE) and in documentation (https://www.intel.com/content/dam/s...boards/server/sb/e93121003_res2sv240_hwug.pdf) it looks like you just plug it in to a pci-e slot and you go; ie. no HBA as a prerequisite. That kinda conflicts what I understand an expander to be (ie. there are wires coming in from an HBA and more wires going to to the drives). Can someone explain / elaborate on that?

Yes, I am planning on using L2ARC and ZIL, so probably not too much stress on the drives. I was thinking of getting 12TB Exos (https://www.amazon.com/Seagate-256MB-Cache-Enterprise-ST12000NM0007/dp/B0759Q9FXZ) @$365, which is cheaper than the $554 Ultrastar, but yes, more expensive than the $130 8TB one. Still, they are for work, so I'm not paying the bill. :)

Expanders are powered devices. You need to power them from somewhere. Many of them can be plugged into a PCIe slot and/or accept a "Molex" style power connection. The fact that it is plugged into a PCIe slot does not confer any additional features; it does not suddenly become a stealth HBA just because it's plugged into PCIe.

https://www.ixsystems.com/community/resources/dont-be-afraid-to-be-sas-sy.48/

 

[jgreco]

Ugh I just noticed that whoever ported that over to "Resources" screwed the images. Better to look at the original;

https://www.ixsystems.com/community...-sas-sy-a-primer-on-basic-sas-and-sata.26145/

 

[George Kyriazis]

ah, after more careful reading, I realized that the ports and both input/output, so that's how it connects to the HBA.

 

[blanchet]

For a work environment, a FreeNAS certified server from iXsystems may be preferable. You will avoid any compatibility issue.
For the disks, you should buy HGST Ultrastar DC HC500 Series (it is that iXsystems ships with their systems)

 

[George Kyriazis]

Thanks for the disk suggestion. Actually I was thinking of getting a combination of the HGST ones and the Exos ones. I was reading in other articles that it's best to mix-n-match to avoid hitting a bad batch.

 

[jgreco]

Thanks for the disk suggestion. Actually I was thinking of getting a combination of the HGST ones and the Exos ones. I was reading in other articles that it's best to mix-n-match to avoid hitting a bad batch.

Yes, I'm a big fan of heterogeneous pools. Not everyone is, but a lot of that seems to have origins in the RAID/SAN industry where they like everything to be matched for various reasons.

 

[George Kyriazis]

Any difference that you can think of between 9300 and 9305? I'm thinking here driver maturity, general flakiness, etc. Thanks! Would it be better to get a 9300-8i + expander or 9305-16i (no expander)?

 

[jgreco]

It's the same driver. It probably comes down to a matter of cost. Or, if you were ever going to populate the thing with all SSD's, maybe favor the -16i.