Recommended HBA to use for x2 HP StorageWorks MSA60, 28 HDDs, with a HP DL320G6 for FreeNAS 9.10?

[Isuress]

TL;DR - What HBA would you suggest for a HP DL320G6 that will be running FreeNAS 9.10, connected to x2 MSA60s, with x12 1TB drives (24 total), with room to expand to 2TB, 3TB, and 4TB drives in the future as well as adding potentially a 3rd and 4th MSA60?
- - - - -
I don't have that much experience when it comes to RAID/HBA cards.

These are the 4 cards that have been recommended to me by other members thus far:

* LSI Logic SAS9200-8E
* LSI MegaRAID SAS9266-4i
* LSI MegaRAID SAS9266-8i
* StorageWorks H221

Why would I choose 1 card over another?
What are the pros and cons of each card?

I'm going to be using the MSA60 with x12 drives. Initially starting with all 1TB drives, and then slowly upgrading them to 2TB and 3TB respectively. Potentially even 4TB considering I already have x4 DeskStar HGST 4TB connected directly via SATA ports on the DL320G6.

I have 2 MSA60s, 12 drives each, 24 drives in total. Plus x4 4TB drives that are connected to the DL320G6 backplane. 28 total.

I will likely be using SATA drives rather than SAS drives as this is a homelab storage solution. I've read that the MSA60 has a bottleneck with SATA (1.5Gb/s) drives over SAS (3Gb/s) but I've been told I can get passed that with the right HBA.
I would like to get as much performance out of the drives as possible.

Which of these cards would be able to handle potentially x28 HDDs, ranging from 1TB to 4TB, and have reasonable performance on the drives for read/write?

If there's another card that would fit the bill much better that isn't currently listed, what would you recommend?

- - - - -

ALSO, how would I go about hooking up these cards? The /r/homelab Discord has told me that one would be able to daisy chain the MSA60s together? Is this true? If I were to purchase a 3rd or 4th MSA60 in the future, would I be able to continue chaining them?
Here is a diagram describing the possible configurations

If I were to daisy chain them, could I potentially lose read/write performance? If so, I may avoid this method.
Regardless of the configuration, I will still need to purchase 2 more SFF-8088 cables.

I will also consider purchasing x2 of the suggested card if it would mean I would get maximum performance out of each MSA60 unit.

 

[Ericloewe]

StorageWorks H221

* LSI MegaRAID SAS9266-4i
* LSI MegaRAID SAS9266-8i

Who said that? They should have their forum card revoked. The LSIs don't work as HBAs and the HP thing doesn't even say what controller it uses.

* StorageWorks H221

Fine, might be expensive.

Have you read the hardware recommendations guide? It's linked in my sig.

 

[Isuress]

Who said that? They should have their forum card revoked. The LSIs don't work as HBAs and the HP thing doesn't even say what controller it uses.

Fine, might be expensive.

Have you read the hardware recommendations guide? It's linked in my sig.

Hey man, I appreciate you responding to both threads!

The LSIs were recommended by people in the original sister thread for the other ESXCLI question I had.
I think they recommended them because I'm using an HP ProLiant DL320 G6 server and some HPs are very picky about what parts you use with them. Otherwise their fan speeds SHOOT up and you have VERY loud servers for no reason. So typically if you use HP branded parts and accessories, you're less likely to trip an alarm somewhere.
From my understand, the majority of HP RAID cards are actually just rebranded LSIs, so when you purchase these cards rebranded, you flash them in IT mode to get an HBA or JBOD setup.
So yeah, those might be recommended due to HP's stupid requirements.


As for the StorageWorks H221, it was also suggested somewhere in that thread; I know nothing of it.
Chances are regardless of what card I choose, I would be purchasing them via Ebay to avoid the extreme cost associated with RAID equipment.
Some of those LSI's go for like $30-$40 from what I've seen thus far.
I just don't know which to use and why.


*EDIT* - about the read the hardware .pdf too.

 

[Ericloewe]

The LSIs were recommended by people in the original sister thread for the other ESXCLI question I had.

Oh God, not reddit. They're full of crap and then have the audacity of calling us forum regulars rude, aggressive and whatever (which is brilliant coming from such a toxic community as reddit).

Otherwise their fan speeds SHOOT up and you have VERY loud servers for no reason.

I've heard that one before.

From my understand, the majority of HP RAID cards are actually just rebranded LSIs, so when you purchase these cards rebranded, you flash them in IT mode to get an HBA or JBOD setup.

That might be the case, but you'd have to figure out exactly what controller they're using. LSI SAS2008, SAS2308 and SAS3008 are all fine.

So yeah, those might be recommended due to HP's stupid requirements.

The checks might require HP firmware, which complicates things. I'm not familiar with specifics, though.

 

[Isuress]

Oh God, not reddit. They're full of crap and then have the audacity of calling us forum regulars rude, aggressive and whatever (which is brilliant coming from such a toxic community as reddit).

I mean most people get defensive about whatever community they're part of the most. I use Reddit a fair bit and although they're not always the best resource, with some stuff people are very eager to help via the "up-vote" incentive. Similar to that of Stackoverflow's methodology.
Thus far, the /r/homelab community have been very helpful with some of my projects. Though /r/sysadmin is a bit more "elitist" I suppose.
It all comes down to people at the end of the day; it's just coincidence that they're in that particular community I suppose; but I digress.

I've heard that one before.

Yeah, it's a massive pain to consider. For example: my DL320 came with a RAID controller (wasn't the P411) but that didn't work with FreeNAS, so I took it out. I attempted to connect the chassis' backplane directly into the motherboard and I STILL got the fan speed thing. I had to call HP and specifically order a "Direct SATA to motherboard" accessory because their part came with 1 tiny extra wire for something. That's HP for you...

That might be the case, but you'd have to figure out exactly what controller they're using. LSI SAS2008, SAS2308 and SAS3008 are all fine.

Having briefly read the hardware recommendation PDF, I somewhat understand what you mean. I'm assuming you're referring to the chipsets used by the RAID controllers themselves? So as long as 1 of the 4 LSI cards I was recommended complies with one of those chipsets, I should be okay?

The checks might require HP firmware, which complicates things. I'm not familiar with specifics, though.

Yeah, as am I; which is why I decided to come to the forums for some help, lol.
I'm assuming that once I actually choose a card, finding the appropriate IT firmware might be easier?

Though that's what it comes down to; what card should I choose?
I don't know which, if any, of these fit the workload and compatibility that I'm looking for.
I thought more people on the forums who be familiar with this sort of setup as HP equipment is fairly common.
(Even though many FreeNAS boxes are typically whiteboxes rather than enterprise-grade)

 

[Ericloewe]

finding the appropriate IT firmware might be easier?

LSI's firmware is trivial to find.

So as long as 1 of the 4 LSI cards I was recommended complies with one of those chipsets, I should be okay?

The 9200 uses an SAS2008.

(Even though many FreeNAS boxes are typically whiteboxes rather than enterprise-grade)

They are enterprise-grade, just not by a traditional enterprise OEM like HP or Dell.

I don't know which, if any, of these fit the workload and compatibility that I'm looking for.

For FreeNAS, all supported controllers are going to be functionally the same.

 

[MatthewSteinhoff]

1. The MSA60 is 3 Gb/s (not 6 Gb/s) when used with SAS: something to keep in mind when you are slurping data from 12 drives. (When using SATA, it claims just 1.5 Gb/s.)

2. Make sure the MSA60 will support larger drives. It was born a decade ago and says it supports 'drives up to 750GB'. Not sure if there is a 2TB drive limit cooked into the device.

3. We have an HP D2700 hanging on an HP H221 HBA attached to an HP DL360 G5. The H221 HBA can either be an LSI 9205-8e or LSI 9207-8e chipset depending on when it was released; HP is sloppy with its versioning. If the HBA is way down-level, you have to do a two-step process to get it up to the most recent firmware. Other than that, the card is just as good as an LSI-branded card.

4. We use an LSI SAS9207-4i4e with the newer (6Bb/s) HP StorageWorks arrays, the D2600 and D2700. No problems at all.

5. There is a limit to the number of MSA60s you can daisy-chain off one controller. I want to say that number is three but that could just be a best practice and not the hard limit. In any case, you may want to get an HBA with more than one port or get more than one HBA. The DL320 has just two slots, right?

6. What is the most amount of memory you can throw in a G6 DL320? Will that be enough RAM to meet your disk space requirements? At 4 x 12 x 1TB, you're probably fine. At 4 x 12 x 4TB, you're probably not going to be able to fit enough RAM in the server. (Especially if you're using RAM cache to offset the slow drives.)

Cheers,
Matt

 

[MtK]

Check this: https://github.com/ewwhite/zfs-ha/wiki
At least for the diagram/setup, since freenas is no designed for it.

 

[Isuress]

Sorry I dropped off the earth for a few days! I got kinda busy unfortunately. I really appreciate the answers though!

The 9200 uses an SAS2008.

So if they're LSI SAS2008 or SAS2308 chipset, they're SAS2. But if it's SAS 3008, it's SAS3. Why would I choose a SAS2 chipset vs a SAS3 chipset? According to the hardware .pdf it would related to legacy HDDs? Or maybe a legacy backplane or something like that?

For FreeNAS, all supported controllers are going to be functionally the same.

They'll function the same as an HBA, but wouldn't each card have a throughput speed and a max speed associated with each card?
Isn't that something I should be thinking about considering I'm going to be having 28+ drive?
Or would they really all function exactly the same?

1. The MSA60 is 3 Gb/s (not 6 Gb/s) when used with SAS: something to keep in mind when you are slurping data from 12 drives.

See, I'm not sure about that though. I mentioned that in my OP as well, but I've had others in a Reddit thread suggest that it might be based on the HBA it's connected to? I don't really have anything conclusive though :(

2. Make sure the MSA60 will support larger drives. It was born a decade ago and says it supports 'drives up to 750GB'. Not sure if there is a 2TB drive limit cooked into the device.

This is also something else I'm not entirely sure of. The documentation suggests that it can't go over 1TB; but I've read posts from Google that suggest people are using 2-3TB drives as well? They mention using the latest backplane firmware (v2.28)? Maybe the newer firmware allows for bigger drives? Once again, it's all inconclusive though...

3. We have an HP D2700 hanging on an HP H221 HBA attached to an HP DL360 G5. The H221 HBA can either be an LSI 9205-8e or LSI 9207-8e chipset depending on when it was released; HP is sloppy with its versioning. If the HBA is way down-level, you have to do a two-step process to get it up to the most recent firmware. Other than that, the card is just as good as an LSI-branded card.

4. We use an LSI SAS9207-4i4e with the newer (6Bb/s) HP StorageWorks arrays, the D2600 and D2700. No problems at all.

The HP H221 would seem like the safest bet, I just don't know the limitations of the card.
I don't know ANY of the limitations of the cards suggested thus far. I'm not sure if there's a max drive size, or max drive amount associated with any of these HBAs or if they'll all act the same in the end.
What should I be looking for in an HBA? I'm assuming I need something that can handle 28+ drives and can do 1TB-4TB drives with decent read/write speeds? Is that a tall order is that something most HBAs can do no problem? Do they all work the same or should I be looking deeper?
With my current setup, would you have any suggestions for HBA? Or even which to use over another?

5. There is a limit to the number of MSA60s you can daisy-chain off one controller. I want to say that number is three but that could just be a best practice and not the hard limit. In any case, you may want to get an HBA with more than one port or get more than one HBA. The DL320 has just two slots, right?

According to someone's post in the /r/sysadmin thread, it seems you can do a maximum of 4 daisy chain but that may or may-not be associated with the HBA you choose to use?
Yes! No matter which card I end up choosing, I would prefer to at LEAST get a 2 port SAS just in case.
Yeah, the DL320 G6 has a riser card for 2 PCIe cards. 1 can be full bracket while the other has to be low profile.
That said, both slots are quite close to the chassis top-cover so I would hope the card that I get fits without it being too snug.

6. What is the most amount of memory you can throw in a G6 DL320? Will that be enough RAM to meet your disk space requirements? At 4 x 12 x 1TB, you're probably fine. At 4 x 12 x 4TB, you're probably not going to be able to fit enough RAM in the server. (Especially if you're using RAM cache to offset the slow drives.)

The DL320 G6 has 9 DIMM slots. I don't remember how many are currently populated but I think I have about 16gigs of ECC in there at the moment. I have some more ECC laying around so I could potentially put more - assuming I don't run out of DIMMs.
From what I've read, FreeNAS suggest 1gb RAM per 1TB? Or something close to that.
I have x4 4TB drives, and x7 1TB in MSA60 A, and x7 1TB in MSA60 B; currently.
So would that roughly be 30GB of RAM? Would I HAVE to put that much or is it just a preferred-ish amount?

Check this: https://github.com/ewwhite/zfs-ha/wiki At least for the diagram/setup, since freenas is no designed for it.

Holy crap that's intensive. It looks like he made his own ZFS setup rather than using FreeNAS. I think you linked it though because he's using HP equipment? From the looks of it though, it seems like he's using the StorageWorks H221 controller that I mentioned in OP. That helps with the decision; I just need to know more about the limitations of this card verses the others before I make the final purchase.


I'm sorry this was so huge guys! I thought it would be better to response in 1 big post rather than a bunch of small ones.
Sorry again for the delayed response.

 

[MtK]

Holy crap that's intensive. It looks like he made his own ZFS setup rather than using FreeNAS. I think you linked it though because he's using HP equipment? From the looks of it though, it seems like he's using the StorageWorks H221 controller that I mentioned in OP. That helps with the decision; I just need to know more about the limitations of this card verses the others before I make the final purchase.

It's not his own ZFS, since ZFS is one. And it's the same FreeNAS is using, just on a different Operation System.

I linked it, because you might want to consider not using FreeNAS and use a similar structure like he did. Mainly because FreeNAS does not and was not design to support this kind of a layout even though it's using the same ZFS.

 

[Ericloewe]

Why would I choose a SAS2 chipset vs a SAS3 chipset?

More stable, more compatible, more thoroughly tested.

They'll function the same as an HBA, but wouldn't each card have a throughput speed and a max speed associated with each card?
Isn't that something I should be thinking about considering I'm going to be having 28+ drive?
Or would they really all function exactly the same?

For the same controller, any card will be exactly the same. Naturally, the 2308 is faster than the 2008 and has more connectivity, so it might be faster. You'd never notice, I'm sure.

 

[MatthewSteinhoff]

Why would I choose a SAS2 chipset vs a SAS3 chipset?

What is your use case? What level of performance do you require?

The MSA60 is a SAS1 disk array according to the HP QuickSpecs. If you hook an SAS2 or SAS3 controller to the MSA60, it's still only going to talk at 3 Gb/s as far as I can tell.

A year or so ago, HP locked up their firmware. You can't just go to their web site and get updates. So, even if you could upgrade the MSA60 backplane from SAS1 to SAS2 or there was a firmware update to use drives larger than 2TB, you should know if you can get that firmware in advance of spending any money.

What should I be looking for in an HBA? I'm assuming I need something that can handle 28+ drives and can do 1TB-4TB drives with decent read/write speeds? Is that a tall order is that something most HBAs can do no problem?

Pretty much any LSI-chipped card (including the HP H221) will support more drives than anyone would actually attach - upwards of 256, some 512.

Once again, you really need to look at your use case. What is your parallel load? Even the slowest HBA with hundreds of drives will work fine if you're just slowing filling drives with backup data. On the other hand, if you're storing active databases, streaming lots video, hosting active virtual machines or any one of a dozen other load cases, you really have to start examining what hardware you're throwing at the load.

SAS2 is pretty much the sweet spot... good performance, good reliability, good price. SAS1 isn't much a of a cost savings and you take a hit in performance and universal compatibility. SAS3 is super fast, cutting edge but you pay dearly for the speed and you're looking at technology with just millions of hours of bug testing instead of billions.

So would that roughly be 30GB of RAM?

You really gotta look at your use case.

For us, 40TB of disk was stable and fast enough with 32GB of RAM. When we built our new box, we ended up with 28TB of disk space and 96GB of RAM which is insanely fast (for our use case).

Cheers,
Matt

 

[Isuress]

I linked it, because you might want to consider not using FreeNAS and use a similar structure like he did. Mainly because FreeNAS does not and was not design to support this kind of a layout even though it's using the same ZFS.

Yeah, the setup I'm going for is very disk heavy; probably more than most, haha. But you don't think FreeNAS is made to be used in such a setup? My understanding is that TrueNAS is basically FreeNAS, just made for enterprise. The back-end is the same, no? If they are, I would assume their enterprise line of devices would facilitate a 28+ drive setup easily, no? From just skimming the article you sent, it seems like it's WAY out of my level of knowledge. I don't have that much of a background with Linux and it seemed somewhat command-line heavy. I definitely like that idea but I don't think it's within my realm.

That said, FreeNAS 10 looks really sexy... I don't know if it actually adds any major functionality though. (somewhat off-topic, lol)
I think it's mostly just cosmetic?

For the same controller, any card will be exactly the same. Naturally, the 2308 is faster than the 2008 and has more connectivity, so it might be faster. You'd never notice, I'm sure.

Are you suggesting that I'll be limited to my DL320's onboard controller verses the HBA? Which is why it doesn't really matter which card I end up choosing because I wouldn't be able to see its full potential anyway?

A year or so ago, HP locked up their firmware. You can't just go to their web site and get updates. So, even if you could upgrade the MSA60 backplane from SAS1 to SAS2 or there was a firmware update to use drives larger than 2TB, you should know if you can get that firmware in advance of spending any money.

Yeah, I noticed that a year and a half ago when I first bought the DL320s and wanted to update them to the latest firmware. It's really shitty when companies put their stuff behind paywalls. Especially things like drivers and firmware... Dell's EqualLogic is also like that. Though Lenovo isn't so I may be considering their stuff in the future, haha.
Anyway, with that said; there IS a way to download those files that are behind the paywall!
You see, when you try to download the file - it'll say that you need a "support contract". BUT! They give you the file name AND the MD5 hash.
The HP FTP website is completely open to the public for whatever reason, lol. Probably a massive security hole but you can get in without paying.
So all you need to do is search their FTP (Which is a godawful mess mind you) for the filename, and then check the MD5 once you downloaded it to make sure it's the right file!
I've done this to get the latest HP SPP 2016 ISO and the firmware I needed for my DL320, haha. Nuts to them! :p
But yeah, coincidentally their firmware for the MSA60 (v2.28) says that you need a support contract to download it, but when you click continue - it just downloads it anyway. I didn't even have to do the FTP trick, lol. So yeah, both of my MSA60's backplanes are fully updated to v2.28 . Now whether or not that actually makes a difference, I don't know, haha. Hopefully!

Pretty much any LSI-chipped card (including the HP H221) will support more drives than anyone would actually attach - upwards of 256, some 512.

Oh wow

Once again, you really need to look at your use case. What is your parallel load? Even the slowest HBA with hundreds of drives will work fine if you're just slowing filling drives with backup data. On the other hand, if you're storing active databases, streaming lots video, hosting active virtual machines or any one of a dozen other load cases, you really have to start examining what hardware you're throwing at the load.

I would say 80% of the time my FreeNAS will probably be sitting idle; just storing my files? This is for a homelab setup.
Though with how much space I have, I would definitely consider using it as a Datastore for my ESXi hosts.
I do have a Dell EqualLogic PS6000E (with x16 drives) waiting to be setup and initialized for my ESXi's but it would probably be a good idea to have more than one iSCSI target to backup to?

I would say FreeNAS will mainly be for file storage, media storage, snapshotting, and maybe a small Datastore?
So maybe a midranged HBA solution would be advantageous? Some safe headroom assuming I want to expand later on?

SAS2 is pretty much the sweet spot... good performance, good reliability, good price. SAS1 isn't much a of a cost savings and you take a hit in performance and universal compatibility. SAS3 is super fast, cutting edge but you pay dearly for the speed and you're looking at technology with just millions of hours of bug testing instead of billions.

I suppose I should just stick with a SAS2 card then. I don't want to use SAS1 and you're making SAS3 sound like a bad choice, haha.
What would be a good SAS2 HBA with at least 2 ports that works for my setup? (the considerations in the quote above)

For us, 40TB of disk was stable and fast enough with 32GB of RAM. When we built our new box, we ended up with 28TB of disk space and 96GB of RAM which is insanely fast (for our use case).

So FreeNAS will still work with 16GB of RAM at 30TB, it just might not be fast? Would that be my main concern then?
If I have more HDD than I do RAM, that would also make things like dedup out of the question, right?
My understanding is that dedup works heavily on RAM? Which would mean I would ideally need 30GB of RAM for 30TB, but probably another 10GB of RAM just for dedup?

 

[depasseg]

Why would I choose a SAS2 chipset vs a SAS3 chipset?

SAS2 stuff is cheaper, much broader user base, more mature/less interoperability issues, and unless you are running all-flash disks it has more than enough throughput.

 

[Ericloewe]

But you don't think FreeNAS is made to be used in such a setup?

Of course it is. That's where FreeNAS and ZFS shine - where you have so much storage that the added cost for higher-end components is completely irrelevant to nonexistent.

That said, FreeNAS 10 looks really sexy... I don't know if it actually adds any major functionality though.

Well, a not-insane GUI seems like it counts as functionality. As for hard features, there are a few that won't make their way to 9.x - the file browser in the GUI for CLI-averse folk, GUI logins with users other than root and the ability to connect to the server remotely for administration without using the GUI (basically, FreeNAS exposes an API that apps can use for server administration - that's how the new interfaces work).

Yeah, I noticed that a year and a half ago when I first bought the DL320s and wanted to update them to the latest firmware. It's really crapty when companies put their stuff behind paywalls. Especially things like drivers and firmware... Dell's EqualLogic is also like that. Though Lenovo isn't so I may be considering their stuff in the future, haha.
Anyway, with that said; there IS a way to download those files that are behind the paywall!
You see, when you try to download the file - it'll say that you need a "support contract". BUT! They give you the file name AND the MD5 hash.
The HP FTP website is completely open to the public for whatever reason, lol. Probably a massive security hole but you can get in without paying.
So all you need to do is search their FTP (Which is a godawful mess mind you) for the filename, and then check the MD5 once you downloaded it to make sure it's the right file!
I've done this to get the latest HP SPP 2016 ISO and the firmware I needed for my DL320, haha. Nuts to them! :p
But yeah, coincidentally their firmware for the MSA60 (v2.28) says that you need a support contract to download it, but when you click continue - it just downloads it anyway. I didn't even have to do the FTP trick, lol. So yeah, both of my MSA60's backplanes are fully updated to v2.28 . Now whether or not that actually makes a difference, I don't know, haha. Hopefully!

Why am I not surprised?

So FreeNAS will still work with 16GB of RAM at 30TB, it just might not be fast?

Yeah.

If I have more HDD than I do RAM, that would also make things like dedup out of the question, right?

Dedup is really always out of the question. Maybe once the "add more RAM or the pool won't import" feature gets fixed.

My understanding is that dedup works heavily on RAM? Which would mean I would ideally need 30GB of RAM for 30TB, but probably another 10GB of RAM just for dedup?

No, your numbers are off... by an order of magnitude. Think 150GB of RAM recommended for 30TB of dedup.

 

[Isuress]

SAS2 stuff is cheaper, much broader user base, more mature/less interoperability issues, and unless you are running all-flash disks it has more than enough throughput.

Makes sense I suppose. I decided earlier that I would probably just go the SAS2 route. It just comes down to which card I actually end up going with.

No, your numbers are off... by an order of magnitude. Think 150GB of RAM recommended for 30TB of dedup.

Jesus Christ! 150GB of RAM for 30TB!? That's insane! How is that even possible?
I was looking at some QNAP, and Synology stuff; I also went to a NimbleStorage training course last Thursday. All of their storage appliances provided dedupe but none of the devices had anything close to 150gb of RAM! How is it possible that they can provide dedupe with only like 10-20gb of RAM while FreeNAS requires an absolutely exuberant amount of RAM?

Cheers, Matt

Any input on the previous postings by chance?


Aside from this HP ProLiant DL320 G6 project, I'm in the processes of upgrading our FreeNAS storage at work.
I'm finally switching the equipment from a tower to a rackmount case and adding PSU redundancy and 8 hotswap bays verses the 2 internal bays the tower had.
The SuperMicro chassis that I've chosen has a backplane which connects each individual HDD to the motherboard.
The motherboard currently in use only has 2 SATA3 ports and 4 SATA2 ports. Mixing SATA speeds will probably make my drives much slower and also I wouldn't have enough SATA ports for all 8 hotswap bays.
In a case like this, what would be a good LSI card to use for this purpose?
I've gone ahead and read Cyberjock's ECC RAM post, as well as JGreco's posts about burn-in and SAS drives.
Both gentlemen mention the IBM M1015 as the most popular RAID card to use. It's also mentioned in Ericloewe's Hardware Recommendation guide.
Would that give me enough capacity, and speed to handle the 8 drives in the SuperMicro 8 bay chassis?
The 9211 and 9207 are also mentioned fairly frequently and supposedly work out of the box without having to be flashed?
Which of the 3 cards would fit the best use-case for the production FreeNAS?
Regardless of which of the 3 I purchase, would I still need to purchase a "SAS to 8-port SATA cable" to plug in all of the hotswaps on the backplane? Or is there a different method of doing so?

Remember, SuperMicro chassis is an internal backplane with 8 SATA drives.
While the DL320 is being connected to external DAS via SFF-8088 cables.

I'm still not sure what card I should go for with regards to the DL320 build. My understanding is that it may not matter which card I choose as they'll all work the same?
Though this doesn't make sense to me as I would think that the cards would have different max-drive capacities and speeds? Especially if the DL320 is going to be doing a 28+ drive setup. I'm just looking for something that'll fit what I need for 28 drives with potential headroom for when I attach a 3rd DAS with another 12 drives.

I'm sorry if these are getting long again.

 

[Ericloewe]

How is it possible that they can provide dedupe with only like 10-20gb of RAM while FreeNAS requires an absolutely exuberant amount of RAM?

It works differently, basically. In ZFS, everything has to be atomic, which means, in this case, that you have to load a crapton of data into RAM or panic the kernel.
The good news is that dedup isn't particularly useful in the vast majority of cases. If it is in your case, a client-side solution would work.

The 9211 and 9207 are also mentioned fairly frequently and supposedly work out of the box without having to be flashed?

No crossflash required, but you need to flash the correct version.

Would that give me enough capacity, and speed to handle the 8 drives in the SuperMicro 8 bay chassis?
The 9211 and 9207 are also mentioned fairly frequently and supposedly work out of the box without having to be flashed?
Which of the 3 cards would fit the best use-case for the production FreeNAS?
Regardless of which of the 3 I purchase, would I still need to purchase a "SAS to 8-port SATA cable" to plug in all of the hotswaps on the backplane? Or is there a different method of doing so?

Remember, SuperMicro chassis is an internal backplane with 8 SATA drives.
While the DL320 is being connected to external DAS via SFF-8088 cables.

Reply to this thread to remind me to take a look at this when I get home.

 

[MatthewSteinhoff]

I would think that the cards would have different max-drive capacities and speeds? Especially if the DL320 is going to be doing a 28+ drive setup.

You are moving the contents of a studio apartment. You're asking if there is a difference in the carrying capacity of a Peterbilt, Kenworth or Freightliner tractor-trailer. Yeah. Maybe. Sure. Probably. But, for your apartment, any one of those will have greatly more capacity than you require to move across the state.

Any supported SAS2 HBA will perform more or less equally and support the number of drives you plan on using. All will likely support more than 120 drives, four times the number you plan on using. All meet the SAS2 standard, the bandwidth is the same, the throughput is same.

While there is likely a performance difference between the absolute best HBA and the absolute worst HBA but you'll never notice. All the LSI HBA SAS2 cards (and their alternately-branded cousins) are going to perform within a hair of each other and none should be taxed by the configuration you describe.

No matter what HBA you purchase, you'll likely need to flash the firmware to bring it up to the recommended level. That's typically not a problem so long as you buy any one of the dozen HBAs typically recommended.

If you needed to host 288 drives, I'd suggest you really dig into the HBA's specs. Some stop at 256 while others support 512 or 1024. But, at 30 drives, you're way under any common limit.

Cheers,
Matt

 

[Isuress]

It works differently, basically. In ZFS, everything has to be atomic, which means, in this case, that you have to load a crapton of data into RAM or panic the kernel.
The good news is that dedup isn't particularly useful in the vast majority of cases. If it is in your case, a client-side solution would work.

Yeah, for mine in particular I don't think I necessarily need dedupe (for work) but I was considering it for my homelab. Though seeing as though it takes WAY too much RAM, I suppose I won't, lol.

No crossflash required, but you need to flash the correct version.

Is there a thread that has ALL the tutorial links for each card type? Or do I have to search/Google the specific card and get X firmware, etc, etc?

Reply to this thread to remind me to take a look at this when I get home.

Reminder ~

Any supported SAS2 HBA will perform more or less equally and support the number of drives you plan on using. All will likely support more than 120 drives, four times the number you plan on using. All meet the SAS2 standard, the bandwidth is the same, the throughput is same.

If that's the case, why is there different options? I would've thought choice would suggest that there was a difference for the cards.
Does that mean all the cards will also support HDDs ranging from 1TB to 4TB? All the HBAs work with 4TB?

Anyway;
For the Production FreeNAS

• x1 - LSI Internal SATA/SAS 9211-8i
• x2 - SFF-8087 to 4x Latching SATA (8 total SATAs)

Considering this is for production, would it be a better idea to buy the 9211-8i brand new off of Newegg or our vendor's website rather than buying it off of EBay? Considering the card is being flashed with different firmware, wouldn't that void any warranty I would be getting anyway? Would EBay be fine?

For the Homelab FreeNAS

• x1 - HP Smart Array P800 OR HP H221 OR LSI 9205-8e OR LSI 9207-8e
  
• x2 - SFF-8088 to SFF-8088 cables

The HBA card I would buy from EBay as I don't think it matters in a non-production setup.
That said I still don't know if the non-HP HBAs would cause my DL320G6's fans to become jet engines (HP doesn't like you using non-HP stuff). Do any of you have any of these cards in an HP machine and can confirm they don't destroy your fans?
But I'm not sure which of the HBA to go for. It seems like I need the 'E' edition of whatever card I go with, as to ensure I have 1-2 external SFF-8088 ports.
The forum post that MattheSteinhoff linked for the H221 seems to be easy enough.
Though it seems that the H221 goes for $120+ on EBay, plus like $20-$50 shipping to Canada.
I was hoping for something in the $50-80 total as I still need to buy $60~ worth of SFF cables :(

What do you guys think? Is that adequate? I'll wait till tomorrow to put all the purchases through so I can get confirmation with you guys.

 

[Ericloewe]

Is there a thread that has ALL the tutorial links for each card type?

The instructions are the same for all but the weird cards (the Dells and stuff). Firmware is naturally dependant on the controller, but is easily acquired from LSI's site by going to the card one wants to end up with, so to say.

If that's the case, why is there different options?

• Different generations (The SAS2308 adds PCI-e 3.0 support as well as support for the new software stack that is supposed to replace decrepit mfi).
• Different feature sets - features that get blown away once the card is crossflashed, so they're irrelevant for FreeNAS.
• Some have different layouts, with the ports being placed either at the end or at the outside edge - besides internal/external combinations.

The SuperMicro chassis that I've chosen has a backplane which connects each individual HDD to the motherboard.

Don't do that. Get the SFF-8087 backplane, since it massively reduces your cable clutter. Or the expander backplane for even fewer cables (but higher cost).

Mixing SATA speeds will probably make my drives much slower

Not at all.

In a case like this, what would be a good LSI card to use for this purpose?

Of course. They're also the only good option in any case.