Would anyone kindly recommend a Non-Raid PCIe 3.0 card for 4 and/or 8 SATA III ports?

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
It'll take me time to process all this input and to see where and how my misunderstandings have repercussed along the way. That probably makes no sense. I'm feeling a bit fried..also after spending much time reading (or blinking at) the crossflash guide, and between Rufus and bootice trying to flash this HP H220 (2308-2) card.

Don't feel bad. Here in the shop I crossflash cards in batches when I can because the workflow can be somewhat annoying. Working with these things professionally, it sometimes seems like my time is entirely devoted to waiting for BIOS's to probe, waiting to hit the right magic key during startup, and doing trite and meaningless firmware updates just to make sure things are all correct. You can sometimes spend an entire day and feel like you've made no real progress.

@jgreco -apologies for contributing to the spread of inaccurate information, and thank you for the detailed and enlightening corrections.

Really, not that. You came with what was basically an observation that works out to being more-or-less functionally correct, and wondered how it applied to SAS. I don't mind wading into a nuanced situation and explaining. I'm just sitting here bored waiting for VM builds to progress.

It's entertainment to me, educational (possibly needlessly) to you.

As far as why I want what I want..obviously I'm far from the professional realms of many in this forum. I guess I've been coy about that, which is annoying, but -OK, I have no good earthly purpose in that, on some level, I'm just goofing around with my PC, and learning things as I go, but only within a relatively limited capacity for grasping all the concepts. If that discourages anyone from further replies, I certainly understand.

I think you'll find most of the users here are NOT professionals. There are a whole lot of them who do FreeNAS just because "they can" or "it's there." This forum has numerous resources that exist to help drag you through most of the challenging bits. There's usually someone who can answer a question that isn't already covered somewhere.
 

uoR

Dabbler
Joined
Feb 23, 2019
Messages
11
@jgreco very kind, thanks!

@Chris Moore Just to clarify, for me and the thread, I don't see any difference between the 9205-8i HP H220 that I bought and the one you most recently linked to -except the one I bought was advertised as IT Mode "P20" (but came as P15 - see post 13 of this thread).

1) I'm now looking at this one: LSI SAS9300-8i full profile 12Gbps SAS HBA P16 IT mode for ZFS FreeNAS unRAID
2) Question: @Chris Moore Is it safe for me to connect some SSD drives to this P15 firmwared 9205-8i to test things out? In post 9 you said
Having older firmware can cause odd errors and reduce performance.
. I don't mind errors so long as I'm not risking data or other corruption on drives that are NOT connected to the 9205-8i. I <think> the P15 is actually already a crossflashed firmware because the card's boot screen devices list, or POST, says "Vendor Name: LSI".
3) Would it be appropriate to edit this topic title to say "..for Windows 10 x64 desktop" ?
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
@Chris Moore Just to clarify, for me and the thread, I don't see any difference between the 9205-8i HP H220 that I bought and the one you most recently linked to -except the one I bought was advertised as IT Mode "P20" (but came as P15 - see post 13 of this thread).
Some vendors are not good about updating the firmware before selling. One card comes with cables and a P20 firmware and the other is advertised as a P15 firmware and no cables. Upgrading the firmware is a pain. Other than that, look, cables.
No point spending the extra money for 12Gb SAS if you are running mechanical disks . The disks are not fast enough to matter. If the server comes with it integrated, that is one thing, but no reason to spend extra for it.
2) Question: @Chris Moore Is it safe for me to connect some SSD drives to this P15 firmwared 9205-8i to test things out? In post 9 you said
No. I wouldn't even try to use the P15 firmware in FreeNAS. That needs to be upgraded to the latest version which should be 20.00.07.00. If it is already in IT mode, it is easily updated from within the FreeNAS OS because the sas2flash program is included.
I don't mind errors so long as I'm not risking data or other corruption on drives
Errors, in this context, can mean data loss and failed drives. It is far better to just update the firmware.
Would it be appropriate to edit this topic title to say "..for Windows 10 x64 desktop" ?
Even if you are only using this on Windows 10, the same applies. There were good reasons for the firmware to be updated. Reasons that involve the proper communication between the OS and the disk.
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
@Chris Moore , curious question about 12Gb SAS: in your experience, how many mechanical drives in an external enclosure off a SAS extender does it take before the 12Gb becomes saturated? Let’s assume 7200 rpm and 4-8TB per drive.

Edit: that was a silly question. Should have asked “how many vdevs” I suppose
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
@Chris Moore , curious question about 12Gb SAS: in your experience, how many mechanical drives in an external enclosure off a SAS extender does it take before the 12Gb becomes saturated? Let’s assume 7200 rpm and 4-8TB per drive.

Edit: that was a silly question. Should have asked “how many vdevs” I suppose
It isn't that it is a silly question, but it is a complicated one. A 12Gb SAS lane (theoretically) allows for 2400MB/s and there is always overhead, so you can't use all of that. A SAS controller usually (not always) has 8 SAS lanes, so you could (theoretically) have 19200MB/s of data running through.

The fastest mechanical drive that I am aware of is rated by the manufacturer at 250MB/s, so we could only manage about 76 drives if they were all running full speed, but that is not usually what a drive does. Most drives I have monitored in a production system are poking along at around 100MB/s because of the time they spend looking for the next block in random IO and other housekeeping. That more than doubles the potential number of drives to around 190 before the actual speed of the drives becomes an issue for the controller. Lucky thing is, you can have multiple controllers. Another department where I work has a server with over 300 drives attached and it uses four controllers with mulipath configured. That system is one of the infrastructure system for our organization and serves over 1200 client systems.

This is based on my observations in the environment where I work.
 
Joined
May 10, 2017
Messages
838
A 12Gb SAS lane (theoretically) allows for 2400MB/s and there is always overhead,

I'm guessing a misprint by @Chris Moore, a 12GB SAS lane is good for 1200MB/s theoretical max, usually around 1100MB/s usable, so that gives you roughly 4400MB/s usable if single linked to a SAS3 expander, or 6000/6500MB/s using dual link, this assuming a PCIe x8 HBA like the 9300-8i, where the PCIe 3.0 slot will be the bottleneck.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I'm guessing a misprint by @Chris Moore, a 12GB SAS lane is good for 1200MB/s theoretical max, usually around 1100MB/s usable, so that gives you roughly 4400MB/s usable if single linked to a SAS3 expander, or 6000/6500MB/s using dual link, this assuming a PCIe x8 HBA like the 9300-8i, where the PCIe 3.0 slot will be the bottleneck.
Probably a math error on my part. I went looking for your previous post on the subject and couldn't find it. I recall you explaining it quite well at least one time before.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Hi, any recommendations for a < $100 card for freenas, for use in a T20, currently with 4 x 4TB wd reds? Goal is to add an ssd cache, and a 5th or 6th drive in the case (with fans). Thanks. Am sort of looking at this one: https://www.ebay.com/itm/HP-H220-6G...9205-8i-P20-IT-Mode-From-US-Ship/192639052923
I have four of those cards myself and they are great. Just double check the firmware version when you get it. The ones I ordered came in with an old firmware and needed to be updated.
This guy reliably delivers the correct version of the firmware:
https://www.ebay.com/itm/HP-H220-6G...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162862201664
 

tedm

Cadet
Joined
Jun 23, 2019
Messages
6
Ordered one. Now need to get some cables, and one or two more WD reds. What size and quanty of SSD caches would be recommended? Amt of ECC RAM inserver is 12GB, should this be increased as well? Currently, the system will saturate 1Gbps, but next plan is to move to 10Gbps maybe later this year.

I have four of those cards myself and they are great. Just double check the firmware version when you get it. The ones I ordered came in with an old firmware and needed to be updated.
This guy reliably delivers the correct version of the firmware:
https://www.ebay.com/itm/HP-H220-6G...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162862201664
I have four of those cards myself and they are great. Just double check the firmware version when you get it. The ones I ordered came in with an old firmware and needed to be updated.
This guy reliably delivers the correct version of the firmware:
https://www.ebay.com/itm/HP-H220-6G...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162862201664
 

uoR

Dabbler
Joined
Feb 23, 2019
Messages
11
@tedem - Be sure to search around the forum about SSD caches...seems I read some debate as to its questionable benefits.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
@tedem - Be sure to search around the forum about SSD caches...

I guess we're to assume you mean L2ARC and/or SLOG.

When you're talking in a technical forum, precision is helpful. We have a terminology primer available to help avoid ambiguity.

seems I read some debate as to its questionable benefits.

Questionable benefits?!?!?

No, the benefits are unquestionable. However, end user understanding of what it is and what you're going to get from it is often questionable.

For those that think that a SLOG is *any* type of "SSD cache", you're entirely wrong. Start here for enlightenment.

For those that think that L2ARC is going to be awesome, it isn't going to be awesome on your home media fileserver. If your pool is running at very low levels of activity, it isn't going to be awesome. If your system has less than 64GB, L2ARC isn't likely to work as well, because ZFS lacks enough ARC to identify your data hot spots. If, however, you are running a departmental fileserver with extremely busy disks, L2ARC is going to be a great thing. If you are running VM storage, L2ARC can make your read speeds feel like SSD even on a highly fragmented pool. Fortunately, flash prices have dropped enough that even if you make a mistake and add L2ARC onto a system that doesn't benefit from it, it probably isn't a huge financial blunder.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
As @jgreco points out, the use case for L2ARC and SLOG should be guiding you, not a blind assumption that "add SSDs = good".

FWIW, my largely static NAS benefitted tremendously in limited testing from adding a "metadata only" L2ARC. But my use case (repeated rsync of a lot of unchanging data for backup purposes) may not be representative of your workflow requirements. But the improvement was pretty dramatic and consistent once the L2ARC cache got "hot" through repeated use of rsync. Disk directory browsing was also improved a lot.

But to @jgreco's point, I was running my former C2750D4i motherboard with 64GB of RAM and my present motherboard clocks in at 2x of that, so I don't hurt system performance by starving the system re: RAM by having a L2ARC. I only recently upgraded to a 1TB L2ARC because my "hot" L2ARC maxed out at about 250GB and my pool isn't that full yet.

There are lots of threads debating the benefits of a SLOG and I'd do the research, then try to find a thread with data and a use case as close to yours as possible. Collectively, we've done some benchmarking here.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
FWIW, my largely static NAS benefitted tremendously in limited testing from adding a "metadata only" L2ARC. But my use case (repeated rsync of a lot of unchanging data for backup purposes) may not be representative of your workflow requirements. But the improvement was pretty dramatic and consistent once the L2ARC cache got "hot" through repeated use of rsync. Disk directory browsing was also improved a lot.

That's basically a variation on a VM virtual disk server, database or other lots-of-small-access model. Without the L2ARC, it's always going to be pounding lots of small random I/O at the pool. Because there are so many creative things people can be doing, it's always worth outlining the specifics of what you're trying to accomplish, and someone like @Constantin will wander along and give useful feedback on that.

As @Constantin notes, the benefit of L2ARC is very clear with a workload that is able to take advantage of it. The right ratio of RAM and SSD to your working set can yield a system that rarely reads from the actual pool, and will be amazingly faster. On the other hand, adding unnecessary RAM or SSD to your system can result in significant disappointment and thinner wallet.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Every use case has to be looked at individually. FreeNAS is incredibly flexible re: setups and while there are some standard “start here” configurations to try out, nothing really beats testing use cases with representative data, workflows, etc. Lots of details can have a big impact in the right context, especially for users like me that come new to FreeNAS and consider it a steep learning experience.

Hence, what makes this forum so valuable is the deep insights and knowledge the Demi-gods roaming the halls here bring to the discussion, a level of understanding that only prolonged experience can bring. All the group-sourced content adds immense context to the manual, resource pages, etc. for example.

So often, I run across gems of information that I can pack away for later use while I go off and search for my toupee having attempted to follow the details of the discussion and realized instead how little i actually know about FreeNAS, FreeBSD, or whatever. (Ie it was over my head). In that spirit I try to reciprocate where I can with information that i hope will be helpful to others facing similar challenges.
 
Top