1) An HBA is a Host Bus Adapter. I'm not using an HBA.
Bad.
4) FreeBSD may or may not have good support for other HBA's/RAID controllers. The ciss driver was introduced (I think) in FreeBSD 10.3, which was released in 2014, so it's got 7 years of availability.
Incorrect history of the CISS driver.
CISS was introduced in FreeBSD 4.8R, released April 2003. Allowing some liberty to keep things simple, CISS was an idea that host-side interfacing for RAID controllers could be standardized, and hoping that additional cards would jump on that bandwagon.
The bulk of this bullet though is about high loads placed on the HBA by ZFS, and there won't be any in my config and use case (see #6)
Thank you for explaining to us what you think the bulk of the bullet is about. In my opinion, you're wrong. The bulk of the bullet is about "
FreeBSD may or may not have good support for other HBA's/RAID controllers." As author of that line, I think I get to define what my bullets are actually about. And these guys here on the forum have similar experiences or have been listening to me kvetch on about this stuff for years, so many of them have a pretty good idea about it too.
I have spent years working with these things, not necessarily in a ZFS context. I have a bunch of Proliants that have CISS-based controllers in them and were used for years in high traffic contexts. The CISS stuff isn't horrible if you use the controllers for RAID1 or RAID5; they cleanly hide problems with failed sectors and if you can afford a bay for a warm spare, will swap out a failed disk. However, array I/O may still lock up while the controller is waiting for a disk to time out. More or less "the expected stuff."
Because in the USENET business we normally did redundant servers rather than relying on on-host RAID, and the storage needs were immense, we did NOT use these in RAID1/5, but rather in a JBOD configuration. And there, they sucked. They often hid what the exact failure was behind the scenes, would return zero-filled blocks without reporting an actual error when a sector failed to read, and had a variety of other foibles once things started to go wrong. Foibles that made it hard to spot things going wrong, and hard to identify WHAT was going wrong.
It is actually that which shares a lot in common with ZFS, because ZFS uses drives in a similar manner.
I don't disagree with the content, but I don't see how it applies to me. Quite comfortable being wrong though.
I'll concede the quoted article doesn't specifically discuss your RAID5 case, because the quoted article is primarily trying to scare people off using RAID card JBOD modes, but a lot of the article absolutely does apply to you and you are absolutely wrong.
The discussed bits include needing to use a driver that is known to be as close to 100% as possible (CISS isn't), just for example. That absolutely applies to you.
The undiscussed (i.e. your) problem is that ZFS relies on its redundancy to maintain the integrity of the pool. When ZFS encounters an error, let's say for example a block checksum fails, it rebuilds the block from redundancy, writes it back out, and then gets on with life, happy once again.
Having redundancy with ZFS is pretty much not optional -- it's required. If you do not have redundancy, a corruption introduced into the pool is uncorrectable. ZFS does not have tools like a "fsck" tool, which would require immense amounts of memory and pool I/O to do, so ZFS *relies* on its amazing ability to have a reliable and correct pool.
Once you introduce an error into the pool, that's bad. If it is merely file data, that's "recoverable" in that you can remove the file and the pool itself is not permanently damaged. However, if pool metadata gets hosed, that may well be unfixable, and at that point, your options quickly dwindle down to the dreaded "back up whatever you can and then create a fresh ZFS pool."
All these people here, myself included, are trying to help you get to a point where you have a more reliable system. Arguing to the contrary is pointless, except insofar as it educates you as to the realities, but that is already largely done in stickies so that you have a high quality answer to refer to.