Tuning ZFS for no disk write cache

[fa2k]

I have a non-recommeded config with single disk RAID0 devices on a hardware RAID controller, but I hope you can consider helping.

Is there any options I can tune when there's no write caches on the drive (or controller)?
I get only 30 to 40 MB/s write throughput per drive (simple local test with dd).
I don't have a good battery for the RAID controller at the moment, but if I force the RAID controller's cache, I get 170MB/s per drive. I'd prefer not to use a battery because it's another component to monitor and replace. [If I enable the physical drive's caches, I get 80-90 MB/s, but the RAID controller won't forward the cache flush commands, so that's out of the question]. I've done some testing with vfs.zfs.vdev.async_write_max_active, but it doesn't seem to change much.

I hope someone may have some tips, but if not, I have to try to find a battery, or bite the bullet and buy a proper HBA. I need to get something with PCI3 because I'll be adding SSDs too. I'm in the all too common position where I've stumbled upon some nice enterprise gear, but it's not a perfect fit for ZFS.

Show : See My freeNAS system

CPU: 2 x Xeon CPU E5-2697 v2
RAM: 128 GB DDR4 ECC
HBA: RAID controller MegaRaid 9280-16i4e; single disk RAID0 mode
DISKS 8 x ST1000NM0033 1TB SATA drives (to be replaced with 4 TB spinning SAS drives)

 

[Ericloewe]

I need to get something with PCI3 because I'll be adding SSDs too.

I'm not sure I follow. What's your reasoning?

 

[Patrick M. Hausen]

Replace that controller with an HBA and be happy ...

 

[fa2k]

I'm not sure I follow. What's your reasoning?

Sorry, I shouldn't really have thrown that comment in, it's a bit off topic. There are many cheap HBAs based on LSI2008, which has PCIe 2.0. I've read that they aren't well suited for SSDs. The PCI bandwidth is 2GB/s. I think the bandwidth could be a bit limiting for internal things like jails, VMs and resilver, even though I will only have a single 10 GbE connection to the network.

 

[Ericloewe]

Ah, I figured you meant NVMe SSDs.
In that case, just get something with an SAS2308. They should been only moderately more expensive. There’s also a lot of SAS3008 stuff out there by now.

 

[fa2k]

Ah, I figured you meant NVMe SSDs.
In that case, just get something with an SAS2308. They should been only moderately more expensive. There’s also a lot of SAS3008 stuff out there by now.

Thanks for the advice, both. I'll probably do that.