All SSD Array Optimizations

[r34lj4k3]

Hello, I have built an all SSD array with the following configuration:

• Motherboard make and model: Dell r720xd
• CPU make and model: Dual Intel E5-2643 v2 (6 core 3.5ghz, HT, 3.8ghz turbo) (these can be swapped for 2697 v2 cpus if you think it would be better)
• RAM quantity: 256GB ECC DDR3
• Hard drives, quantity, model numbers, and RAID configuration, including boot drives
  • Generic 128GB NVME SSD for boot
  • 1tb Samsung 960 pro (for possible l2arc, not currently in use)
  • 26 pm863 SSDs, all mirrored and put into a single pool
  • 2 Radian RMS-200 SLOG drives, striped
• Hard disk controllers: Dell PERC 710 flashed to IT mode
• Network cards: Broadcom 57800 2x10Gb DA/SFP+ + 2x1Gb
• Version: TrueNAS Core 12.0U6
• Use Case: iSCSI target for a pair of HA/Redundant esxi 7.0 hosts, round robin MPIO is configured and working.

The questions I have are, should I use the Samsung 960 pro for l2arc? Secondly, are there any optimizations needed for an all SSD array? I've already enabled TRIM. As an example, I found this tunable on a thread from 2019: vfs.zfs.metaslab.lba_weighting_enabled: 0 and they said it should be set on all SSD arrays, but I don't know if that is still relevant. How do I see tunables? Should I be changing any of them? Also, should I consider upgrading my PERC 710 to something else?

 

[jgreco]

The 2643's are a better choice.
What's the point of the L2ARC here?
Why the Broadcom network? Please pick from options listed in the 10 Gig Networking Primer for best luck and good performance.

 

[Ericloewe]

Dell has Intel X710 plus I350 or quad-port X710 cards for their proprietary slot (rNDC) and they're pretty cheap. Definitely the way to go. There's also the X520 plus I350 option, as well as the X540 plus I350 model.

 

[r34lj4k3]

The 2643's are a better choice.
What's the point of the L2ARC here?
Why the Broadcom network? Please pick from options listed in the 10 Gig Networking Primer for best luck and good performance.

I figured the speed of the 2643s would be helpful.

I am just trying to maximize performance, that's the point of my post in general. I am trying to mimic prod's performance as much as possible, with different hardware. Prod is very performant. I know more RAM is preferred before l2arc, but I don't necessarily have more budget and I do have a 960 pro on hand.

The broadcom came with the server and had two 10gb sfp+ already and they're working so far, so I hadn't thought to change them. Do you know for certain that this is a thing I need to swap or is TrueNAS just more tolerant of Intel NICs?

 

[jgreco]

Broadcom isn't Realtek and they don't make crap, but the fact that an ethernet chipset is decent does not make it decent under every OS.

I know that Broadcom ethernets are frequently associated with problems on FreeNAS/TrueNAS. If yours works for you, that may be specific to the chipset and driver combination you happen to have. I totally believe that different products from the same vendor can behave differently, so it's quite plausible that you have one that works, or at least isn't a tragedy in motion. If it gives you trouble, be aware that the resolution might be to add in a Chelsio or Intel card.

 

[r34lj4k3]

Broadcom isn't Realtek and they don't make crap, but the fact that an ethernet chipset is decent does not make it decent under every OS.

I know that Broadcom ethernets are frequently associated with problems on FreeNAS/TrueNAS. If yours works for you, that may be specific to the chipset and driver combination you happen to have. I totally believe that different products from the same vendor can behave differently, so it's quite plausible that you have one that works, or at least isn't a tragedy in motion. If it gives you trouble, be aware that the resolution might be to add in a Chelsio or Intel card.

I appreciate that and will look out for any network oddities. That being said, does anyone happen to know if there are any SSD related tunables I should be setting/changing? Also, googling how to list the current tunable settings has come up empty. Anyone know how to do that?

Will the 960 pro help my random writes? Is the only way to know that to test it?

 

[jgreco]

Currently set tunables are listed by the GUI under the Tunables section.

A 960 Pro will tend to do random writes better than HDD will, yes, but if there is sufficient free space on the HDD pool, the effect won't be that noticeable, as ZFS tends to lay down writes contiguously where possible. It could definitely be useful for jails or other high-processing activity.

If you use the 960 Pro as L2ARC (read cache), with sufficient memory, it will potentially accelerate reads of frequently accessed data.