X11SSL-CF NVMe (bifurcation) vs Sata SSD mirror pool

[Platy]

Hi everybody,

I would like to create an dedicated SSD mirror pool to run my VMs there (4-5 bhyve VMs running Linux/docker with another 15-20 docker instances) and use it as a 'staging pool' if I need more performance before archiving/moving it to the slower pool.

All my important stuff is backed up in a 3-2-1 way, so I don't really need a 3-way mirror.

my current system (Truenas CORE one zpool with 2 mirror pools):

MOTHERBOARD: Supermicro-X11SSL-CF
CPU: Xeon E3-1230 v6
RAM: 64GB DDR4 ECC
BOOT: Samsung MZ-76E250B 860 EVO, 250 GB, 2.5" SATA III
HDD: mirror pool: 2x Western Digital WD Red HDD 4000 GB, SATA III
mirror pool: 2x Western Digital WD Red HDD 10000 GB, SATA III
CASE: Fractal Design Define R5 Case
PSU: Be quiet! Pure Power 11 700W CM

Connectivity: 1 GbE Ethernet (possible upgrade to 10 GbE in future)

Two options:

SATA III SSDs:

+Easier connectivity (it's just SATA), and the motherboard has an LSI3008, thus providing another 8 ports I can use for SATA.
+Cheaper, especially for 4TB models
-Only max 600 MB/s speed (probably doesnt matter as the throttleneck is 1 GbE Ethernet)
-Very few consumer models available, and the most well-known Samsung 870 model appears to have significant firmware problems.

2x Samsung 870 Evo 2TB = 110 * 2 = 220 Euro
2x Western Digital 2TB Red SA500 = 129 * 2 = 258 Euro

NVMe SSDs:

*More expensive for 4TB, more or less the same for 2TB.
+much higher speeds around 3 - 3.5 GB/s (my board only has PCIe 3.0 support)
+a lot of different model choices
-No Built-in NVME slots, the motherboard does not support bifurcation so I need to buy a card with PLX controllers e.g. ANM24PE16 Controller Ceacent NVMe M.2, Quad Port, its around 140-160 Euro.

2x Samsung 980 PRO NVME 2TB + Ceacent = 110 * 2 + 140 = 360 Euro


In the end they are the same cost, its just the bifurcation card that costs extra (which however gives me 4 NVme slots). To be honest, due to the same price but more speed and options, I'm more inclined towards NVMe. However, I'm open to points I might have overlooked.