Upgradeability of TrueNAS Mini X (10GbE)?

[ByteMan]

Hello all,
based on my research, the Mini X uses Supermicro's A2SDi-4C-HLN4F (Atom C3558, 16W TDP), whereas the Mini X+ utilized the better equipped Supermicro A2SDi-H-TF (Atom C3758, 25W TDP) with 2x 10GbE on board.
Per the data sheet, the Mini X is advertised as "PCIe expansion = N/A", but the A2SDi-4C-HLN4F does in fact have one PCI-E 3.0 x4 slot. I understand that there is some lane sharing going on (configurable lanes), but I do not understand the actual impact as far as the number of lost SATA ports, assuming a 10GbE AOC was added (if that is even possible)? How many lanes would remain available for SATA devices after such an upgrade?
Is the fact that using the PCI slot will cut into the available SATA lanes the likely why PCIe Expansion is not being advertised?

*Number of PCI-E lane is configurable via BIOS setup: 0, 2, or 4. Total combined PCI-E lanes and SATA ports is up to 8.

At PCIe 3.0, two lanes should provide 2 GB/sec = 16 GBit/sec, which seems adequate for 10GbE.

Thank you for your support and guidance!

 

[ByteMan]

....and adding on to my previous post: Would the on board m.2 port still be usable in case the PCIe slot is used for a 10GbE AOC + assuming the remaining available SATA ports are also in use?

A2SDi-4C-HLN4F Manual:

 

[Ericloewe]

Total combined PCI-E lanes and SATA ports is up to 8.

As far as I can tell, although it's not 100% crystal clear, this statement applies to the set of {PCIe slot; SATA ports}, meaning that the M.2 slot has two dedicated PCIe lanes. I suspect that the SATA port for this slot is also dedicated

The SoC itself has 12 High-Speed IO channels, which can (subject to limitations) be configured either as PCIe lanes or SATA ports.

So, at that point, your options are:

1. PCIe x4 + 4x SATA via the SFF-8643 connector
2. PCIe x4 + 4x SATA via single ports 0 to 3
3. PCIe x2 + 4x SATA via the SFF-8643 connector + 2x SATA via single ports 0 and 1
4. 4x SATA via the SFF-8643 connector + 4x SATA via single ports 0 through 3

So, the factory configuration for a TrueNAS Mini X presumably is set to allow for 7 SATA drives, meaning option 4. above, which supports no PCIe outside the M.2 slot. If you're okay with losing one of the bays, you should be able to use option 3. and providing the NIC with PCIe x2. Beware that you'll definitely want a recent-ish NIC that supports PCIe 3.0.

 

[ByteMan]

Thank you very much for the detailed feedback!
Using option 3, I should be able to install 6 drives (5 in case I continue to use the SATA DOM supplied with the Mini) + add 10GbE capability. Combined with the lowered power usage (assuming modest work load), this modified version of the Mini X seems to be more attractive than the Mini X+; or is there something I am missing?
Should I be concerned that the increased CPU load imposed by the 10GbE connection will reduce the remaining processing capacity to a point where the system becomes instable (depending on number of additional plugins that are running)?
Are there any other red flags in regards to this approach of using a Mini X and adding a 10GbE AOC? I would also add another 16GB of RAM.

 

[Ericloewe]

Combined with the lowered power usage (assuming modest work load), this modified version of the Mini X seems to be more attractive than the Mini X+; or is there something I am missing?

Power differences at idle will be small. While doing something, the total energy consumption probably won't be too different.

Should I be concerned that the increased CPU load imposed by the 10GbE connection will reduce the remaining processing capacity to a point where the system becomes instable (depending on number of additional plugins that are running)?

Might it not perform well enough to saturate the connection? It's possible, hard to say without a concrete scenario. Unstable system? No.

The X+ seems like the better option here, unless you're extremely cost-sensitive and willing to compromise on the number of disks supported. Even then, cost can end up being a wash in some situations.

 

[ChrisRJ]

Just be aware that saturation of 10 Gbps with a small number of spinning disks will be between "not that great" and rather bad. You should probably also read the 10 Gbps primer from the "recommended readings" in my signature

 

[ByteMan]

Thank you for all the input!
For background, at least for now my goal is not to saturate the 10G connection, but to squeeze out what I can from two 4TB IronWolf Pro 125 SSDs (SATA connected).

Power differences at idle will be small. While doing something, the total energy consumption probably won't be too different.

In case idle consumption of both is almost the same, would you guys expect the additional 10GbE AOC to increase the idle power consumption of the lower-powered Mini X beyond that of the X+?

Is anyone aware of actual power consumption measurements (not under full load) between Mini X and X+ that are comparable (similar drive setup)?

The X+ seems like the better option here, unless you're extremely cost-sensitive and willing to compromise on the number of disks supported. Even then, cost can end up being a wash in some situations.

My goal is to have the lowest possible power consumption when the unit is idle or has low CPU utilization, while ensuring the unit still functions properly.

For reference:

 

[Ericloewe]

In case idle consumption of both is almost the same, would you guys expect the additional 10GbE AOC to increase the idle power consumption of the lower-powered Mini X beyond that of the X+?

Quite possible. The integrated NIC is manufactured on a more advanced process than most discrete NICs that aren't cutting edge.

 

[ByteMan]

Comparing the Atom C3558 (Mini X) to the C3758 (X+), I noticed that both support Intel QAT, but at different speeds (low vs medium).
I understand that QAT helps with ZFS gzip compression, but would anyone know if QAT speed (low vs medium) would actually make a noticable difference in the case of running a NAS?

 

[ByteMan]

I came across this showing actual QAT throughput limits:

Would someone have the understanding to assess whether 5 vs 10 Gbps QAT would make a noticeable difference?

 

[Ericloewe]

The first question is "does any version of TrueNAS support this". I'd be interested to know the answer, too.

 

[ByteMan]

The first question is "does any version of TrueNAS support this". I'd be interested to know the answer, too.

That is a good point. I read somewhere that QAT compatibility has been incorporated into ZFS but it still might not apply to TrueNAS