Tell me why my NVMe build sucks (or doesn't)

[geruta]

I am planning to buy a bunch of new hardware for the purposes of serving as iSCSI storage for 3 ESXi servers in my home lab. I run only 5-10 VMs at most and they are nearly always idle as it's just a lab for learning.

Granted this hardware is more expensive then last gen and DDR4, but it's also not that bad for the number of M.2 PCIe 4.0 ports that I am looking for (5x)

Motherboard: ASUS Pro WS W680-ACE IPMI LGA 1700 ATX Motherboard

CPU: Intel Core i5-13600K 3.5 GHz 14-Core LGA 1700 Processor

Memory: 64 GB DDR5 ECC (Micron mtc20c2085s1ec48br)

5x SAMSUNG 990 PRO SSD 2TB PCIe 4.0 M.2 to run in RAID5

1x SAMSUNG Electronics 870 EVO 2TB 2.5 Inch SATA (Boot Disk)

NIC: X550-T2 - Dual 10GbE Card

I'm not seeing any builds using NVMe drives for the main storage pool, and I'd like to know why. Given that Samsung NVMe drives are just 10-20 bucks more than the SATA counterparts, it seems like going NVMe would be a great way to future proof myself.

So what am I missing? Why are NVMe SSDs seemingly absent from most builds?

 

[jgreco]

I'm not going to take a lot of time here, you asked for feedback, and I had some major points I wanted to say, but if I wait it'll be tomorrow before I get to it. Sorry if it's a bit rushed.

5x SAMSUNG 990 PRO SSD 2TB PCIe 4.0 M.2 to run in RAID5

1) TrueNAS uses ZFS, and does not support RAID5. Even if we assume you meant RAIDZ1, this is still a problem because it's clear you didn't read the block storage resource. Which will tell you why you don't use RAIDanything for block storage.

1x SAMSUNG Electronics 870 EVO 2TB 2.5 Inch SATA (Boot Disk)

2) That's a waste of about 1.8TB or $100, whichever. You could save money.

I'm not seeing any builds using NVMe drives for the main storage pool, and I'd like to know why.

3) Because NVMe requires lots of PCIe lanes, and PCIe lanes tend to be scarce on the systems hobbyists or SOHO folks can afford. Fixes for this such as PLX switch chip cards are expensive.

NIC: X550-T2 - Dual 10GbE Card

4) And didn't read the 10 Gig Networking Primer. Bad choice, go with SFP+.

Given that Samsung NVMe drives are just 10-20 bucks more than the SATA counterparts,

5) Wasted money. If we were to assume that you built a pool with three mirror vdevs, the theoretical speeds might be around 21,000 MBytes/sec. The theoretical speed of 10GbE is 1,250MBytes/sec. So it would be more practical to look at the much more easily connected SATA SSD's with their 530 MBytes/sec speed cap because three mirror vdevs of that might theoretically be around 1,590 MBytes/sec. The read speeds could be as much as double that. However, unlike NVMe, SATA is easily expandable, so getting yourself a 24-bay 2U SFF and stuffing it with SSD's would give you lots of expandability, the option to have warm spares online, etc.

it seems like going NVMe would be a great way to future proof myself.

6) Future proof against what?

 

[Etorix]

Granted this hardware is more expensive then last gen and DDR4, but it's also not that bad for the number of M.2 PCIe 4.0 ports that I am looking for (5x)

Motherboard: ASUS Pro WS W680-ACE IPMI LGA 1700 ATX Motherboard

Hopefully, this is not bad as a desktop—though still crazy expensive compared with the Z690 counterpart. But if you want to shoot for the Moon, get something which looks like a rocket rather than an air balloon.
That is: A server platform with lots of PCIe lanes, and the ability to bifurcate x16 all the way to x4x4x4x4. Something like a X11SRM-VF with a Xeon W-2100/2200 or a 1st/2nd gen. Xeon Scalable (X11SPM, X11SPL, X11SPi…). And lots of cheap DDR4 RDIMM in any case.

 

[QonoS]

Motherboard: ASUS Pro WS W680-ACE IPMI LGA 1700 ATX Motherboard
<<< kind of does not suck ;-), WS is good for workstation and maybe useable as server hardware but you can get better stuff, BMC/IPMI is good you know ;-)

CPU: Intel Core i5-13600K 3.5 GHz 14-Core LGA 1700 Processor
<<< sucks; power consumption bad, no server CPU

Memory: 64 GB DDR5 ECC (Micron mtc20c2085s1ec48br)
<<< does not suck, because ECC is good

5x SAMSUNG 990 PRO SSD 2TB PCIe 4.0 M.2 to run in RAID5
<<< sucks for TrueNAS, (1) this SSD expects to get time with no load at all for garbage collection, it is possible that this time is not available; (2) no power loss protection (PLP), ESXi and main storage w/o PLP is particularly bad

1x SAMSUNG Electronics 870 EVO 2TB 2.5 Inch SATA (Boot Disk)
<<< kind of does not suck, but SSD with PLP should always be preferred, why 2 TB?

NIC: X550-T2 - Dual 10GbE Card
<<< does not suck ;-) but you can save power using 2.5Gbit/s Ethernet if that fits your needs to, otherwise Fiber requires less power

 

[geruta]

After an unhealthy amount of internal debate, I've decided that I am going to try and source out older hardware and aim for 128 GB of RAM.


Does this new build make more sense?

Motherboard: Supermicro X11SPL-F Intel C621

Intel® Xeon® Gold 5122 Processor
CPU Fan: Dynatron B11 Intel Skylake Socket FCLGA3647

4x SAMSUNG 32GB 2Rx4 DDR4-2666 PC4-21300 Registered ECC Server RAM M393A4K40BB2-CTD

2x SATA DOM (64 GB)

6x SAMSUNG Electronics 870 EVO 2TB 2.5 Inch SATA

1x Intel X710-BM2 (Dual port 10Gb SFP+)


Concerns:
- Uses slower 2666 MHz DDR4 speed
- Uses slower SATA vs NVMe
- Motherboard tops out at 6 SATA ports (I could always add more via PCIe card I suppose).
- Motherboard is X11, so IPMI/BMC firmware is going to stop getting updates (i assume) which makes me concern about issues accessing it a year from now. (if older ciphers/TLS versions stop being supported, this already hit me once with ASUS iKVM).

 

[Etorix]

@QonoS, while we share a conclusion, I beg to differ on nearly every individual item:

Motherboard: ASUS Pro WS W680-ACE IPMI LGA 1700 ATX Motherboard
<<< kind of does not suck ;-), WS is good for workstation and maybe useable as server hardware but you can get better stuff, BMC/IPMI is good you know ;-)

Have you noticed the word "IPMI" in the board's name?

CPU: Intel Core i5-13600K 3.5 GHz 14-Core LGA 1700 Processor
<<< sucks; power consumption bad, no server CPU

Hybrid architecture is not supported by TrueNAS (CORE or SCALE).
TDP is actually irrelevant to a NAS, and power draw at idle isn't bad. Supports ECC.
What's a "server CPU" by the way?

Memory: 64 GB DDR5 ECC (Micron mtc20c2085s1ec48br)
<<< does not suck, because ECC is good

Kinda sucks in my book because 64 GB is already half of the board's ability, and this is expensive DDR5 UDIMM rather than cheap DDR4 RDIMM.

5x SAMSUNG 990 PRO SSD 2TB PCIe 4.0 M.2 to run in RAID5
<<< sucks for TrueNAS, (1) this SSD expects to get time with no load at all for garbage collection, it is possible that this time is not available; (2) no power loss protection (PLP), ESXi and main storage w/o PLP is particularly bad

PLP is required for SLOG duty, not necessary in any other role. 4 or 6 of these should do fine for iSCSI as mirrors.
If I understand the OP correctly, the inquiry is about storage for ESXi, NOT about running virtualised TrueNAS under ESXi.

1x SAMSUNG Electronics 870 EVO 2TB 2.5 Inch SATA (Boot Disk)
<<< kind of does not suck, but SSD with PLP should always be preferred, why 2 TB?

Capacity is indeed ridiculous for a boot drive. But there's no reason to seek PLP for a boot drive.

NIC: X550-T2 - Dual 10GbE Card
<<< does not suck ;-) but you can save power using 2.5Gbit/s Ethernet if that fits your needs to, otherwise Fiber requires less power

2.5 GbE sucks in any version, Realtek or Intel.
Our beloved Resident Grinch concedes that X550 is "serviceable" when found on-board. As add-in card, prefer SFP+. Solarflare SF7122F go for $50.

 

[QonoS]

Have you noticed the word "IPMI" in the board's name?

Overlooked that one indeed, thanks.

Hybrid architecture is not supported by TrueNAS (CORE or SCALE).
TDP is actually irrelevant to a NAS, and power draw at idle isn't bad. Supports ECC.
What's a "server CPU" by the way?

Server CPUs (e.g. Xeon, EPYC) are made and validated for these kind of workloads and therefore perform better. This is a minor argument of course. Hybrid architecture not being supported is a major point imho.
Concerning power draw: idle is most relevant right, but depending on workload higher power states are relevant too. 5-10 VMs can be enough for that.

Kinda sucks in my book because 64 GB is already half of the board's ability, and this is expensive DDR5 UDIMM rather than cheap DDR4 RDIMM.

Good points but ECC is most relevant in my eyes.

PLP is required for SLOG duty, not necessary in any other role. 4 or 6 of these should do fine for iSCSI as mirrors.
If I understand the OP correctly, the inquiry is about storage for ESXi, NOT about running virtualised TrueNAS under ESXi.

Less complexity is usually best. Means a pool of fast SSDs without SLOG, Special Devices, etc performs best.
Why PLP? iSCSI ESXi does Sync Writes (for data safety). Only PLP SSD perform well here.
PLP also protects against data corruption when power failures happen and to a certain degree in events of system crashes.

Capacity is indeed ridiculous for a boot drive. But there's no reason to seek PLP for a boot drive.

As stated above PLP is good for reliability.

2.5 GbE sucks in any version, Realtek or Intel.

Maybe just with TrueNAS CORE.

 

[jgreco]

What's a "server CPU" by the way?

I would point out that there's historically been some significant differentiation between server, workstation, desktop, and gaming CPU's.

Gaming CPU's are perhaps easiest to spot because they often end in "K", for overclockable/tweakable.

Xeon typically denotes server or workstation CPU's, with further differentiation depending on the Intel gameplan of the era. For example, through much of the Xeon E5 era, a Xeon E3 would be a lightweight server CPU, E5-24xx would be midrange compute server, E5-26xx would be dual socket server with extra cache and I/O. Xeon E5-16xx would typically be considered as a single socket workstation CPU or midrange server CPU. These roles are somewhat conflatable, as (for example) the trashcan Mac Pro (2012 era IIRC) used an E5-2697v2 as its high end workstation CPU, despite technically being a server CPU.

Intel typically indicates the intended market segment in the ARK entry for a CPU, just in case there's confusion. However, as with the trashcan Mac example, the rules are not absolute.

 

[geruta]

Mr Grinch (if I may call you that ),

What do you think about this build?

Motherboard: Supermicro X11SPL-F Intel C621

Intel® Xeon® Gold 5122 Processor
CPU Fan: Dynatron B11 Intel Skylake Socket FCLGA3647

4x SAMSUNG 32GB 2Rx4 DDR4-2666 PC4-21300 Registered ECC Server RAM M393A4K40BB2-CTD

2x SATA DOM (64 GB)

6x SAMSUNG Electronics 870 EVO 2TB 2.5 Inch SATA

1x Intel X710-BM2 (Dual port 10Gb SFP+)


Concerns:
- Uses slower 2666 MHz DDR4 speed
- Uses slower SATA vs NVMe
- Motherboard tops out at 6 SATA ports (I could always add more via PCIe card I suppose).
- Motherboard is X11, so IPMI/BMC firmware is going to stop getting updates (i assume) which makes me concern about issues accessing it a year from now. (if older ciphers/TLS versions stop being supported, this already hit me once with ASUS iKVM).

 

[Whattteva]

Concerns:
- Uses slower 2666 MHz DDR4 speed

I'm using 224 GB of even "slower" 2400 DDR4. You're running a NAS, not a gaming machine or some 3D intensive workload that requires crazy bandwidth.

- Uses slower SATA vs NVMe

So you're concerned about using slower SATA. Fair, but.......

- Motherboard tops out at 6 SATA ports (I could always add more via PCIe card I suppose).

Now you're concerned about only topping out at 6 SATA ports.... which is it? NVMe or SATA?

- Motherboard is X11, so IPMI/BMC firmware is going to stop getting updates (i assume) which makes me concern about issues accessing it a year from now. (if older ciphers/TLS versions stop being supported, this already hit me once with ASUS iKVM).

I don't understand why that lack of support matters honestly. One of my machines is still running an X10XLL-F board which has been discontinued for some time, but I'm still accessing it just fine. Do note that I don't own an Asus board with iKVM so my knowledge here is limited.

 

[NickF]

There's alot of good reasons why this isn't a common usecase right now. I was, not all that long ago, asking the same question.

TrueNAS Scale NVME Performance Scaling

Over the past year or so I have been obsessively exploring various aspects of ZFS performance, from large SATA arrays with multiple HBA cards, to testing NVME performance. In my previous testing I was leveraging castoff enterprise servers that were Westmere, Sandy Bridge and Ivy Bridge based...

www.truenas.com

In my own testing, performance didn't scale quite as linearly as I had hoped. In your original post, you mentioned that ahrdware is getting better. It's true that DDR5 will mitigate some of the performance bottlenecks through sheer brute force, but taking some time to understand whats under the hood will yield some better insights.

Important changes are coming to ZFS soon which will improve NVME performance pretty drastically...

Scaling ZFS for NVMe - Allan Jude - EuroBSDcon 2022

Learn about how ZFS is being adapted to the ways the rules of storage are being changed by NVMe. In the past, storage was slow relative to the CPU so request...

youtu.be

DirectIO for ZFS by Brian Atkinson

From the 2021 OpenZFS Developer Summitslides: https://docs.google.com/presentation/d/1f9bE1S6KqwHWVJtsOOfCu_cVKAFQO94hDetails: https://openzfs.org/wiki/OpenZ...

www.youtube.com

DirectIO For OpenZFS Shows Very Promising Performance - Phoronix

www.phoronix.com

 

[Etorix]

What do you think about this build?

Ticks all right boxes in my book.

Concerns:
- Uses slower 2666 MHz DDR4 speed

Indeed. But what's the use case for extra RAM bandwidth? More never hurts, of course, but would it be worth the extra cost?

- Uses slower SATA vs NVMe

SAS/SATA can have the power of number, while adding more NVMe storage quickly goes into dual-socket category.

- Motherboard tops out at 6 SATA ports (I could always add more via PCIe card I suppose).

If using a SAS HBA, and possibly expanders to add lots of SATA drives, it doesn't matter.
Going for a X11SPM, X11SPi or X11SPH rather than X11SPL ('L': low-end) would provide 10-12 SATA ports straight out the chipset.

- Motherboard is X11, so IPMI/BMC firmware is going to stop getting updates (i assume) which makes me concern about issues accessing it a year from now. (if older ciphers/TLS versions stop being supported, this already hit me once with ASUS iKVM).

Many here would trust Supermicro to keep supporting X11 boards for quite some time still.
With respect to BIOS and BMC updates however the official and explicit recommendation from Supermicro is to NOT update firmware unless there's a known issue in the present configuration.

 

[Etorix]

I would point out that there's historically been some significant differentiation between server, workstation, desktop, and gaming CPU's.

Thanks for the detailed answer. However my question was largely rhetorical, as I see much of the differentiation as artificial segmenting rather than actual technical difference.

Xeon typically denotes server or workstation CPU's, with further differentiation depending on the Intel gameplan of the era. For example, through much of the Xeon E5 era, a Xeon E3 would be a lightweight server CPU, E5-24xx would be midrange compute server, E5-26xx would be dual socket server with extra cache and I/O. Xeon E5-16xx would typically be considered as a single socket workstation CPU or midrange server CPU. These roles are somewhat conflatable, as (for example) the trashcan Mac Pro (2012 era IIRC) used an E5-2697v2 as its high end workstation CPU, despite technically being a server CPU.

I guess you mean "marketically"? As the trashcan MacPro illustrates, mid-/high-end server and workstation duties are largely overlapping or interchangeable. Xeon W-3000 "workstation" parts tend to come with slightly higher clocks than their "server" Xeon Scalable counterparts, but Xeon Scalable do fine in workstation-intended C621(A) boards, and officially take on workstation duties in high-end dual-socket systems since W-3000 drop the UPI links for some extra PCIe lanes.
The case of the Xeon E3/E-2000/W-1000/"Core with ECC" lines is most telling. Xeon E3 were both lightweight servers and low-end workstations (aka. "desktop with ECC support"). Intel holding separate "launch events" for Xeon E-2000 as servers and the very same Xeon E-2000 as workstations was… ridiculous (to be polite). In those days, Core i3 supported ECC as well: The very same Core i3-9100F CPU could serve for a cheap gaming build, with a H310 motherboard and whatever mid- to low-end nVidia GPU was fashionable then, or for a lightweight server, in a X11SCH-F. A Xeon E-2224 without the "Xeon tax".
Then Intel dropped ECC support in Core i3 CPUs and bifurcated workstation W-1200 from server E-2300. That made for a double Xeon tax, on the CPU itself and then on the W480 or C256 chipset over Z490, but was possibly the most convincing technical differentiation between server and workstation CPUs since the W-1200/1300 were Comet Lake/Raptor Lake desktop parts with ECC, using up to 250 W of power under full load, while server E-2300 were based on mobile Tiger Lake parts and did not ramp as high in power.
Now Intel has changed its gameplan again with Alder Lake/Raptor Lake: The newly introduced W-1000 "workstation" line is no more; "regular" Core CPUs, up to Core i9, can do ECC when used on W680 boards. And there's no E-2400 "server" line in sight. So a Core i5-13600K can equally be a desktop with a B660 motherboard, an overclocked gaming system with a Z790 motherboard or an official "workstation" with a W680 motherboard. Fine. It's mostly of matter of intent. No technical difference in the CPU.

So if someone wants to use a i5-13600K as "server" it's fine with me. I think there are better options out there with respect to price, capabilities and OS support. I would personally not buy a WS W680-ACE IPMI and matching CPU anew for use as server. But if I had such a system—for use as a desktop/workstation—I could consider recycling it as a server five years from now and I'm confident it would then do fine in this use.

 

[Dice]

Mr Grinch (if I may call you that ),

What do you think about this build?

This is likely among the shortest possible paths to achieve the exact opposite of your intention.

 

[jgreco]

As the trashcan MacPro illustrates, mid-/high-end server and workstation duties are largely overlapping or interchangeable.

As the trashcan MacPro also illustrates, the E3-2697v2 was more than four times (~$2600) more expensive than a high end workstation-targeted CPU (~$600)., although that's not an apples-to-apples comparison because there are no 12-core workstation-targeted CPU's. But if you compare the single-socket E5-1650v2 ($583) to the E5-2643v2 (~$1600) you see a 3x price jump for what is effectively just more cache and the ability to run dual socket. It's clear that even if you CAN repurpose a CPU for other tasks, which I've often done over the years, they are nevertheless targeted at certain market segments and then also priced accordingly to wring maximal profit.

If anything, this is only accelerating as new Intel CPU's are being ever more targeted; ten years ago, we weren't seeing cloud/hyperscalers able to significantly impact Intel's strategy roadmap, but now we're seeing some of that.

 

[geruta]

There's a lot of threads branching off here so to get back to center...

Thanks all for your feedback, I've learned quite a bit here. As a recap:

- NVMe drives aren't practical yet given difficulty/cost in getting the right hardware to support 6+ NVMe drives
- NVMe drives will provide speed increases BUT won't be appreciated once we transverse the network without some having more serious network IO (unlikely to saturate a 10Gbps NIC using 6x SATA SSDs)
- DDR5 is obviously faster than DDR4, but still expensive and hard to find. Especially if looking for straight ECC (many claim ECC but are just ECC on-die)
- SFP+ is better for 10Gig NICs as they consume less power draw then copper, and produce less heat
- Block storage requires a lot of RAM, so 128GB is strongly recommended.
- Boot disk(s) don't require a lot of storage, so choose a more economical disk. (I've decided to go the SATA DOM route)
- For block storage, RAID bad, mirrors good.
- SuperMicro X11 motherboards although two revs behind now (x13 is current) it's only a few years old (looks like it came out in sometime 2020?). And it's believed that SuperMicro will support this board for a few more years.

For those wondering why X11 being "dated" was an issue for me, I bought an ASUS motherboard a few years back and installed a iKVM6 card so I could remotely access the system via IPMI. Well fast-forward a year or two after purchase and Chrome/Firefox/IE no longer are compatible with the system as legacy TLS ciphers were dropped and since it uses Java for KVM access, Java had to be hacked to oblivion to accept the older (insecure) certs and ciphers, which means that I don't really get to enjoy the benefits of IPMI/KVM for that old ASUS server anymore. And I don't necessarily enjoy ripping apart config files to accept older certs/ciphers to get that working and make my system less secure in the process.

I will forgo the previous all-NVMe build in favor of something more suitable.

Motherboard: Supermicro X11SPL-F Intel C621
Intel® Xeon® Gold 5122 Processor
CPU Fan: Dynatron B11 Intel Skylake Socket FCLGA3647
4x SAMSUNG 32GB 2Rx4 DDR4-2666 PC4-21300 Registered ECC Server RAM M393A4K40BB2-CTD
2x SATA DOM (64 GB)
6x SAMSUNG Electronics 870 EVO 2TB 2.5 Inch SATA
1x Intel X710-BM2 (Dual port 10Gb SFP+)


Thanks!

 

[Whattteva]

As the trashcan MacPro illustrates, mid-/high-end server and workstation duties are largely overlapping or interchangeable.

Speaking of Trashcan MacPro's.....

Reddit - Dive into anything

www.reddit.com

 

[Etorix]

I will forgo the previous all-NVMe build in favor of something more suitable.

Your revised list is very suitable. But I should stress that I initially suggested Xeon W-2000/Scalable because you wanted to go NVMe, which required plenty of PCIe lanes. If you're going for mirrors of SATA drives, and now only require about 6-8 SATA onboard and a x8 slot for the NIC, you can also scale down to the like of X12STH or X11SCH with a matching Xeon E-2300 or E-2200 (or then even a Core i3-9100, as mentioned in the above branching—sorry for that). More RAM is always nice and good, but for a few TB of storage 32 GB of DDR4 UDIMM would be plenty.
On these platforms, small and cheap NVMe M.2 SSDs are preferred boot devices over SATADOM or SATA SSDs, because they do not take precious SATA ports for the storage pool—and there's little other use for the M.2 slots anyway.

 

[Dice]

Your revised list is very suitable.

Agree there are clear improvements. Well done to OP!

On these platforms, small and cheap NVMe M.2 SSDs are preferred boot devices over SATADOM or SATA SSDs, because they do not take precious SATA ports for the storage pool—and there's little other use for the M.2 slots anyway.

I think the m.2 slot is among the highest valued resource on the motherboard.

It's a fantastic spot for the SLOG for example..

 

[Etorix]

…provided one needs a SLOG. And if one does, my preferred position would then be in a PCIe slot from the CPU (as Optane AIC, or U.2 in an adapter) over a M.2 slot from the chipset. Also, M.2 drives with PLP are typically 22110 and may not fit everywhere.

With iSCSI storage, the OP has a potential use case for a SLOG. But with a SSD pool, there's a question whether the pool could actually be faster than a single SLOG device.