First Truenas build, suggestions needed!

[Nassi]

A small update on this case.

I have bought a used Xeon Silver 4112 (4c/8t) processor which actually arrived a few days ago.
No doubt I could have gotten more cores for the same money, but I still tried to minimize the "overkill" even on a thought level.

A Supermicro X11SPL-F is currently being planned as the motherboard.
As far as I understand, the processor only supports a memory speed of 2400 MHz, and this speed was not found in Supermicro's tested memory list.

Supermicro tested memory list:
DDR4 1.2V-2666 -> MEM-DR432LC-ER26
DDR4 1.2V-2933 -> MEM-DR432LC-ER29
DDR4 1.2V-3200 -> MEM-DR432LC-ER32

I assume that the suitable 2400MHz version would be MEM-DR432LC-ER24?
Supermicro's product page lists the memory as the "original" model number M393A4K40CB1-CRC, which is Samsung memory.
These 32GB memory sticks are available on EU Ebay used for ~40€/pc.

The motherboard has 8 memory slots, but according to the manual, it is recommended to use all 8 slots only when necessary.
I have understood that the processor has a 6-channel memory, so 6 RAM sticks are probably recommended in this case?

I would like to receive confirmation of the memory selection before I place the order. Of course, I also accept other recommendations!

What are your thoughts on the Seasonic 650W PRIME PX-650 or TX-650 as a power supply?

 

[Etorix]

650 W looks small for a Xeon Scalable system, but that depends on the number of drives.

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can...

www.truenas.com

As for RAM, there's no need to pick exactly 2400 MHZ modules, and 2666/2933 MHZ may come in handy if you ever upgrade the CPU.

 

[Nassi]

As for RAM, there's no need to pick exactly 2400 MHZ modules, and 2666/2933 MHZ may come in handy if you ever upgrade the CPU.

Price difference between 2400MHz and 2666MHz is about 250€ vs 300€. (6x 32GB sticks)
Not much, but 100% wasted money if I never upgrade the CPU. Since the rack will also have a separate proxmox server for VM's etc, at least in my opinion there is no reason to upgrade the processor to a more powerful version? Or what do you think?

Is there a difference between 2400MHz vs 2666MHz in NAS use? (if supported by other parts)

How about boot devices?

I've been thinking about mirrored Kingston A400 128GB SSDs or is there some better alternatives?
Some other thread recommended Transcend 32GB SSD370S SSD's because of MLC.

 

[nabsltd]

Is there a difference between 2400MHz vs 2666MHz in NAS use? (if supported by other parts)

Real world benchmarks show that going from essentially the fastest DDR5 RAM to the slowest DDR4 (which can be done with the same CPU on different motherboards) has a less than 10% difference in processing speed.

In your case, you'd see about 1% difference at most.

 

[Etorix]

How about boot devices?

I've been thinking about mirrored Kingston A400 128GB SSDs or is there some better alternatives?
Some other thread recommended Transcend 32GB SSD370S SSD's because of MLC.

Anything small and cheap really… My pick is Optane M10 16 GB—not for any Optane feature but because they are dirt cheap (currently 7 E apiece on eBay) and do the job just fine without using a SATA port.

If you really, really, want a redundant boot system, study this:

Highly Available Boot Pool Strategy

I've pounded a few versions of this out over the years, but I hate explaining over and over. TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw. Due to...

www.truenas.com

 

[Whattteva]

Price difference between 2400MHz and 2666MHz is about 250€ vs 300€. (6x 32GB sticks)
Not much, but 100% wasted money if I never upgrade the CPU. Since the rack will also have a separate proxmox server for VM's etc, at least in my opinion there is no reason to upgrade the processor to a more powerful version? Or what do you think?

Is there a difference between 2400MHz vs 2666MHz in NAS use? (if supported by other parts)

More RAM capacity will always trump marginally faster speed RAM, especially when you use ZFS. Save your money and use it somewhere else.

How about boot devices?

I've been thinking about mirrored Kingston A400 128GB SSDs or is there some better alternatives?
Some other thread recommended Transcend 32GB SSD370S SSD's because of MLC.

Boot device has really low performance requirement. I don't even bother mirroring mine as it's so easily replicated (~5 mins) as long as you back up your config file. I'm still using some cheap 60GB SSD from 10 years ago.

 

[Nassi]

The idea of mirrored boot disks comes from recommendations that I have found in different threads on this forum.
I don't think there is a real need for a mirrored boot device in home use. However, you can get a spare disk in few days from local seller.
On the other hand, the mirror is not a very big financial investment.

I also looked at those Optane M10 16 GB on ebay before. An estimated 90% of them are located in China and the rest in the US.
In the Europe, there were only a couple of sales announcements, and those were from private individuals.
In terms of price, this would certainly be a superior option.

However, I have been wondering if there would be a need for a slog drive at some point...?
The idea would be to build a 10gbe LAN at least between the NAS and the Proxmox server.

There is one m.2 slot on the motherboard, so would it make sense to leave that spot to a possible slog device or would it make sense to have 2 of these mirrored?

 

[Etorix]

I don't think there is a real need for a mirrored boot device in home use.

Then go for a single boot device. Have a spare in a drawer and always keep a recent copy of the configuration file outside of the side.
Recovery from boot drive failure:
1° Replace the failed boot drive.
2° Install TrueNAS anew.
3° Load the configuration file.
Done!

I also looked at those Optane M10 16 GB on ebay before. An estimated 90% of them are located in China and the rest in the US.

Yes, from China, delivered by post in 7-10 days.

However, I have been wondering if there would be a need for a slog drive at some point...?
The idea would be to build a 10gbe LAN at least between the NAS and the Proxmox server.

A SLOG is for sync writes, not espacially for 10 GbE.
For that, you'd want more performance and more endurance than what the 16 GB M10 provides. Minimally, the 64 GB M10 variant; realistically, a 900p, ideally, a DC P4800X/4801X/5800X.

 

[Whattteva]

In my

The idea of mirrored boot disks comes from recommendations that I have found in different threads on this forum.
I don't think there is a real need for a mirrored boot device in home use. However, you can get a spare disk in few days from local seller.
On the other hand, the mirror is not a very big financial investment.

Well, it may not be very big financial investment, but it could be a big port investment as it could potentially take up a port that could otherwise be used for another data drive.

However, I have been wondering if there would be a need for a slog drive at some point...?
The idea would be to build a 10gbe LAN at least between the NAS and the Proxmox server.

A SLOG is really more for sync writes. In my opinion, this is probably more useful on the Proxmox server as the VM stores, by default, use sync writes unless you disable it. You could check this number with pveperf on the Proxmox server. Fsync numbers would typically be very low even with SSD's unless you use enterprise ones. This would typically manifest in lengthy IO Waits on your VM's when you perform any kind of sustained IO intensive operations like large file copies inside the VM's.

 

[Nassi]

I have tried to investigate this SLOG thing more closely.

I use a linux pc and I intend to connect the pc, truenas and other homelab devices via NFS.
My understanding is that NFS uses sync transfer between devices, so when writing to the HDD pool, ZFS uses the pool's internal ZIL?
This, in turn, significantly slows down the write speed. So is it the case that with a separate SLOG device I could get the speed of the 6x wide HDD RaidZ2 pool closer to (async) speed? As a data pool, I have planned 6 x RaidZ2 18TB or 20TB Seagate Exos drives.

Intel Optane P1600X would be an attractively priced SLOG device, opinions on this? The 58GB version would certainly be enough for its capacity, but would the slightly larger 118GB version offer any advantage?

Correct me if I'm wrong, an async (SMB) transfer just throws the transferred files to the recipient, and doesn't care if the received data is 1:1 with the source. sync transfer, on the other hand, ensures that the data is the same as the sender's, so "corruption" during the transfer cannot happen. This is the main reason why I am interested in sync transfers and as a result the SLOG device.

 

[danb35]

Correct me if I'm wrong

I'm afraid you are very badly wrong. sync vs. async has nothing to do with the integrity of the data sent over the network; it's instead a matter of whether TrueNAS (in this case) will report a write as completed once it's in a RAM buffer (async), or only once it's written to persistent storage (sync). This appears to explain it pretty well:

ZFS sync/async + ZIL/SLOG, explained – JRS Systems: the blog

 

[Etorix]

No SLOG will get you close to the speed of async writes; it only makes sync writes suck a little less.
For regular file sharing (no VMs, no database, as suggested by the raidz2 layout), just set sync=never on the relevant datsets and enjoy asynchronous NFS.

 

[Nassi]

I'm afraid you are very badly wrong. sync vs. async has nothing to do with the integrity of the data sent over the network; it's instead a matter of whether TrueNAS (in this case) will report a write as completed once it's in a RAM buffer (async), or only once it's written to persistent storage (sync). This appears to explain it pretty well:

ZFS sync/async + ZIL/SLOG, explained – JRS Systems: the blog

Thank you for the clarification! For some reason I was under the impression that the data is transferred to both the RAM and SLOG devices separately, and the pool checks the correctness of the data from two different sources before sending the confirmation to the sending device. With this thought model, I assumed that the sync transfer would be more reliable in terms of the correctness of the data.

If I understood correctly, the pool is not interested in what kind of data the SLOG device has under normal circumstances and that data is deleted from the SLOG every 5 seconds without any read. Only at power loss or kernel Panic etc. that 5 sec data will be restored from SLOG device as long as it has PLP feature.

What do you think about adding a special VDEV or L2ARC? I understood that L2ARC can only be used to store metadata? Does the definition of "metadata" include thumbnails and such? The purpose is to use the Truenas server to store all my photos as well. These features can be useful in this case to make the user experience as snappy as possible.

The special VDEV cannot be removed and the failure destroys the pool. Would for example a 3 wide mirror or a 4 wide RaidZ2 be a good solution in this case? Can consumer grade SSDs be used here? For example Samsung 990 PRO 2TB etc.? As for L2ARC, I can probably use consumer grade NVME devices? Does L2ARC benefit mirrored solution?

 

[Nassi]

I just updated the first post to better reflect the current situation!
In addition, I added a list of already selected components, as well as questions and thoughts about the missing ones.
No need to wade through the entire thread to find out what is going on at this moment.

I hope this makes it easier to discuss and give answers to my questions!
I will update the first post as new hardware parts are selected or ordered.