New Upgrade around X10DRH-IT

[Dave Hob]

Hi all again. I wrote this thread before and somehow I did not hit the post thread button or something cause I lost it, will try to do a better job.

Would like to upgrade my current Truenas Core setup (Dell PowerEdge T410 with 6 x WD Red Nas 3TB raidz2).

This upgrade is planned around my new needs and using some equipment I already have. The most difficult decision was either going with some dedicated to Truenas (ideally Supermicro X11SCH-LN4F MB - since regular F is hard to find & Intel Xeon E-2100 or E2200) or something that takes advantage of some parts that I already owned.

Components list - that I am going for at the moment

Case - Fractal Define 7 XL
Motherboard - X10DRH-iT
CPU - 2 x Intel Xeon E5-2690 v3
CPU - 2 x Intel Xeon E5-2699 v4 (when I removed the cpu from my machine that haven't turn on in almost a year, I realized the model was slightly different)
* I do own 4 of this CPUs as part of some HP gen 9 servers so was planning to use at list for the moment, with an upgrade in mind in the future for higher frequency
Memory - 4 or 8 x Samsung M393A4K40CB1-CRC4Q 32GB PC4-19200 Ddr4-2400Mhz 2RX4 Ecc (total 128GB or 256GB)
* I do own 1TB of this memory so was really hoping to use. Unfortunately, that was not so much of an option with other miniATX X11 / 1cpu motherboards more popular for dedicated Truenas setups
Drives - 6 x Seagate IronWolf Pro 18TB NAS / Raidz2
* option to add as a second Vdev stripped with new drives the old Vdev from original setup 6 x WD RED Nas CMR (Raidz2), with the idea that it would potentially improve my read performance
PSU - SeaSonic Electronics FOCUS 750W 80 PLUS Gold
* This is something I did not purchase yet
APS - CyberPower CP1500PFCLCD PFC Sinewave UPS System
SSDs or NVMEs - ??? might add 2 x mirror depending what direction I end up with, open to recommendations for models
HBA card - not sure I would need it at the moment, if things change will pick up LSI SAS 9300-8I SAS3008
* would this be one of my best bets for an HBA card?

Intended use case:
- Truenas Scale setup
- home storage server solution
- serving smb share for 2 x workstations (mostly videography and photography projects)
* one workstation is a Hackintosh
- backup for windows, macos devices, other servers in my home lab (linux based)
- try to max of 10GB ethernet potential between Truenas box and workstations
- stream 2 x 4k simultaneous at least and move all my movie collection on Truenas box
- help with home lab VMs need (considering to run Truenas virtualized under ESXI, not set on this though. Need to do more research)

Posting this early since my power is flickering, due to a snow storm (don't want to lose this again)

Need help and advise with:
1. Motherboard Sata ports - should I expect to be able to use all 10 Sata ports before I need an HBA card? Any problems expected with the Raid Intel® C612 controller
2. CPU fan cooler - can you recommend some fan coolers that would work well with this?
3. Motherboard bought used on ebay / Should I expect issues with the attaching fan attaching bracket? (red something about this on other threads, forgot the details)
4. I would like to use the Memory I have, am I taking a big risk since I expect it not to be on the recommended list for this? It is ECC at least
5. GPU recommendation for 4k Plex transcoding. Should a Quadro P400 work for this?
6. How do you feel about adding the second Vdev with original drives 6 x WD RED NAS 3TB (raidz2) stripped with the new vdev for increased read performance
7. Virtualize Truenas or not. I understand ESXI is more stable than Proxmox. Also though some people do it with a good success rate it's a more risky setup. Is there a written tutorial for how to do it and set it up where the risks are at least considerably reduced? I did read the primer on that and some basic tutorials online
8. PCIe bifurcation - assuming this motherboard supports it
9. ARC - L2ARC - considering my general need mentioned, would I be able to benefit from adding a read and write cache ( general feedback I saw from reading primers and other threads is to invest more in Ram - which I own plenty)

I apologize, a lot of questions. I did read a lot but it makes me more comfortable to post this questions here. Note: power went down connecting to phone wifi to post.

Thank you in advance

 

[MisterE2002]

You can save some money using a 2308 chipset HBA.
1: You can use onboard SATA but why bother? Just buy 2 SAS-to-4-SATA cables.
4: if the specs match it should work. Use memtest anyway.

 

[ChrisRJ]

1: You can use onboard SATA but why bother? Just buy 2 SAS-to-4-SATA cables.

Why would I not want to use what is already there? Any particular reason for this recommendation?

 

[MisterE2002]

btw: 12 drives seem to fit in the Fractal. Did you consider buying a (second hand) supermicro rack server case?

 

[MisterE2002]

Why would I not want to use what is already there? Any particular reason for this recommendation?

Sure it will work fine. But all separate sata cables is a mess to work with. You have to buy different length cable and tie them up. The Topicstarter can at the point think ahead. And 12 drives are a bit a tipping point to a server rack imho (and just use a case with a expander so just 1 sas cable. The excellent sas guide of jgreco made me choose my hardware)

 

[Dave Hob]

You can save some money using a 2308 chipset HBA.
1: You can use onboard SATA but why bother? Just buy 2 SAS-to-4-SATA cables.
4: if the specs match it should work. Use memtest anyway.

I will look into that HBA, what is the main diff between the 3008 and the 9300-8i?
Watching the youtube link to get more information

Thank you

 

[Dave Hob]

btw: 12 drives seem to fit in the Fractal. Did you consider buying a (second hand) supermicro rack server case?

I did, and I have a large server rack to do it that it's not setup yet, but at the moment due to existing setup might be easier and more efficient for me to start with the tower. Defenetly something to consider once I experience that noise and heat become an issue. Also to make time to organize the server rack I have

 

[Dave Hob]

1. Motherboard Sata ports - should I expect to be able to use all 10 Sata ports before I need an HBA card? Any problems expected with the Raid Intel® C612 controller
2. CPU fan cooler - can you recommend some fan coolers that would work well with this?
3. Motherboard bought used on ebay / Should I expect issues with the attaching fan attaching bracket? (red something about this on other threads, forgot the details)
5. GPU recommendation for 4k Plex transcoding. Should a Quadro P400 work for this?

Anyone has any recommendation for CPU Fan Cooler and potential used motherboard bracket issue? Also confirmation if Quadro P400 would be a good fit?

Do you guys think that 750W psu is enough for this setup with 12 + drives? or should I go 850W?

Thank you

 

[Etorix]

PSU - SeaSonic Electronics FOCUS 750W 80 PLUS Gold
* This is something I did not purchase yet

Undersized if you ever go to 12 drives, plus pair of Xeon E5.

SSDs or NVMEs - ??? might add 2 x mirror depending what direction I end up with, open to recommendations for models

For the VM pool I suppose? Any pair of drives, sized according to your needs (aim for <50% used space for block storage).

Need help and advise with:

1. Motherboard Sata ports - should I expect to be able to use all 10 Sata ports before I need an HBA card? Any problems expected with the Raid Intel® C612 controller

Use the controller in AHCI mode and all is fine. @MisterE2002 has a point about using a rack case rather than the fiddly trays of a FD Define and cable management with breakout cables… but 4 of your ports appear to be from a SFF-8087 (MiniSAS) so the last point is moot for now.

2. CPU fan cooler - can you recommend some fan coolers that would work well with this?
3. Motherboard bought used on ebay / Should I expect issues with the attaching fan attaching bracket? (red something about this on other threads, forgot the details)

No. Use whatever suits you and fits in the case.

4. I would like to use the Memory I have, am I taking a big risk since I expect it not to be on the recommended list for this? It is ECC at least

I wouldn't expect any issue with any RDIMM module from Micron/Samsung/SK Hynix.

6. How do you feel about adding the second Vdev with original drives 6 x WD RED NAS 3TB (raidz2) stripped with the new vdev for increased read performance

It will work, but performance will not double due to the large imbalance between vdevs.

7. Virtualize Truenas or not. I understand ESXI is more stable than Proxmox. Also though some people do it with a good success rate it's a more risky setup. Is there a written tutorial for how to do it and set it up where the risks are at least considerably reduced? I did read the primer on that and some basic tutorials online

If you're not comfortable with that, it may be easier and safer to run TrueNAS bare metal and run VMs from it.

9. ARC - L2ARC - considering my general need mentioned, would I be able to benefit from adding a read and write cache ( general feedback I saw from reading primers and other threads is to invest more in Ram - which I own plenty)

SLOG is NOT a write cache, and is only useful if some of your workload absolutely requires sync writes. No sync writes = no SLOG.

Some insights into SLOG/ZIL with ZFS on FreeNAS

they (all of a sudden) care more about the "speed" (exactly latency) than about the (small) possible data loss You can work with this. The lack of power loss prevention (PLP) on the drive is (or rather will be) directly related to the poor SLOG performance; most drives are very slow if you...

www.truenas.com

With 128-256 GB RAM—and possibly up to 1 TB—you're unlikely to benefit from a L2ARC. Build, use, measure and then assess.

 

[Dave Hob]

I just removed the CPUs from my machine and to my surprise I was wrong, cpu model is E5-2699 V4

this is getting for sure overkill for a Truenas box. Most likely that I need to look more into the ESXI virtualized direction, will see.

 

[Dave Hob]

I received the motherboard today and trying to plan the install

The motherboard has 6 regular Sata 3 ports
+

A vertical S-SATA connector,
supported by the Intel SCU chip, offers four S-SATA 3.0 connections. These SATA
ports provide serial-link signal connections, which are faster than the connections
of Parallel ATA. See the table on the right for pin defi nitions

I am not able to find a S-Sata cable for that, unless I am missing something

Attaching some photos from manual:

Thank you

 

[Dave Hob]

It also has this:

SAS Ports (X10DRH-C/CT Only)
Eight SAS ports, supported by the LSI 3108 SAS controller, are located at JS39
on the motherboard. SAS 0-3 and SAS 4-7 support RAID 0, 1, 5, 6, 10, 50, 60
(Hardware RAID + 2GB cache) from the LSI 3108 with optional CacheVault available
for system data protection. (Note below). See the layout below for the locations
of SAS connectors.

Though I read mixed reviews on using this on another thread (older).