Build advice double checking some assumptions

Alphonse

Cadet
Joined
Sep 11, 2022
Messages
9
Hello, Long time lurker, first time builder.
I am double checking things before I commit to my first actual NAS build.

Some background on use case
I have a media server with about 15tb right now, but its dirty, and not in any form of raid. Not good.
I also run a variety of servers some for managing the media, home assistant, and plex, others for Running Ansible Automation Platform for testing for work/learning, which is 6 vms all said and done. I've hit a memory bottleneck on my old ryzen with 64gb, So I was looking at TrueNas Scale as I could get my data in a better place, run the VMs and plex on a better system, and have room to run things like Openshift on VMs and learn there as well.

So the proposed setup is a
Motherboard Asrock Rack ROMED8-2T/BCM
CPU: EPYC 7302 or better if I can find a good deal
Memory: NEMIX RAM 256GB (4x64GB) DDR4-2400 LRDIMM 4Rx4
CPU cooler: Freezer 4U SP3
Case: Fractal define 7 xl
Drives 6x Seagate IronWolf NAS 12 TB 3.5" 7200RPM Internal Hard Drive in raidz2

I know that its not fully loaded out, and I can add more memory down the line, or even higher.

So A few questions that I haven't been able to answer.
1. The minSASHD ports, Are these good to run off of, or should I get an LSI, how about the occulus ports converted through cabling/things to SATA. I know some posts were saying make sure it runs in IT mode for HBA cards and I remember reading about avoiding some Motherboard connectors so I wanted to see if people knew of an issue.

2. CPU cooler, do you think its enough, have contemplated getting a water block and doing a custom loop, but that I haven't done before either.

3. Cooling Putting more drives in, anyone have experience with if push out of case should create enough airflow to keep a server motherboard cool?

4. The drives I have heard that there are performance issues with various sizes for Virtual machines, vs Plex media serving. Should I have a separate pool for each? Even searching on the internet and here I have not found a decent answer. should I do 4 or 5 drives for a certain pool? I know to stay away from some of the WD Reds due to SMRs

5. With using the it for VM's, Should I get a SLOG, I have heard intel Optanes are good for this, and faster = better, so something like this 32gb optane should do?

thanks in advance for looking this over.
 
Last edited:

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,906
4. The drives I have heard that there are performance issues with various sizes for Virtual machines, vs Plex media serving. Should I have a separate pool for each? Even searching on the internet and here I have not found a decent answer. should I do 4 or 5 drives for a certain pool?
How does fit with your statement about 6 drives in RAIDZ2, which is a reasonable configuration for storing a media library, if files are accessed sequentially.

HDDs are not so good for VM workloads (unless you have many mirror vdevs). Instead I would use a mirror of SSDs as a starting point. Whether SATA SSDs are good enough or not, depends on your requirements.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,151
Are you sure you can put 6 3.5'' HDDs in that case? From the pictures I don't see many mounting points or bays.

Anyway, more than the motherboard I would be concerned by the HDDs temps, especially since the case looks like it has few holes for air to enter.

I also would not put water in a system that is going to run continuously and unattended.

You can decide to not use that optane if you use an UPS.
 
Last edited:

Alphonse

Cadet
Joined
Sep 11, 2022
Messages
9
Are you sure you can put 6 3.5'' HDDs in that case? From the pictures I don't see many mounting points or bays.
The case can mount 16 drives before taking up fan space, I was not worried about airflow as the front of the case can push air through them, it was the air past the HD that I was worried about. Was thinking of putting some Push on the top of the case and the back,
adding image to show fans and then drive cages and the place for 12 more going up.
case_fans.PNG

How does fit with your statement about 6 drives in RAIDZ2, which is a reasonable configuration for storing a media library, if files are accessed sequentially.

HDDs are not so good for VM workloads (unless you have many mirror vdevs). Instead I would use a mirror of SSDs as a starting point. Whether SATA SSDs are good enough or not, depends on your requirements.
I think this is what I was looking for, that I should have a separate pool of SSDs to store VMs, because they don't run well on hdds without decent mirroring. This is looking like another rabbit hole, but it gives me a starting point, as I was just going to throw it all on RAIDZ2, thinking it was sufficient.

Still thinking on the other questions, thank you both for taking the time,
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,906
The case can mount 16 drives before taking up fan space, I was not worried about airflow as the front of the case can push air through them, it was the air past the HD that I was worried about. Was thinking of putting some Push on the top of the case and the back, [..]
Are you sure you didn't mean pull? The airflow should be from front intake to rear exhaust. An additional exhaust at the top may or may not be helpful, as it can(!) cause turbulences that reduce the effectiveness of airflow.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,151
The case can mount 16 drives before taking up fan space, I was not worried about airflow as the front of the case can push air through them, it was the air past the HD that I was worried about. Was thinking of putting some Push on the top of the case and the back,
adding image to show fans and then drive cages and the place for 12 more going up. View attachment 58462

I think this is what I was looking for, that I should have a separate pool of SSDs to store VMs, because they don't run well on hdds without decent mirroring. This is looking like another rabbit hole, but it gives me a starting point, as I was just going to throw it all on RAIDZ2, thinking it was sufficient.

Still thinking on the other questions, thank you both for taking the time,
That's perfect then! Make sure to place the HDDs behind the intake fans in the front, that will be enough for them.
I would just put a single exaust fan in the rear, with the CPU cooler blowing in the same direction, to maintin positive pressure.
You might consider adding two more intake in the top area, at the rightmost spots in order to directly feed the CPU cooler.

What I was, and am, concerned about is the small area the air can enter in the front. I have a similar case for personal use, and I'm not sure the fans are eating enough air.
case_fans.PNG

EDIT: Instead of how I showed in the image, you should split them in 3 groups of 2 and leave an empty slot between them (1-2-empty-3-4-empty-5-6) for better airflow. If you are able to remove the optical drive slots, you can have empty space between each of them and even mount an additional intake fan.
 
Last edited:

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,906
The problem with the Fractal cases are the sleds on which you mount the HDDs. They cover the bottom of the disks completely and that reduces the effective heat dissipation surface by 50%. So if you are mounting the drives next to each other, like I did, you need some really beefy (and loud) fans. If you can leave every second drive slot empty, this may look different, though.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,151
Any reasons you choose the Delfine instead of the Meshify? It would be able to feed your fans with much more ease, it's easier to clean and it's cheaper.
 

Alphonse

Cadet
Joined
Sep 11, 2022
Messages
9
Any reasons you choose the Delfine instead of the Meshify? It would be able to feed your fans with much more ease, it's easier to clean and it's cheaper.
The problem with the Fractal cases are the sleds on which you mount the HDDs. They cover the bottom of the disks completely and that reduces the effective heat dissipation surface by 50%. So if you are mounting the drives next to each other, like I did, you need some really beefy (and loud) fans. If you can leave every second drive slot empty, this may look different, though.
Instead of how I showed in the image, you should split them in 3 groups of 2 and leave an empty slot between them (1-2-empty-3-4-empty-5-6) for better airflow. If you are able to remove the optical drive slots, you can have empty space between each of them and even mount an additional intake fan.
Looking back on it, I should have gone with Meshify, and may return this and go that route, I am not sure how I overlooked that. And good point on the Leaving space and doing the Intake, I was mistaken when I said Push, but that is what I was thinking , but did not verbalize well, would better to something like you put with the arrows.

I am looking to doing 2x2 mirrors of ssds for the VMS, I think the SSDS should do, as its mostly actions, and writing to a postgres server, which they already have a queue system to write, and the media VM is Sonarr/Radarr interacting with an external rtorrent server.

The Point 1 thing I haven't seen anyone with issues with those, was just hoping someone had done a board like it with the miniSASHDs and/or occulus that as I don't know how they will show up in the OS. I don't think its going to be an issue, but We shall see.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,151
The Point 1 thing I haven't seen anyone with issues with those, was just hoping someone had done a board like it with the miniSASHDs and/or occulus that as I don't know how they will show up in the OS. I don't think its going to be an issue, but We shall see.
Well... with a good IT-flashed HBA you can't go wrong.
As a side note I can't find the manual for that mobo, but it looks like you need to correctly set a jumper if you want to use both Mini-SAS and the two OCuLink.

EDIT: Found for the one witout the BCM.
Screenshot_1.png

Oh, and remember to choose a PSU of proper size.
 
Last edited:

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,112
SATA ports from Intel or AMD chipsets (or in this case the EPYC IO die) are fine.
 

Alphonse

Cadet
Joined
Sep 11, 2022
Messages
9
Well... with a good IT-flashed HBA you can't go wrong.
As a side note I can't find the manual for that mobo, but it looks like you need to correctly set a jumper if you want to use both Mini-SAS and the two OCuLink.
Oh, and remember to choose a PSU of proper size.
I just updated the above link to point to the non bcm one, but the newegg has the BCM one for sale.
And it looks like the m.2/2nd sata set and ocu share lanes.

In regards to Power supply I was looking at the bequiet dark rock 1000 watt, but am still mapping out how many drives It can support and things so I don't run out of space in the future, and may upgrade to the 1200.
 

Alphonse

Cadet
Joined
Sep 11, 2022
Messages
9
SATA ports from Intel or AMD chipsets (or in this case the EPYC IO die) are fine.
Thanks, it was more the whole IT flashed HBA things was thinking it might be an issue that the mobos have the same issues as the cards
 

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,112
"MiniSAS" does not imply the port can actually do SAS… It's just a (not-so-helpful) moniker for the connector. In this case, the ports just package 4 lanes of IO from the CPU, which may be PCIe or SATA but not SAS; there's no SAS HBA and nothing to flash to IT mode.
From the manufacturer's page, the OcuLink port is pure NVMe

Re:2 With such a large case, go for a Noctua NH-U14S TR4-SP3 rather than a 4U cooler for rackmount cases.

Re:5 For VMs, just go for a pair or NVMe drives (M.2 or U.2) in mirror. The need for SLOG depends whether you absolutely need sync writes. If so, Optane drives are good SLOGs but that applies to pure Optane (DC P4800x/4801x/5800x, 900/905p), not to SSDs with a tier of Optane cache.

As proposed, you have a huge case for just 6 drives and a motherboard with lots of I/O, most of which will not be used. The CPU is certainly very capable for running multiple VMs with lots of RAM, but isn't the whole setup oversized?
 

Alphonse

Cadet
Joined
Sep 11, 2022
Messages
9
As proposed, you have a huge case for just 6 drives and a motherboard with lots of I/O, most of which will not be used. The CPU is certainly very capable for running multiple VMs with lots of RAM, but isn't the whole setup oversized?
So in terms of the motherboard, up until yesterday I saw it cheap on newegg direct business, so it was more a deal in that regard, and the 6 drives was for now, Plan was able to scale it up to 18 drives with 2 more sets of 6 in the future as needed.
In terms of IO you are spot on, its likely overkill, but when I priced it, this was a few deals compared to a Threadripper or a last gen Ryzen, it left room for expansion in Memory if needed, and drives.

In regards to the Noctua, I looked at that, the Pipe placement is central, and it does well for Threadrippers, but is bad for Epycs as where the chips are not as central, so with the plate on the die aren't covered as well, the 4U cooler does a better job of covering the entire die, I don't have source, but this was noted from at least two places that brought it up, and it may have been corrected since then, but I didn't want to risk it.
 
Top