Building a "Reasonably Quiet" SSD Rack Mount Server for Office and Home use

[bollar]

Everything's here but the RAM. I wound up buying from one of those outfits that you think is in the US, but is actually in China, so I'm dealing with USPS saying "Departed Shipping Partner Facility, USPS Awaiting Item," so who knows when I'll actually get it.

In the meantime, I'm thinking about drive layouts. Looking at SCALE Test (in my signature), iostat says my data VDEVs are 90% read / 10% write. I also have a metadata VDEV that includes files up to 4MB. That VDEV has 260GB of data on mirrored 1TB SSDs and it's the opposite -- 10% read / 90% write. Same for my Applications pool -- 10% read / 90% write.

I doubt any of us using the system would actually notice any performance differences, so this is partially educational for me. I'm thinking about the following configuration:

The X12STH-F motherboard has eight SATA ports. I'll do:

- Mirrored 32GB SATA DOMs for boot
- Striped & 3-Way Mirrored 2TB Samsung Pro 850s for the metadata and "small files." I might include small files up to 8MB. A lot of my thinking on this is to reduce the writes on the data drives.

The 216BE1C-R920LPB chassis has twenty-four SAS3 ports -- I'll go "cheap" on those and fill with 3 VDEVs of RAIDZ2 8TB Samsung 870 QVOs. Bought them from four different vendors to maximize my chances of having different lots.

I'll put the Applications pool on mirrored 2TB Samsung 980 Pro NVMe M.2 placed on a PCIe dual M.2 adapter card on the motherboard's PCIe 3.0 x 4 slot.

That leaves the 360GB Optane on the motherboard's M.2 slot. I have negligible sync writes and I don't see using it for L2ARC. Maybe a scratch or transcoding drive, or take it out and save it for another project.

And then next year, when we move everything back into the new server closet, I'll look at lobotomizing SCALE Test and cascade the new HBA to the drives on it. Seems like that would save quite a bit of electricity by not having that dual Xeon motherboard cooking 24/7.

That's it -- idle thoughts while I wait for RAM...

 

[bollar]

The RAM finally arrived and I was able to start the server and load TrueNAS. It's not silent, but it is reasonably quiet and got a thumbs up from all of the affected parties. There are some instances where the fans ramp up briefly and it's quite loud then, but it settles back after a few seconds. I guess we're hitting some threshold that can be adjusted.

Haven't tried any benchmarking yet, but may look into that if I get bored.

 

[Davvo]

Looks lovely, what is the model of those boxes using the expansion slots on the bottom right of the photo?

 

[bollar]

Looks lovely, what is the model of those boxes using the expansion slots on the bottom right of the photo?

Those are just generic SSD holders -- there are lots on eBay:

PCI Slot 2.5Inch IDE/SATA/SSD/D Rear Panel Mount Bracket Hard Drive Adapt R4L5
https://www.ebay.com/itm/374541029585

Anyway, it deals with my SSD mounting OCD...

 

[Davvo]

Anyway, it deals with my SSD mounting OCD...

Understandable.

Monitor the temperatures of that area since HBAs are known to run hot sometimes.

 

[joeschmuck]

what is the model of those boxes using the expansion slots on the bottom right of the photo?

You know, I had the same question when I saw the photos. Glad you asked. That is a good way to find extra storage locations. Much safer than velcro

 

[bollar]

Now that we've been living with the server for a few days, we notice there are some instances where the chassis fans spin up to assist with CPU cooling. These brief periods are not "Reasonably Quiet" I know I caused this -- the chassis has three fans that I have connected to FAN2, FAN3 and FANA. The CPU cooler is on FAN1. I have the shroud installed and the FAN2 & FAN3 fans are inside the shroud with the CPU.

I'm aware of the difference between the CPU and PWM fans. I've considered moving all of the chassis fans to FANA, but I don't want to do this randomly. I've read some of the threads here about fans and I've reviewed this script that looks pretty cool: https://github.com/khorton/nas_fan_control

Has anyone done a deep dive on the things I should be thinking about before trying to reconfigure? I'm happy to be pointed to existing threads either here or elsewhere.

 

[Davvo]

You should look at the following first.

How To: Change IPMI Sensor Thresholds using ipmitool

Many server motherboards, including the Supermicro X10 series, have their fans monitored by the BMC, which automatically attempts to recover stalling (of the motor, not in the aerodynamic sense) fans by spinning them up to maximum (PWM set to...

www.truenas.com

 

[bollar]

I have adjusted the fan thresholds, which has eliminated the pulsing -- what I'm noticing now is that the CPU is 80 degrees C or more most of the time. CPU fan is running at full speed at 1900 RPM and now the two case fans attached to the CPU are close to full speed at 6500 RPM.

I wonder if the Arctic CPU cooler just isn't enough for the Xeon? Might need to try the stock Supermicro Fan and see what happens.

On the good side, the front of case drives are mostly around 30 degrees C and the ones at the back 36. Seems pretty good to me.

 

[joeschmuck]

Oh yea, those are not good CPU temps at all. A few questions:
1. Did you install the cooler correctly? If there was plastic on it, did you peal it off? Did you have to apply your own thermal compound?
2. Are you running any VM's, jails, etc when the CPU temp is 80C?

I don't know what your experience level is with this kind of thing.

 

[bollar]

Oh yea, those are not good CPU temps at all. A few questions:
1. Did you install the cooler correctly? If there was plastic on it, did you peal it off? Did you have to apply your own thermal compound?
2. Are you running any VM's, jails, etc when the CPU temp is 80C?

I don't know what your experience level is with this kind of thing.

I've done this enough times to know that it's critical to get it right, but not enough times to be confident that I get it right each time... So, thanks for the tips!

1 -- the thermal compound was pre-applied and I confirmed the plastic shield was off. The interesting thing to me was that I had some fiddling to make it feel like it flat against the CPU, so maybe it's not mounted correctly. I have thermal compound, so I can take it off and redo it.

2 -- Yes, it's about 40-50% with various things I'm playing with. At idle, it's around 40 degrees.

I also verified the TDP for the Intel E-2388G is 95 degrees and the Arctic Freezer 11LP is 100 degrees -- so right at the limit -- but I also haven't (yet) found a fan with a higher TDP that would fit in the 2U case.

 

[bollar]

BTW, @jgreco if you haven't gotten to your E-2388G project yet, hardware transcoding does work with this chip. I was able to set up a Plex instance using the TrueCharts app and Plex was able to see and use Intel Quick Sync (once I figured out that I needed to enable QSV in the bios).

 

[Davvo]

There could be a mounting or contact issue, I would pull the plug and reinstall the cooler.

 

[joeschmuck]

but I also haven't (yet) found a fan with a higher TDP that would fit in the 2U case.

I think the Intel Fan would do it. Mine came with the fan and it's nice an fairly small. Even my new AMD came with a nice Wraith cooler, still small compared to the coolers of a decade ago.

I do suspect the heatsink is not mounted properly. I'm not sure the exact mounting style you have for that cooler, I suspect four spring loaded screws or four spring loaded push pins. I prefer the screws myself. I've had one pish pin pop out on me once and I could not get it to stay installed. That really sucked. Not that this is your problem, but make sure all the legs of the heatsink are secured with proper force before removing the heatsink if you can. You might be able to tighten a screw and make all things better.

Run Prime95 for at least 2 hours, monitor the CPU temps. I like my maximum temp to be about 75C during this test, but lower is better. I think my current CPU runs at a little over 60C maximum under full load. It took me to learn how to apply the thermal compound properly. I like the drop in the middle then carefully lay the heatsink on top to spread the compound out. There have been actual studies on what works best, and this is the better one in general. It help reduce low contact area or bubbles in the thermal compound. I use to spread the thermal compound out but then I would run into the low 80's C so I had to figure out where I could improve.

 

[bollar]

Took off the heat sink and replaced the paste. Maybe the original was not the best job, but it looked like the cpu had complete coverage. Temps are pretty much the same before and after. A couple of things I learned:

1. The fans connected to the CPU header are running at 75-80% when the CPU is at 70-75 degrees C. Switching the BIOS fan profile from Optimal to Full Speed can lower the CPU temp to 60-65 degrees C.
2. HTOP shows Plex running all sorts of processes and turning Plex off can lower the temp even further into the 50s. I don't need Plex on all the time, so that's easy enough.
3. The peripheral zone fan is running at a relaxed 35% and the peripheral sensor temp is 32 degrees C.

I haven't tried Prime95 yet -- spinning up Windows here is painful. If I could find a live CD / USB of something comparable, that would be easier.

I think there's an opportunity to lower CPU temps (and therefore fan speeds) if I can find a cooler with a higher TDP -- hard in the limited space I have in the case.

 

[Davvo]

60-65 degrees is better.

If you want to change but remain in the consumer-grade solutions, take a look at the AXP120-X67: total height is 67mm, which is 7mm higher than the supermicro's 2U cooler, see if you can fit in there. Otherwise you have to go smaller.

Anyway, always keep in mind that top-down coolers are less efficent in such closed spaces: consider a standard 2U tower cooler.

 

[joeschmuck]

I haven't tried Prime95 yet -- spinning up Windows here is painful. If I could find a live CD / USB of something comparable, that would be easier.

The Ultimate Boot CD is what you are looking for. Very good live cd to run a CPU Stress Test and MemTest.

It's probably the photo but the paste looks thick. Maybe too much? But you said earlier that the heatsink came with the thermal compound already applied so it "should" be good.

So when you had Plex running, was it transcoding which is why the CPU temps were so high? Or was it basically doing nothing? If that is the case, you have something odd happening.

The 60C to 65C is fine, provided you were running Prime95.

I still think it's the CPU heatsink, maybe the thermal compound, maybe the heatsink, I can't know for sure. Wish I lived nearby to drop over and take a look at it. Now an out of the box suggestion is to try TrueNAS Core if you were running Scale. See if you find lower CPU temps there. I know Scale might be what you want but it's only for troubleshooting. Of course the UBCD would help diagnose that too. Use your IPMI to examine your CPU temp and fan speed. I'm running at 28C with a fan speed of 1000RPM.

I hope you find the source of the problem.

 

[ChrisRJ]

It's probably the photo but the paste looks thick. Maybe too much?

That was also my immediate thought. For a CPU of this size the amount of thermal paste should roughly be as much as a single pea.

 

[jgreco]

That was also my immediate thought. For a CPU of this size the amount of thermal paste should roughly be as much as a single pea.

Having done some hundreds (at least) of CPU repastes, I would note that some pastes such as Arctic Silver 5 look a bit like that when pulled off. but that specific pic also looks like it may be a bit too thick. I feel a pea's worth is too much but then again that's what you need if you don't hand spread the paste. I am a proponent of the "credit card" spread; don't actually use a credit card though since they are usually full of hand oils or other contaminants. A used-once hotel keycard having been cleaned with isopropyl and held in a ESD gloved hand is fine. I hear people talking about air bubbles but never find this to be a problem. This allows you to get a very fine skim coat of paste on the surface, which in my opinion is really a lot more valuable in getting a good paste layer attached to both sides. You will find a pea's worth is probably too much unless you have a really big CPU, if you are hand spreading.

It is worth it to try several pasting and cleanup cycles when learning to paste a CPU. AS5 is nice because Arctic makes their ArctiClean product as well. Make sure you use lint free cloths or microfiber cloths when cleaning the paste and don't be lazy or stingy. Anyone can do a crappy job of pasting. But it doesn't take a huge amount of effort to learn to do an awesome job of pasting either. Just patience and practice. Take it off several times and look to see how it went. Be very gentle. Paste and supplies are cheap.

 

[bollar]

The Ultimate Boot CD is what you are looking for. Very good live cd to run a CPU Stress Test and MemTest.

Thanks -- will check it out.

It's probably the photo but the paste looks thick. Maybe too much? But you said earlier that the heatsink came with the thermal compound already applied so it "should" be good.

Maybe, but yes, it was pre-applied.

I used the "spread" technique and my temperatures are ever-so-slightly lower -- maybe 1-2 degrees over the past two days. I think the thermal paste with this fan is now as good as it gets.

If you want to change but remain in the consumer-grade solutions, take a look at the AXP120-X67: total height is 67mm, which is 7mm higher than the supermicro's 2U cooler, see if you can fit in there. Otherwise you have to go smaller.

Thanks -- I'll check that out, along with some other options and report back.