Show me the FreeNAS: SC846+X10SRi-F (ESXi/FreeNAS AIO)

AVSION

Contributor
Joined
Dec 28, 2016
Messages
128
My new upgraded toy just a promo for now will update more info and photos soon....


Part List

Hypervisor: ESXi 6.7 U2
SAN: FreeNAS-11.2 U5
Case: SuperMicro SC846 rev J 24 Bay (inc 10x 2.5“ HDD Tray in 4th Generation 3.5” HOT SWAP TRAY)
Motherboard: SuperMicro X10SRi-F
CPU: Intel Xeon E5-2683 V3 MHz 2 GHz LGA 2011 14x Cores 28x Threads
Cooler: Noctua NH-U9DX i4
HD Fan: 3x Noctua 120mm NF-F12 industrialPPC IP52 PWM Fan (Max 3000RPM)
Exhaust Fan: 2x Noctua 80mm NF-R8 Redux Edition 1800RPM PWM
PSU: 2x PSW - 920P-SQ
RAM: Crucial 2x 16GB DDR4 ECC 2400MHz + Crucial 2x 32GB DDR4 ECC 2400MHz total 96GB
Backplaine: SuperMicro BPN-SAS846-TQ
HBA Pool: LSI SAS 9305-i24 (Flashed to LSI 9305-i24 P16 IT)
HBA Datastore: LSI SAS 9300-i8 (Flashed to LSI 9300-i8 P16 IR)
Pool: 8x HGST Deskstar 4TB
ESXi Boot device: Samsung 860 EVO 250GB SSD
SLOG/L2ARC/Swap: Intel® Optane™ SSD 900P series 280GB NVMe PCIe
Datastores: 2x Samsung 860 PRO 256GB SSD , 2x Samsung 860 EVO 250GB SSD, Intel SSD 540 120GB



4AHBK2cvQK+5LKpW2Ad%lw.jpg
8RJGR0DdQXejY+YsSjiHWA.jpg
HTVsNuNoSa2DqEUNAj7ISA.jpg
kvx0mWToTvqWGqvRchtSvg.jpg
OHi8ScfzRv6n8znXqf5vQQ.jpg
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
You have your drives filled wrong.

Leave an empty row between the two filled rows.

Keeps them much cooler.
 

southwow

Contributor
Joined
Jan 18, 2018
Messages
114
I hope you'll give us some updates on the longevity of drives with the noctua fans. I'd considered this route, since they work well in desktop enclosures, but have heard cautionary tales about their use in server chassis where moving CFM's of air and maintaining negative pressure is critical. I don't buy it all, but I'm too big of a chicken to risk data testing it, lol.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Well then, let me reassure you, the Noctua are consumer grade fans intended for gamer workloads, where the fan lifetime isn't really a big factor. The workload necessary to create the air pressure differential in a 4U-24bay chassis is very high, and you should only use fans designed for the purpose. There's a reason that these fans aren't 39 cent Shenzhen specials.
 

AVSION

Contributor
Joined
Dec 28, 2016
Messages
128
I hope you'll give us some updates on the longevity of drives with the noctua fans. I'd considered this route, since they work well in desktop enclosures, but have heard cautionary tales about their use in server chassis where moving CFM's of air and maintaining negative pressure is critical. I don't buy it all, but I'm too big of a chicken to risk data testing it, lol.

I have been using the HGST drives on a desktop chassis for 3 years now, fan noise levels is a major issue as the server is in the same room we work. stock chassis had 2x 120mm on the front for the HDD and 1x 120mm for the exhaust. all stock chassis fans where non PWM except the CPU fan that were controlled by the IPMI(BMC). HDD temperature was around 38˚-40˚ (C) which is a bit over. in general i didn't have issues yet as result of the fans setup i was using, however i did had error with one drive (can't recall if it was the same one) once in few months but when i rebooted the FN the issue was gone, so i can't tell if it was an software or temperature issue. looking back on that setup wasn't idle and i was bit pushing the HDD.

This new setup has proper chassis designed for data centre where noise levels are not an issue compare to the cooling. i have removed all the stock chassis fans and used PWM Noctua 3x 120mm "industrial grade" for the HDD, 2x 80mm for the exhaust and 2x CPU/Cooler. i also replaced the stock PSW - 902-IR to PSW - 920P-SQ which made a huge difference with the noise levels. in addition i use now STUX script to control the funs. temperatures now are between 30˚-36˚ and noise levels are comfortable to work next to.

The funs are not the only factor that can affect the cooling of the HDD and the system overall and actually there are many. for example bad cable management will block the airflow which will cause the hot air spinning inside the chassis for longer rather exiting, using a script will maintain the HDD and CPU temperature, blocking holes in the chassis to allow air flow only through the HDD will increase cooling, having GAP between the rows (if its still empty) as @jgreco suggested will avoid the HDD heating each other and so on.
 
Last edited:

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Great work on the setup and posting photos.

One small point... labelling drives with their serial numbers is the right way to do things... the way you have it, you could find yourself pulling the wrong disk after a reboot and the system deciding a different assignment order.

You may also want to look out for airflow on your middle card there... looks like the i8... it has cards either side that get hot and only those few poxy holes at the end... with only one of the drive wall fans blowing at it, you may eventually find a problem there.

I'm not sure if you can space them out, but if your slots allow for it, I would recommend doing so.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Top