BUILD New FreeNAS Box (Supermicro X11 Edition)

[Morpheus187]

Hi

As I'm one of those Asrock hosed guys with a board that is probably soon gonna hit the dust, I decided to plan a new system.

The Hardware supply is a bit limited here in switzerland, especially supermicro stuff is hard to get, so I may have not choosen the optimal parts.

- Case
Supermicro 825TQ-R720LPB
- Mainboard
Supermicro X11SSH-CTF: LGA1151, E5-1200v5 ( 8x Sata onboard, 8xSAS3 LSI3008, 2x10G 10base-T )
- Ram
Crucial CT4K16G4WFD824A (4x, 16GB, DDR4-2400, DIMM 288)
- CPU
Intel Xeon E3-1245 V5 (LGA 1151, 3.50GHz)
-Disk
8x 6TB WD Red ( already existing )


My thoughts about this setup:
The board offers 10 Gig Ethernet for future connectivity upgrade and an LSI3008 SAS controller with 8 Ports. That allows me to use my existing disks with the option of additional SSD's for L2ARC

The ram is compatible on the crucial homepage, I've chosen 64GB because the current 32GB seems to be quite on the edge, my ARC Hit rate is at about 50%

The case is basically what's available here for a reasonable price. The backplane is not sas3 compatible but I don't need that performance for 8 magnetic discs.

I'm grateful for any thoughts or warnings if something doesn't match up.

 

[joeschmuck]

The ram is compatible on the crucial homepage

Is it on the Supermicro QVL? This may be fine RAM, I alwasy question this kind of stuff.

I've chosen 64GB because the current 32GB seems to be quite on the edge, my ARC Hit rate is at about 50%

What are you using yoru system for? If it's a lot for Plex or similar then of course ARC hits will be lower. Just trying to jkepp things in perspective because your hit rate may remain the same. Of course I would buy 64GB as well.

Crucial CT4K16G4WFD824A (4x, 16GB, DDR4-2400, DIMM 288)

The RAM speed for the board is limited to 2133 MHz so if you can save a few bucks, buy the correct speed RAM. And I didn't look at the Crucial RAM specs, sorry I'm running out of computer time today, got a Beer screaming my name.

Thats all I can offer up for now.

Cheers!

 

[Morpheus187]

Thank you for the advice.

The ram is not on the Supermicro QVL but Crucial lists this board as explicitly supported. And it's the only ram I found here in Switzerland, even after asking different shops.

I use the system as filestore and vmware datastore and daily backups.

Enjoy your beer :)

 

[darkwarrior]

Hi there,
I just double-checked on the crucial site and could not find the specific reference you listed, but if you took it from there, using the compatibility tool, you're safe and granted it will work. So no worries :D

 

[Morpheus187]

As reference, I took this Link to verify ram:
http://www.crucial.com/usa/en/x11ssh-ctf/CT7982777

 

[darkwarrior]

You're browsing Crucial on the US based site.
If you will want delivery to Europe, you should take the "EU" shop ;)

Personally, I would have chosen this reference for the sake of having a CL=15 instead of 17.
But that will not give you any particular advantage in a server environment.
You will just feel cooler

Just kidding either one is fine :p:D

 

[Morpheus187]

I just checked the compatibility on the US site, I try to buy it from a local dealer here in Switzerland and this piece is the only one available here, so I'm stuck with that
I don't even try to buy from abroad because customs here really sucks, they add random fees and stuff so it could cost more than 50% more.

 

[darkwarrior]

Ah yeah, right.
I forgot how funky the swiss customs can be :p

 

[Mr_N]

can't you just take a short trip to france or uk and buy your things and return home?

I bought both my current mobo's from US as there's somewhat of a poor supply here too :)

 

[Morpheus187]

Just as a heads up, I'm still waiting for the RAM to be delivered, it's now over a month, I guess this type of RAM is just not available here. I'll wait for 2 months max and then I will cancel the whole order and re asses my situation, maybe I can get another board and cpu that supports DDR 4 ECC RDIMM rams and not only UDIMM.

 

[Morpheus187]

Just an update:
I finally got the ram! yay
System is already installed with a test installation, I'm still waiting for the sas breakout cables, they were obviously not part of the mainboard :(

The downside is.
Memtest86+ shows ram errors after 25 Minutes of runtime on test 7 :(
But Memtest86+ 5.01 is a bit old and it detects my ram as ddr3, so I downloaded Memtest86 7.2 and its now 9 hours into the test circle and still no errors. So I GUESS that Memtest86+ 5.01 has problems with ECC DD4 Ram, the error is always in the 0.0MB to 0.8MB zone. Of course I will not just hope for the best, I will do additional testing with ZFS on a test system to force an integrity failure on the data and even on the live system I will do some tests before I call it OK.

 

[joeschmuck]

If you run into problems with stable RAM, check your DDR speed. But it sounds like the old MemTest86+ wasn't cutting it.

 

[Morpheus187]

Just a quick update from my side

I finally got the 8643 Brakeout cables and installed 8 Test SATA-Drives.

The only issues I still have is that I don't have any backplane lights at all.
I've switched the SAS-825TQ backplane to SGPIO mode and when I connect just the SATA cables from the Brakeout cable, all disks are present, if I connect the SGPIO cable that is included in the brakeout cable, I only see 3 disks of 8. It seems that they have odd enclosuere ID's starting with 28 and with duplicates. That even happens when I connect the SGPIO cables wrongly ( Inverted )

If I connect the SGPIO with seperate cables to the SGPIO ports on the mainboard, I see all 8 disks but the SGPIO cable makes no difference, I still have no lights.


When I start up the system, all LEDs light up RED for about 1 seconds and then they go out.


The backplane has a seperate 10 Pin connector for SATA activity lights, but I don't have any connector on the mainboard for that.

I think the backplane doesn't really work with SATA or with the controller / board. I guess I have to revert to creating stickers with serial number of the hard disk. I can use sas3ircu to tell the backplane to light up a bay, the command succeeds but nothing happens.


Thanks in advance if anyone has an idea

 

[Morpheus187]

Next update

Productive disks are installed and activity leds are now working ( you can't check activity led without activiy :( Layer 8 error )

SGPIO is not connected so far, but everything seems to work, just locating drive bays don't work, but I have printed out serial number stickers on the disks.

 

[joeschmuck]

You know the WD drives have a sticker on them already, it just may not be within your view depending on how you have them mounted.

 

[Morpheus187]

Next Update:
The System is running fine so far. I redid the cabling a bit to improve airflow. I now running a scrub and closed the doors to the room to increase temperature to simulate "summer".

Temps are a bit uneven with a hotspot at the bottom drives, they don't get a really good airflow. I'm thinking about buying a thermal pad to put in under the disks so that they touch the chassis. At the moment they hover about 1 mm above the chassis in the tray, with physical contact the case should act as radiator.

Disk temps are now from 39 to 42 degree C range under scrub load. Without load they stabilize around 35 to 39. The only "data" I got about temperature is the google study which is from 2007. Backblaze found no correlation between hdd temp and failure rate. Operating range is 0 to 65 according to Western Digital ( I'm using 8x6TB RED )





I'm also trying to write a little bash script to control the fans via IPMI.

 

[Stux]

I'm also trying to write a little bash script to control the fans via IPMI.

Or, you know, use mine ;)

https://forums.freenas.org/index.php?threads/script-hybrid-cpu-hd-fan-zone-controller.46159/