BUILD Supermicro X9SRL-F Build Check

[mchatterton]

I have been reading this forum for a while now and I’m hoping to get your thoughts on my FreeNAS server rebuild.

Background

I built my first ZFS server 3 years ago when I knew nothing about FreeNAS. After learning a lot more about FreeNAS and ZFS (thank you cyberjock, jgreco, and many, many others), I’ve come to the conclusion that my current underpowered E-350 with non ECC ram is just about the worst combination possible for ZFS. ​

So it’s time to rebuild the server properly. ​

Intended Use

• Home use
• Backup target for up to 6 machines (Windows and OSX)
• Long term file storage (with proper off site backups )
• Media library to maximum 3 computers at one time
• Maybe add Plex at a later date if I get adventurous. Maybe.

Proposed Build

Motherboard: Supermicro X9SRL-F​

CPU: Xeon E5 1620V2​

Memory: Kingston 32gb (16gbx2) (likely Kingston KVR16LR11D4/16HA). More 16gb sticks to be added as the pool size increases.​

SAS Controller(s): M1015(s) flashed to IT mode​

Hard Drives: vdevs of 6 drives each in RaidZ2. Currently 2 vdevs in one pool of 12 drives (2tb and 3tb drives, total 17tb useable now). I would like to continue to add vdevs (in batches of 6 drives, 4tb or greater drives) to that pool up to, eventually, a total of 36 drives (for a pool size of around 80tb to 100tb).​

Reasoning behind the Proposed Build

The overall goal is to build a server that will last me for the next 5 to 10 years (with drive/memory/M1015 additions) and not require a complete rebuild.​

The X9SRL-F and E5 1620V2 give a lot of expandability in terms of pcie slots (for SAS controllers or a 10gbe NIC) and memory. Due to the varied nature of the clients that are connecting to the server, I’m stuck with CIFS, so the higher clock speed is preferable to more cores. Adding to the pool in batches of “only” 6 drives is primarily to spread the cost out. No ZIL or L2ARC drives since this server is for home use, with the most intensive application being media streaming, and since I like the philosophy of having fewer parts means there is less to break. ​

Questions

1. Does anyone see anything immediately wrong with the above build?
2. I’ve read about potential issues with SAS expanders and ZFS and I can't find a conclusive answer if the issue was disproven/fixed. Based on that, I’d prefer to avoid the whole issue altogether and go with 4 M1015s. Is there an issue with running that many SAS controllers on one motherboard?
3. Are there an issues with having a single pool with 36 drives in it?
4. Does anyone have experience flashing M1015s in the X9SRL-F motherboard?
5. Philosophical question - What are your thoughts about riding out this hardware until the Xeon E3/E5 refresh come out in the second half of 2014 (making sure, of course, to have very good and completely independent backups)?

Thank you.

 

[cyberjock]

1. Does anyone see anything immediately wrong with the above build?
2. I’ve read about potential issues with SAS expanders and ZFS and I can't find a conclusive answer if the issue was disproven/fixed. Based on that, I’d prefer to avoid the whole issue altogether and go with 4 M1015s. Is there an issue with running that many SAS controllers on one motherboard?
3. Are there an issues with having a single pool with 36 drives in it?
4. Does anyone have experience flashing M1015s in the X9SRL-F motherboard?
5. Philosophical question - What are your thoughts about riding out this hardware until the Xeon E3/E5 refresh come out in the second half of 2014 (making sure, of course, to have very good and completely independent backups)?

Thank you.

1. Looks good to me.
2. I'm using an Intel SAS expander(friends have them too) and nobody has ever had a problem. I've also used an Areca expander and had no problems there. I think the conservative answer is "there is room for potential problems". I've been using SAS expanders for years.. no problems.
3. No. As long as you aren't doing a 36-drive vdev then all is fine. ZFS is supposed to scale up accordingly.
4. I haven't flashed with your board exactly (I have X9SCM-F) but you can expect that you will need to flash it with the EFI tools.
5. FreeNAS isn't exactly a "high load" OS. If you need it now I'd just get it now. If you wait until Q3/Q4-2014 then why not wait until 2015 or 2016? Regardless of what you buy, technology marches on.

 

[mchatterton]

Thanks cyberjock!

I have read that many of you have been using expanders for years (with sata drives) without a problem. This is the thread that had me worried http://hardforum.com/showthread.php?t=1548145 though. If I can ignore the issue all together by using multiple M1015s then that may be the safer way to go (and it may cost less in the end). Have you seen any issues with using up to 4 M1015s in the same build?

On the hardware side, my thoughts are the same as yours - there is always something better on the horizon, so you might as well buy what you need now. I guess I am more hoping that the E3's will get a bump in memory capacity and remove my main reason for needing a E5.

Thanks again!

 

[jgreco]

So, like, you did your reading and you seem to have made good choices. The hop up to E5 is a bit pricey and may burn more watts than a smaller E3 system. I haven't walked through compatibility for your parts since I would guess that you've already done so. It'd be nice to see 10GbE on the board but there just isn't a smart choice for that. The X9SRH-7TF has 10GbE but also shortcomings and your selection can at least "fix" it with a card. Don't know that anyone's actually run four M1015's on a board. But basically you came in here having done your homework and all we can really do is say "looks like it'd be great."

 

[Dusan]

Due to the varied nature of the clients that are connecting to the server, I’m stuck with CIFS, so the higher clock speed is preferable to more cores.

A note here: Samba is not multithreaded, but it will spawn a separate process per connection. The higher clock speed instead of more cores recommendation applies to home scenarios with only few clients. However, if your use case includes many parallel CIFS clients/transfers you will benefit from more cores.

 

[jgreco]

Without 10GbE I would not expect it to be an issue. A fast quadcore vs a quartet of gigE ought to be pretty competitive.

 

[mchatterton]

Thank you jgreco and Dusan.

jgreco - I agree on the E5s being a bit on the spendy side but the expandability isn't available in the E3s yet and I don't want to have to rebuild in another 3 years because I've outgrown my server again. If an E3 could take more memory, I'd be all over that. And I'd love to have a 10GbE NIC but it's going to be a while before the prices come down far enough for me to buy a 10GbE switch, so it wouldn't be utilized for a few years.

Dusan - Agreed on the Samba. My worst case load on the system will be streaming video to 3 clients, so I'm not going to need those extra cores but there are definitely other people out there that would benefit from them.

 

[vikingboy]

hi mcchatterton,
Im looking at a similar build to this and wondered if you had any thoughts now you have hopefully built it and put it into production use?
Ill be using the x9srl-f, a 1620v2 and 2 * LSI 9211-8s with 10gbe.
Im not 100% sure on drives and freenas setup just yet but need something in the region of 50TB.

Im also still undecided to go with 64GB UDIMMs or allow for further extensibility and put in 64GB of RDIMMs which would permit expansion to 256GB theoretically.

thx in adv for any feedback

 

[mchatterton]

Hi vikingboy,

Unfortunately, I haven't completed this build yet. I'm still planning to but haven't finished saving up for it yet.

Depending on your usage profile, 64gb should be plenty for 50tb but, in my experience, I've never regretted leaving room for future expansion.

 

[Yatti420]

hi mcchatterton,
Im looking at a similar build to this and wondered if you had any thoughts now you have hopefully built it and put it into production use?
Ill be using the x9srl-f, a 1620v2 and 2 * LSI 9211-8s with 10gbe.
Im not 100% sure on drives and freenas setup just yet but need something in the region of 50TB.

Im also still undecided to go with 64GB UDIMMs or allow for further extensibility and put in 64GB of RDIMMs which would permit expansion to 256GB theoretically.

thx in adv for any feedback

Make sure the x9 series supports that much ram..

Sent from my SGH-I257M using Tapatalk 2

 

[jgreco]

The X9S is an E5 board and E5 boards typically support 256GB-1.5TB depending on density and number of slots.

 

[Black Ninja]

Let bring this topic back to life and build the perfect socket 2011 NAS. I am buying X9SRL-F and planing to build another Freenas server with the intention to have bigger pool.Keeping the same specs that mchatterton put as follows:

MB:X9SRL-F
CPU: E5-1620 v2 or E5-1650 v2
HBA: 3 x LSI 9211-8i
HDD: 24x6TB HGSTor 32x4TB HGST
RAM: 128GB*

I need advice on the ram choise , not exactly which particular brand to buy , but what kind of ram is best for the setup.

Example: For socket 1155 with xeon E3 up to 32GB I whent to old "Kingston memory recommendation tool" link jgreco posted long time ago, put my MB and select a 32GB ECC kit on 1600Mhz.

But for 128 GB there is no kit to buy, kits go as high as 64GB kits unregistered, and then from what I gather you need registered memory if you want 128GB , so if I bought 64 unregistered kit , I am screwed when decide to upgrade. I am not very familiar with capacities above 32 GB , can someone help me to buy the right type of memory and not screw up.

128GB is over $1000 and I really can't affort to make a mistake with this much ram.

 

[jgreco]

You're better off going to the Supermicro site, pulling up the recommended memory for the X9SRL, and looking for those modules. With the E5-16xx, MAKE DAMN SURE you do NOT look at LRDIMM modules.

I just did it and that looks kind of like slim pickin's, but also maybe like they never bothered to update it later. So here's my suggestion, given that I don't have the time to do the work myself.

One, look at this guide:

http://www.supermicro.com/support/resources/memory/X9_DP_memory_config.pdf

This is actually for the Dual Processor boards, but quite a bit of the discussion is applicable to a single socket too. It will discuss things like ranks and how you need to arrange your memory.

Two, usually the cheapest memory is the Samsung. You can find the Supermicro part number. By the way you can even write to Supermicro support and ask them what part numbers to use for a valid 128GB configuration... not a bad way to go... Then take the Supermicro part number and see if the Samsung equivalent part number is any cheaper. Same DIMM, just no Supermicro warranty.

Then, three, remember that your best course of action is only to buy 32GB or 64GB until you've proven you need more, but to make sure that you're buying the parts you need for the 128GB build. You do not need to fill all four channels right away. You can buy two 16GB DIMM's, if that's the "way to go," and populate just two channels. Then in six months, when you feel more's justified, add two more.

 

[Black Ninja]

I did wrote to Supermicro from time to time in the past like everything these days is hit or miss experience, depends who will answer you. Supermicro web site did list Samsung, Hynix and Micro in their tested memory list, I just thought kingston will be better quality consistence.

I'll stay away from LRDIMM modules, I am not sure what are they but I trust your advice, so I'll eliminate that option.

It has to registered memory if it's more that 64GB in the MB , do I understand that right ?

With stability and reliability and number one priority what about 1.35V vs 1.5 V is it matter for anything ?

 

[jgreco]

The E5-16xx series explicitly does not work with LRDIMM. Intel designed the 16xx as a "workstation" CPU and they kinda knew that there would be some of us kooks out here who might actually want to do large RAM configurations on a single socket CPU. If you read that guide it'll explain why LRDIMM is attractive in a high end server (basically boils down to rank reduction) and Intel wanted larger RAM server builders to angle towards the E5-26xx series instead.

The 1.35v modules would tend to run a bit cooler and eat less power. As far as I know, the ceiling for UDIMM is still 64GB, beyond that you need RDIMM.

I think the big question is whether or not there are any Supermicro approved 32GB RDIMM modules, which would probably be "the way to go."

 

[Black Ninja]

I don't know how many times you save me from disaster, but is another case of it. I order 1650 v2 last night for my new super-freenas build which seem to be the ideal choice with high clock speed and 6 cores, but now I have to make decision very fast to cancel my order after reading what you said about the cpu ?!

 

[jgreco]

That's up to you. The 1650's a great CPU ... the two things I see as major strikes against it are the lack of LRDIMM and that it cannot be used on a dual socket board. We've been ordering E5-1650 v3's almost exclusively for maybe two years now, so I definitely believe in that little powerhouse. We used to buy a lot of dual CPU boards but often never populated the second socket. Back in the days of Opteron that made sense because there was less of a price premium.

With the current gen E5-26xx CPU's that qualify as "in that league", both the 2637v3 (quad) and 2643v3(hex) are around $240-per-core, whereas the E5-1650v3 comes in at $92-per-core.

So if you're building a single CPU system on a single socket board, the only time I'd think the E5-26xx makes sense is if you need more than 256GB RAM.

The question mark in your case is that the X9's an earlier generation, and Supermicro is only listing RDIMM parts up to 16GB. A 32GB part on there would be a better choice, but Supermicro doesn't list any as tested under the RDIMM category.

 

[Black Ninja]

I hope they accept my cancellation order. I can always reorder when I am sure what to get. Besides that can't be use in dual socket also I notice that 1600 v2 has much less cache. 1620 v2 is 10 MB for 4 cores , but then 1650 v2 instead of being 15MB cache it has only 12 MB for the addition 2 cores ?!?! It comes down to , what would be the best E5 for freenas with 128GB RAM for around $500. Something like 1620 v2 ( high clock ans 4 cores) or get 2600 with lower clock like 2.4 but with 6 cores in order to be in the same price range ?!

I don't understand is there trade of when go to register memory ? I know about 64GB it has to be registered to work , but I can build 32 GB system with registered memory if it's better and no trade offs going this way ?!?

I want X9 as proven and perhaps more reliable that newer ones. Newer is not always better anymore even the opposite. Now the technologies don't just go up , they kind of go side ways. Example: new IPMI chip is gigabit instead of 100Mb and when you mount ISO file is much faster , but remote screen is terrible ugly , and good functionality replaced with buggy challenge. I can't affort the build 2011 R3 system just to find out how it will behave, cause if it's worst than older 2011 I'll loose to much money sell on ebay and order new ones.

 

[jgreco]

I think the ins and outs of the X10 gear are well-known by this point. I've had E5-1650 v3's running for well over a year at this point. The differences and performance improvement from X9 to X10 are rather marginal.

The E5-16xx's have some limitations, but are usually the best bang-for-the-buck. If you look at the E5-1650 v2 (hexacore, 12MB, 3.5-3.9GHz, $583) and then at the dual socket E5-2637 v2 (QUAD core, 15MB, 3.5-3.8GHz, $996) ... two less cores for 70% more money. I'm happy to lose a little cache in order to save a lot of cash, usually.

As for the memory, look at where you might want to go and then make sure what you're buying helps you make progress on the road to that ultimate result. We used to have a lot of users here do what I call "slot stuffing", which is when you get an E3 based system with 4 memory slots (and 32GB total capacity) and then they proceed to fill those slots with 4 x 4GB DIMM's. That means to get past 16GB you gotta get rid of what you already bought. Much smarter to use 8GB DIMM's if there's any chance you'll ever want to go past 16GB.

For the E5 systems, this usually means that you want to be looking for 32GB modules if you can find them, or 16GB if you're *sure* you will never want more than 128GB.

 

[Black Ninja]

I really liked the part when you said "lose a little cache in order to save a lot of cash":D
But you are speaking for comparable CPU from 16xx v2 and 26xx v2 group. I am talking about keeping the same budget around $500-$600 , you can buy either get 1620V2 or E5-2630 v2, which one you would pick for the the freenas server , knowing the difference ?

For the ram you are absolutely correct , but it was not the point. I would make such a mistake , the one one you describe people are doing. For E3 it's really simple the way I see it, the rule "ram is cheap" applies in this case - you can easy max the MB Ram capacity (32GB) for little over $200. And this is the only reason I would put 4 sticks of ram on E3 which is dual channel and for less ram only put 2 sticks no mater what config. So I would never buy 4x4 on E3 to get 16 RAM.
BUT I am talking about the E5 and good 128GB ram configuration choice ? This is where I lack of understanding why the register memory is needed , and more important what are the trade offs with registered memory , if there is any ?!



P.S. My current freenas (the one in the signature) is based on the best recommended hardware, except the nic which is intel 10Gb not Chelsio , but other than that I consider to be one of the most proven and reliable freenas hardware for E3. I am trying to do the same thing with E5 so I can put more memory and drive 144TB raw pool data and I liked X9SRL-F for being the most clean and nice X9 2011 MB I know of.