3U Server Build Help

[-MG-]

I did not want to high-jack the other thread around a 4U server. I was looking for a fully assembled server on eBay, but haven't found exactly what I have heard others have mentioned as must haves or better yet, things to avoid in specs.

My uses are a mix of several:

redundant storage server, currently in raidz2 with 6x3TB drives
Plex Server (transcoding)
A few VMs, nothing too intense

I'm thinking I want to put together a new build that will last me for 5+ years and have a storage room with a half rack sitting in it mostly empty and wanted to make sure of the space .Here is what I have learned from other threads and found currently available on eBay: (prices include shipping which is basically just on the Chassis)

$288 Supermicro 3U 16 Bay SAS RAID Storage CSE-836 Barebone Chassis BPN-SAS2-836EL1
$235 Supermicro X9DRi-LN4F+ Dual LGA2011 1.20 Support E5-2600 v2 24x Slot DDR3 I/O
$129 Intel Xeon E5-2670V2 SR1A7 2.50GHz 10 Core CPU 115W 25MB Ivy Bridge LGA2011 (x2)
$99 LOT OF 10 Hynix 8GB DDR3 1333MHz ECC REG PC3-10600R HMT31GR7BFR4C-H9 (x2 not that I need that much but price seems good)
$34.95 SAS9201-8i: LSI 6Gbps SAS/SATA PCI-e RAID controller w/ML-ML SATA cables

Total: $1013.95

I have built many home PCs, but never using server hardware, so please let me know if I am missing something obvious here. This is the maximum budget I would have for this. Does this make sense? Is there a better option I should look at? A 2U setup likely could work, but from an airflow perspective and having the option to grow my array if I wanted to to 16 drives isn't a bad thing. I'll also be adding a few SSDs for the VMs to run and a few jails.

 

[Chris Moore]

I think that system board is a little bit more expensive than it should be. It looks like it should work though.
You will need to ensure that the SAS controller is flashed to the latest version of the IT mode firmware.

 

[-MG-]

You are right on the system board.

It looks like there are a lot more options when it comes to 2U Chassis that include that same motherboard for 250-400 depending on what is in it. Going to spend some time and look that up. 12 bays would still be good enough for me and that likely will save at least $200. I'll post back with what I find to make sure I am not missing something.

 

[-MG-]

Does something like this make more sense? Just need to add more RAM?

Supermicro 2U Server X9DRI-LN4F 2x E5-2650 2ghz 16-Cores / 64gb / SAS2 Raid

There would be 8 RAM slots left to add something to. Likely then total price under $800.

 

[Chris Moore]

Does something like this make more sense? Just need to add more RAM?

Supermicro 2U Server X9DRI-LN4F 2x E5-2650 2ghz 16-Cores / 64gb / SAS2 Raid

There would be 8 RAM slots left to add something to. Likely then total price under $800.

I would never suggest using a 1U or 2U server because of the airflow problems involved with those platforms. If there is no space constraint in your rack, 3U or 4U is preferable.

 

[rvassar]

Seems a little old for a planned 5 year service life. Are you willing to take the NUMA hit to run an older dual socket board? Have you looked at the X11 boards? You may be able to get the same performance out of a newer single socket board like the X11SPL-F.

 

[Chris Moore]

Does this make sense?

There might be ways to get the performance you want with less cost.
What was your reason for going with the dual socket systemboard?

 

[-MG-]

None other than one of my VMs The minimum specs they recommend for the program it will run Is a quad core processor and about 8gb ram. With setting up VMs I have to dedicate those resources right and dnt know what that leaves me for other stuff? I currently don’t have much for VMs besides that.

 

[Chris Moore]

None other than one of my VMs The minimum specs they recommend for the program it will run Is a quad core processor and about 8gb ram. With setting up VMs I have to dedicate those resources right and dnt know what that leaves me for other stuff? I currently don’t have much for VMs besides that.

Are you planning to run the VM on the virtualization capability of FreeNAS? Has that been tested to ensure it will work? The virtualization in FreeNAS is not very mature and some things may not work. What are you planning to run? Details can make or break the plan. Please let us know what the plan is and we will do our best to give advice.

 

[-MG-]

It will run my crypto bot - HaasOnline. Runs in a windows os.

Appreciate the help.

 

[rvassar]

It will run my crypto bot - HaasOnline. Runs in a windows os.

Appreciate the help.

Since that's a Bitcoin/Financial trading application... One of the weaknesses I've experienced in FreeNAS VM's is timekeeping. Bhyve doesn't appear to implement the ESXi hack of passing time calls down to the hypervisor. So each VM maintains its own clock, and it's subject to the hypervisor scheduling it on a CPU to perform ticks. Even with NTP configured, I can't get a Ubuntu Linux VM to stay much closer than a 1/2 second of true UTC. Without NTP, I got multiple seconds of clock drift per day.

@jgreco may have more to say here...

 

[-MG-]

Since that's a Bitcoin/Financial trading application... One of the weaknesses I've experienced in FreeNAS VM's is timekeeping. Bhyve doesn't appear to implement the ESXi hack of passing time calls down to the hypervisor. So each VM maintains its own clock, and it's subject to the hypervisor scheduling it on a CPU to perform ticks. Even with NTP configured, I can't get a Ubuntu Linux VM to stay much closer than a 1/2 second of true UTC. Without NTP, I got multiple seconds of clock drift per day.

@jgreco may have more to say here...

Valid point. I potentially then I consider going the other way around this and virtualize FreeNas instead? Either way, I would guess my hardware requirements wouldn't change tremendously?

 

[rvassar]

Valid point. I potentially then I consider going the other way around this and virtualize FreeNas instead? Either way, I would guess my hardware requirements wouldn't change tremendously?

Only you know what kind of arbitrage you're working with such an application. Is there a requirement that time be tracked that closely? Does Windows have the ability to correct at least as well as NTP on Ubuntu?

As for hardware requirements... There is a guide here for virtualizing FreeNAS, but I'm not very familiar with it, so I can't comment.

 

[-MG-]

TBH, after reading more about putting FreeNas in a VM has me a little more worried. I have no experience in the ESXi world and sounds like you can screw it up. My whole reason for upgrading the hardware revolved around a longer term hardware option that could do it all for me with running that VM.

There is a linux distribution of the software I use. Don't know if solves my NTP time issue.

 

[rvassar]

There is a linux distribution of the software I use. Don't know if solves my NTP time issue.

I can get NTP to keep time in a Linux VM bouncing around 500ms give or take, but due to the jitter, it never raises the poll time above 64 seconds, and that kind of abuses the upstream servers. It's impolite to do that to a public NTP pool. I'm not sure how much better it would be if you set up FreeNAS itself to be an sole source NTP peer. I didn't try that, but at least it would keep the NTP traffic in house. Other than the timekeeping issue, the Linux VM worked OK.

Any chance there's a FreeBSD port of it? Linux and BSD both present a similar POSIX programming API, so it's somewhat trivial to port software between them. FreeNAS supports creating FreeBSD "jails" that can host all kinds of applications. The Jail's do not have the timekeeping problem Bhyve VM's have. They're compartmentalized OS instances that share the FreeNAS kernel.

 

[-MG-]

I don't believe they have developed that.

Out of curiosity, why does it not work if I spin up a windows VM that I cant just install NetTime (a Simple Network Time Protocol (SNTP) )

 

[rvassar]

Out of curiosity, why does it not work if I spin up a windows VM that I can't just install NetTime (a Simple Network Time Protocol (SNTP) )

I can't answer that for you. Is there anything about your VM hosted application that requires it to track time within some margin of error? For example:

Microsoft requires computer clocks to agree within 5 minutes for domain membership.

https://docs.microsoft.com/en-us/wi...-tolerance-for-computer-clock-synchronization

Left on its own, my Ubuntu experiment would drift that far out of sync in less than a week. Unix applications like make and rsync want time stamps to be within seconds. Since you stated your application was a Bitcoin trading Bot, I just kind of assumed there some market timing function in play, or at the very least with money on the table, you'd like the time stamps to be sort of be correct.

SNTP is a simplified version of NTP that does not keep state information. I don't know that it will do any better, but it might. All you can do is try it. I could not get full NTP to maintain anything less than 500ms error bars, and there were step adjustments every few minutes.

 

[-MG-]

As you mentioned. For the software to be as accurate it needs to be pretty close to the exchanges time. Now I don’t execute any arbitration, so being off by like 5 seconds is not material. I should just test on my current build and see how it performs with time or if the software times out.

I suppose I could just leave my current setup and buy a cheap PC to run my software if I need be. My current setup is basic hardware:

Xeon(R) CPU E3-1220 v3 @ 3.10GHz
24GB DDR3 1600 RAM
(Can’t find the mobo specs anymore but likely bought new in 2013)

Knowing all that. If I want to move to something else. Are my previous picks in line or too much? How should I consider a single socket setup? Got an example?

 

[rvassar]

I should just test on my current build and see how it performs with time or if the software times out.

Are my previous picks in line or too much? How should I consider a single socket setup? Got an example?

That's the best thing to do... Test it. See if it hinders your operations.

As for a suggested setup... I'm going to defer to @Chris Moore here, as I haven't made this generational jump myself, and he is quite clearly better at this that I am. Understand the X9DRi is also circa 2013... The E5-2670v2 appeared on Intel's price list September 10, 2013. You're not jumping forward a generation at all. So at the end of your 5 year expected life, you'll be running a 10 year old rig. There's nothing particularly wrong about that, I have a circa 2008 Dell SC1430 system here as a scratch box, it runs just fine. But for the budget you have listed, a more modern X10 or newer X11 based system with equivalent performance, and fewer parts to fail, is possible. It will likely do a better job meeting your 5+ year life expectation.

 

[Chris Moore]

I'm going to defer to @Chris Moore here, as I haven't made this generational jump myself, and he is quite clearly better at this that I am.

Thanks for the vote of confidence. Hardware research and the selection of new hardware is one of the things I do for work. I spend a lot of time looking at specs and trying to figure out how they impact performance.

E5-2670v2

The big advantage of the "new" system the OP asked about is the Xeon E5 instead of the Xeon E3 that he currently has. Even a single socket Xeon E5 board like the one I use is able to easily accommodate 256GB of system memory and it isn't horribly expensive memory because it is Registered instead of Unbuffered.

Xeon(R) CPU E3-1220 v3 @ 3.10GHz

This is not a bad processor, except for the fact it is limited on CPU cores and the maximum RAM is only 32GB (if I recall correctly) and those limitations matter if you want to run some VMs in the same space that you are doing FreeNAS because FreeNAS likes some RAM.

The E5-2670v2 appeared on Intel's price list September 10, 2013. You're not jumping forward a generation at all.

I did the same thing though. I had a SuperMicro X9SCM-F, LGA 1155 system board, which uses a Xeon E3, and I switched to a X9SRL-F, LGA 2011, SuperMicro board with a Xeon E5. The X9SCM-F board with a Xeon E3 only had two of the x8 PCI-E 3.0 card slots, the other two were x4 electrically and PCI-E 2.0 in x8 physical slots. That limitation of expandability and the limitation of the memory and processor is what prompted me to upgrade to a more capable board.
There have not been huge increases in speed generationally with processors in the last few years. It is has been mostly efficiency in that they get a little more performance while using less power to get there.

Depending on the hardware you select, you can get a decent used system for between $400 and $600, not counting drives, but to buy that much performance in a new system would cost quite a bit more. The advantage of spending more is that the power consumption from the processor would be less, but in a NAS, the biggest power draw is from the drives. For example, my NAS (at home) has 32 drives in it at the moment. Each drive is taking somewhere around 8 watts of power. The CPU (if I recall correctly) is 90 watts. Some other components also take power, but I don’t want to over complicate the comparison. I could get a newer processor that might perform equally well and only take 32 watts which saves me 55 (ish) watts, but I still have 256 watts in drives. If you are building a NAS, you need to put a good bit of thought into the goal. Do you want fast access to the data? How much is that worth to you? If you plan to use it for a long time, it might even be worth considering SSDs because they don’t draw a lot of power and over the life of the system it might save you a lot on your electricity and be significantly faster than spinning rust.

There are a lot of variables in this equation and it makes most people have a pain somewhere behind their eyes and they say something like, "Synology doesn't need ECC".