First Rackmount build

[ross0201]

I have an existing FreeNAS system that I'm happy with, but I'll want to expand it and I can't hold more memory in my current motherboard. I found Chris Moore's thread (https://forums.freenas.org/index.php?threads/my-new-48-bay-build.61223/) on his 48 bay build and would be interested in doing something similar. This has a SAS-2 backplane and holds an Extended ATX motherboard. It would be my first attempt at using a backplane and I'd like a confirmation that I'm on the right track. My current system is:

• Supermicro X10SLL-F (https://www.supermicro.com/products/motherboard/xeon/c220/x10sll-f.cfm)
• Intel Xeon E3-1231 v3 (https://www.amazon.com/Intel-Xeon-Processor-E3-1231V3B-BX80646E31231V3/dp/B00KB4A2XA)
  
• 8 x 4 TB WD Reds
• 9240-8i LSI RAID card, flashed to IT
• 32 GB EEC DDR3 memory, can't remember what type.
• 32GB SSD for OS
• Fractal Designs Node 804 (https://www.amazon.com/Fractal-Design-Supply-MicroATX-FD-CA-NODE-804-BL/dp/B00K6OVG0I)

I'd like to move the hard drives to a new system and up the processor a bit for 4K transcoding in Plex. My understanding is my current RAM would go into a X9 dual socket motherboard. Right now I'd just move the current HDD pool over to the new system, with a new motherboard and CPU. HBA would also be free to move. My questions are:

• If I get a motherboard with SAS2 ports, like X9DRD-7LN4F, can I skip the HBA? I think I'd just need a few SFF-8087 cables?
• I've read that moving the OS drive and the hard drives to a new system is simple and will just work, but I'd like someone to hold my hand and say, "yes it is that magical."
• Am I correct that the memory should work in X9 systems?
• A lot of these motherboards need to be flashed to bios 3.0 to use E5-2600 v2 CPUs, not all ebay sellers specify. Is there any sneaky way to update the bios if I only have v2 CPUs?
• Am I doing anything dumb?

Thanks!

 

[kdragon75]

If you can get into the IPMI, you may be able to flash that way. It can't be done in the webUI but perhaps via telnet/ssh. Don't quote me on that one.

Also, yes the IPMI interface/BMC will boot and accept connections without the CPU installed.

 

[Chris Moore]

If you wanted, you could begin with just moving the existing system to the new chassis.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[ross0201]

If you can get into the IPMI, you may be able to flash that way. It can't be done in the webUI but perhaps via telnet/ssh. Don't quote me on that one.

Also, yes the IPMI interface/BMC will boot and accept connections without the CPU installed.

Interesting. Looks like I need a key for some software, but other than that it looks doable.

 

[Chris Moore]

If I get a motherboard with SAS2 ports, like X9DRD-7LN4F, can I skip the HBA? I think I'd just need a few SFF-8087 cables?

I believe that the onboard SAS controller still would need to be flashed to IT mode an it would simply mean that your HBA is integrated on the system board instead of being a card. You would need two of the SFF-8087 cables, one to go to each of the two SAS expander backplanes, if you are looking at using the same chassis that I purchased. He relisted them again and still has 7 available:
https://www.ebay.com/itm/Server-Che...NEW-in-Box-New-Rail-Kit-Included/263519436540
When I bought mine, he was accepting offers of $350. Your mileage may vary.
Just to give you an idea of the performance, my main storage pool has about 8TB of data in it and completes the scrub in 2 hrs and 21 min but the data is spread across 12 drives, so there is less data per drive. The backup pool, containing the same data is on only 4 drives and it takes about 6 hours to finish the scrub. All of that is running on the same model SAS HBA that I had before I moved to the new chassis. The time to complete a scrub actually dropped by almost an hour and I can only guess that is because all the drives are on the same HBA instead of split between two.
This is the SAS cables I ordered:
https://www.ebay.com/itm/Mini-SAS-3...AS-36-Pin-SFF-8087-Data-Cable-1M/201484956159

 

[Chris Moore]

I've read that moving the OS drive and the hard drives to a new system is simple and will just work, but I'd like someone to hold my hand and say, "yes it is that magical."

I have migrated my storage to new hardware several times. It is pretty magical, but you might run into small bumps. Be prepared to manually set the IP configuration from the local console because it might not automatically configure the 'new' network card. This could cause some problems with other things that are depending on the network configuration. You data is not at risk, just some configuration hurdles.

Am I correct that the memory should work in X9 systems?

The dual socket system board that you mentioned X9DRD-7LN4F should accept the DDR3 unbuffered ECC memory you have, but you will want to have all matching memory in that system when you go to upgrade the quantity of RAM to higher levels and you must have some memory in the sockets associated to each CPU socket that is populated. Ultimately, you will want to replace your existing memory and go to Registered ECC memory as it is less expensive. I just picked the first result in the search and found this 64GB kit for only $329:
https://www.ebay.com/itm/64GB-Kit-4...AM-PC3-14900R-ECC-REG-Registered/222727470309
Where ECC unbuffered memory is $245 for only 32GB:
https://www.ebay.com/itm/32GB-4-X-8...OP-MEMORY-FOR-DELL-OPTIPLEX-9020/142320530635
Price wise, you just can't beat the Registered DDR3 memory, as far as I can tell, but I didn't look real hard.

• Am I doing anything dumb?

That depends on exactly what it is you are doing, why you are doing it, and what the budget looks like. There may be a way that you could get acceptable results with less money spent, but I am not sure exactly what your goal is. Did you plan to buy a chassis like I bought?
What CPU were you thinking of using in that new system board? Other than just being able to transcode a 4k stream, what are you trying to get out of this? I don't think you need this much hardware just to transcode a 4k video.

 

[ross0201]

If you wanted, you could begin with just moving the existing system to the new chassis.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

Sure. My thinking was upgrade the MB and CPU while I'm in there, add hard drives later, but I could put those things off.

I have migrated my storage to new hardware several times. It is pretty magical, but you might run into small bumps. Be prepared to manually set the IP configuration from the local console because it might not automatically configure the 'new' network card. This could cause some problems with other things that are depending on the network configuration. You data is not at risk, just some configuration hurdles.

The dual socket system board that you mentioned X9DRD-7LN4F should accept the DDR3 unbuffered ECC memory you have, but you will want to have all matching memory in that system when you go to upgrade the quantity of RAM to higher levels and you must have some memory in the sockets associated to each CPU socket that is populated. Ultimately, you will want to replace your existing memory and go to Registered ECC memory as it is less expensive. I just picked the first result in the search and found this 64GB kit for only $329:
https://www.ebay.com/itm/64GB-Kit-4...AM-PC3-14900R-ECC-REG-Registered/222727470309
Where ECC unbuffered memory is $245 for only 32GB:
https://www.ebay.com/itm/32GB-4-X-8...OP-MEMORY-FOR-DELL-OPTIPLEX-9020/142320530635
Price wise, you just can't beat the Registered DDR3 memory, as far as I can tell, but I didn't look real hard.

That depends on exactly what it is you are doing, why you are doing it, and what the budget looks like. There may be a way that you could get acceptable results with less money spent, but I am not sure exactly what your goal is. Did you plan to buy a chassis like I bought?
What CPU were you thinking of using in that new system board? Other than just being able to transcode a 4k stream, what are you trying to get out of this? I don't think you need this much hardware just to transcode a 4k video.

I noticed you had dropped some hints that you thought he would accept $350. It seems like a really good deal for something I don't think I'll ever outgrow.

That's interesting with the scrub times. I just had a big data dump, maybe 3TB, and the next scrub was 9ish hours. I haven't had another so I don't know if it was just because so much got written or if ongoing I'm just going to have long scrubs with this hardware.

IP address changes are just fine. My gut tells me that I should have to record which hard drive corresponds to each SATA port and somehow correlate it with the new motherboard. My gut is ignorant; I'm happy to be told I'm wrong, that just seems like how it should work.

I knew memory had to be super complicated! That is helpful though, thanks!

For performance my current CPU has a Passmark score of 9640. It can transcode 4K to a very high 1080p without stuttering but it shows 100% CPU usage and makes me nervous. It looked like to 2 socket boards were the way to get lots of memory and I just figured I'd find 2 ebay CPUs with a high dual Passmark score. I'd be very open to suggestions here though. I might also be interested in playing around with VMs in the future, but nothing in particular.

Thanks for all your help!

 

[ross0201]

I do plan to get the chassis, missed that question.

 

[ross0201]

That depends on exactly what it is you are doing, why you are doing it, and what the budget looks like. There may be a way that you could get acceptable results with less money spent, but I am not sure exactly what your goal is. Did you plan to buy a chassis like I bought?
What CPU were you thinking of using in that new system board? Other than just being able to transcode a 4k stream, what are you trying to get out of this? I don't think you need this much hardware just to transcode a 4k video.

I did find this motherboard used locally for $235 which makes it more palatable, otherwise it is definitely more motherboard than I need. The only reason to get rid of my HBA is because I can, maybe use less power/heat.

 

[Chris Moore]

It seems like a really good deal for something I don't think I'll ever outgrow.

That is a big part of the reason I bought it. I figure I will still be using this chassis 6 or more years from now, unless there is some big technological change that makes it totally obsolete. For the enterprise datacenter, it basically is obsolete now because the SAS expander backplanes are only 6GB/s where all the enterprise gear is doing 12GB/s now, but that doesn't matter to me because I am using SATA drives and they only do 6GB/s anyhow.

The only reason to get rid of my HBA is because I can, maybe use less power/heat.

Typical HBA uses about 10 watts and doesn't put off any more heat than a hard drive. The thing that draws power and generates heat is the drives. I was doing a power study on a system I am putting together for work and the drives (just 16 of them) will draw about 120 watts during seak, which is when they draw the most power.

IP address changes are just fine. My gut tells me that I should have to record which hard drive corresponds to each SATA port and somehow correlate it with the new motherboard. My gut is ignorant; I'm happy to be told I'm wrong, that just seems like how it should work.

One of the interesting things about the disks in a FreeNAS storage pool is that because of the way they are indexed by the gptid, you could take them out, shuffle them like a deck of cards and stick them back in and FreeNAS would not care even for a little. You can connect them by SAS or SATA, to any available port, as long as there is a supported driver to make that communications work, FreeNAS will pick the drive up and the pool will just work. The only times I ever had a problem with it was using a SATA port multiplier or USB. I don't suggest using them, they are not reliable in my experience. SAS and SAS expanders, on the other hand, are proven technology and I have had no trouble with the ones I have used.

It looked like to 2 socket boards were the way to get lots of memory and I just figured I'd find 2 ebay CPUs with a high dual Passmark score.

If you can find a good price on them, a pair of Xeon E5-2650 v2 processors should work very well for what you want to do. Should be about double the passmark score that you have now and has plenty of power to run some VMs. If I recall, if you use the Load Reduced (LR) registered ECC memory, you should be able to go up to between 1 and 1.5 TB of RAM on that board. Probably more than you need. It should do all you want and more.

That's interesting with the scrub times. I just had a big data dump, maybe 3TB, and the next scrub was 9ish hours.

There are some factors that matter there, that is the number of disks involved and how many vdevs they are in because that limits the amount of IO that is possible; and the total amount of data that is in the pool. How much total data are you storing? When you add more disks to the pool, the data does not get rebalanced across the new disks unless you write the data to a new location in the pool. More vdevs gets you more IOPS (Input Output operations Per Second) so it can go faster.

 

[Maelos]

Dangit, Chris beat me to the punch once again. Ross, it looks like we are taking similar journeys. This feels awefully self promoting, but my thread, the links and discussion within, may be useful. I believe there are still some deals with X9 boards, CPUs, and RAM out there.

Also, Chris, please don't let me see any more of your deal links. My wallet is already burnt enough...

 

[ross0201]

Price wise, you just can't beat the Registered DDR3 memory, as far as I can tell, but I didn't look real hard.

Thanks. Lots of this on ebay for good prices. As I increase storage how religiously do folks stick to the 1 gb ram per tb storage? I understand everyone has different needs, but does a 100 tb server need 100 gb of ram?

Ever heard of Actica brand memory? There's a decent amount fairly cheap on ebay.

One of the interesting things about the disks in a FreeNAS storage pool is that because of the way they are indexed by the gptid, you could take them out, shuffle them like a deck of cards and stick them back in and FreeNAS would not care even for a little. You can connect them by SAS or SATA, to any available port, as long as there is a supported driver to make that communications work, FreeNAS will pick the drive up and the pool will just work. The only times I ever had a problem with it was using a SATA port multiplier or USB. I don't suggest using them, they are not reliable in my experience. SAS and SAS expanders, on the other hand, are proven technology and I have had no trouble with the ones I have used.

This makes sense but most of my experience with computers is on less thought-out operating systems. I couldn't save Powerpoint presentations for about a month, only save-as. Turned out Windows Explorer's preview window was using the .ppt file and wouldn't let it be modified. How can you release that as a final product!?

If you can find a good price on them, a pair of Xeon E5-2650 v2 processors should work very well for what you want to do. Should be about double the passmark score that you have now and has plenty of power to run some VMs. If I recall, if you use the Load Reduced (LR) registered ECC memory, you should be able to go up to between 1 and 1.5 TB of RAM on that board. Probably more than you need. It should do all you want and more.

That does look like the best bang for the buck, thanks!

 

[ross0201]

Dangit, Chris beat me to the punch once again. Ross, it looks like we are taking similar journeys. This feels awefully self promoting, but my thread, the links and discussion within, may be useful. I believe there are still some deals with X9 boards, CPUs, and RAM out there.

Also, Chris, please don't let me see any more of your deal links. My wallet is already burnt enough...

That is quite a machine, how's it working out?

 

[Chris Moore]

As I increase storage how religiously do folks stick to the 1 gb ram per tb storage? I understand everyone has different needs, but does a 100 tb server need 100 gb of ram?

No, 1GB per 1TB is a vague guideline that is meant to keep people from trying to run a system with 2GB of memory and complaining about it not working well. The system memory is used for ARC (Adaptive Replacement Cache) by ZFS and it will consume as much of your memory as it wants, right up to all of it if you don't have enough. One of the servers I use at work has 128GB of memory and it has stabilized at about 75% usage, but my server at home only has 32GB and ARC will use it all. You just need enough to ensure the system runs smoothly and that depends on what you use it for.

Edit: ARC doesn't actually use "all" of RAM, it saves like 2GB for the operating system to run, but it doesn't set memory aside for your VMs unless you make some manual changes to tell it to do that.

Ever heard of Actica brand memory? There's a decent amount fairly cheap on ebay.

For reliability, I like to go with one of the bigger names but I have had perfectly good memory from companies I never heard of before.
The brands I trust are Samsung, hynix, Crucial, Micron, and I have even used Kingston successfully. You could be alright with the Actica, but it is a concern. Supermicro has a list of memory they have tested and as long as the memory you buy is the same type (not brand) you are usually good.

 

[ross0201]

Thanks for all the help! Put orders in tonight. Now I just need to find a use for the old x10 board. I've been playing with the idea of a pf sense router. This is definitely overkill for that though.