New build - controller card recomendations

[Mugiwara]

Hey everyone,

I've been running an ESXI virtualized FreeNas isntance for the past 3 years or so, and its worked great for me. However, I am starting to encroach on the high water mark, and my E3 v3's 32 GB of RAM limitation is not really allowing me to expand further, so decided to build a new box. My real question is about the different storage controllers. I am looking at going with 12 drives in a ZFS2 configuration, but most of the baords in my price range have an embedded LSI controller with 8 ports, and then the standard embedded motherboard option. Will I be able to simply plug in 8 drives to the LSI controller and 4 to the motherboard controller and call it a day? Or is there a better way to do this. Thanks!

Drives: 12 x 8 gb WD reds (purchased, $2300)
CPU: Intel Xeon Processor E3-1240 v6 (~$275)
Mobo: X11SSH-CTF (~$400)
Ram: Crucial CT16G4WFD824A 16GB x4 (~$150 * 4 = $600)
Chasis: ??? Some 3u or 4U quiet chassis? (~$250)
Boot drive: Samsung 960 EVO 128 GB ($90)
Power Supply: Seasonic Flagship PRIME TITANIUM 650 SSR-650TD ($130)
Total: ~$4,000. Hoping to make this last for at least 5-6 years,goal is to be at around $800/year

Thoughts? Recommendations on any parts? Im thinking I'll run this bare metal and create a couple jails for applications that require frequent NAS pool access. I will miss my virtual 10GBe esci switch, but I think I'll probably get a physical 10gb when I get the money.

This is a home build, so hoping to keep as quiet and cool as I can as well. Thanks, and please let me know your thoughts.

 

[Ericloewe]

Will I be able to simply plug in 8 drives to the LSI controller and 4 to the motherboard controller and call it a day?

Yes, of course.

Chasis: ??? Some 3u or 4U quiet chassis? (~$250)

Bit of wishful thinking there. Your best bet is to get a used Supermicro chassis off ebay. Be prepared to swap out the PSUs for quieter ones.

Boot drive: Samsung 960 EVO 128 GB ($90)

Seems like overkill, any half-decent SSD will do fine.

Power Supply: Seasonic Flagship PRIME TITANIUM 650 SSR-650TD ($130)

I don't think that there's a good rackmount chassis that takes ATX PSUs. There are somewhat dubious ones, with mixed experiences. In any case, if you do end up needing an ATX PSU, I'd suggest the Seasonic Platinum 860 instead, if the pricing is favorable. The fan is much better.

 

[Ericloewe]

but I think I'll probably get a physical 10gb when I get the money.

You mean a switch? The X11SSH-CTF already has 2x 10GBaseT on board.

 

[Dice]

Regarding chassis, recycled SuperMicro boxes of e-bay are usually great value at your drive count. If so, pay attention to what backplane is fitted, reference SM to make sure it supports drives larger than 2TB.
As pointed out, PSU's play a huge factor in the amount of noise they produce. Easily replaced, but ...another dollar....

Other options include Norco RPS-4224 https://www.newegg.com/Product/Product.aspx?Item=N82E16811219038 which I guess @Ericloewe would include in the "somewhat dubious".
Yet, there are users who are Very happy with their boxes after modifications. (IIRC I'm looking at @Stux)

Another case I've found is the X-Case EXtra Value RM 424.
https://www.xcase.co.uk/collections...cts/x-case-extra-value-rm-424-24-hotswap-bays

I'd expect it to be rather close or to the Norco in terms of quality. However, the RM424 features excellent fans.
I managed to pick them up and used hooked up to the PWM controls on a X11SSL (I jerryrigged power directly from PSU).
The fans ARE a lot quieter than the hardcore fans used in SuperMicro, - particularly at low RPM. They do not have the whine/grind/sort of tick noise... Still they push boatloads more AIR than the typical Noctua or Fractal Design case fans. To date they are my favorite fans.

 

[Mugiwara]

Thanks for the feedback, I appreciate it. Yes, I am talking about a 10GBe switch. Here is an image and a bit more background of where I am coming from:
Current build: ESXI server with hardware passthrough of LSI controller card:
Mobo: X10SL7-F
CPU: E3-1275 v3
RAM: Samsung ECC 4x8GB (10 GB to Freenas)
Drives: 5x 4TB WD reds
Network: E1G44HT I340-T4 with ports passed through to individual VMs

As you can see, I currently have a mid tower case in a small cupboard. Unfortunately,with 12 drives, I think I have to go with a rack mount to fit them all, and hotswap is definitely a nice feature to have. However, I really dont have a basement to put this in, and it gets pretty hot out here in AZ, so hoping to find something that isn't terribly loud. I like the SM cases, but really dont need redundant power supplies, and those enterprise based power supplies with the tiny super high RPM fans.

Thanks again.

*Edit* Just saw your recommendations of the cases and fans Dice, thanks. I'll take a look. I really wish someone made a 12-16 bay chasis with good, quiet airflow,but I guess the market for that would be pretty small. I jsut want a compact and quiet file server,is that so much to ask for? :(

 

[Stux]

Regarding chassis, recycled SuperMicro boxes of e-bay are usually great value at your drive count. If so, pay attention to what backplane is fitted, reference SM to make sure it supports drives larger than 2TB.
As pointed out, PSU's play a huge factor in the amount of noise they produce. Easily replaced, but ...another dollar....

Other options include Norco RPS-4224 https://www.newegg.com/Product/Product.aspx?Item=N82E16811219038 which I guess @Ericloewe would include in the "somewhat dubious".
Yet, there are users who are Very happy with their boxes after modifications. (IIRC I'm looking at @Stux)

Another case I've found is the X-Case EXtra Value RM 424.
https://www.xcase.co.uk/collections...cts/x-case-extra-value-rm-424-24-hotswap-bays

I'd expect it to be rather close or to the Norco in terms of quality. However, the RM424 features excellent fans.
I managed to pick them up and used hooked up to the PWM controls on a X11SSL (I jerryrigged power directly from PSU).
The fans ARE a lot quieter than the hardcore fans used in SuperMicro, - particularly at low RPM. They do not have the whine/grind/sort of tick noise... Still they push boatloads more AIR than the typical Noctua or Fractal Design case fans. To date they are my favorite fans.

Yes. After my fan mods the Norco case is super quiet and serves my needs perfectly.

https://forums.freenas.org/index.ph...24-supermicro-x10-sri-f-xeon-e5-1650v4.46262/

 

[Mugiwara]

Awesome, thanks. For my non-enterprise use, while a SM chassis would be nice, I think that those suggestions are more suited for my use case. Stux, after going through your thread, it does seem like you did have to go through quite a few modifications. If you could go back, any changes in your components? Do you think that something like this would be a better idea for a rack? I dont think that actually supports rails, but honestly, I think a shelf would be fine. Not sure I would ever have to pull it out unless something failed, and that should (hopefully) be once or twice in a lifetime. Might have to increase budget by $500 to get the chassis, shelves, and that mini rack, but seems reasonable.

I actually hadn't realized the Coral was being taken out of the primary release path until I was going over the forums yesterday, I will need to also see how this impacts my current build path with direct disk jail access and plan accordingly for some of my VMs.

 

[Dice]

Rack mounted chassis are just as good stacked (well unless you have multiple).

 

[Stux]

I already had a heavily populated 42U rack that I wanted to add the Norco to. That's where my issues came from, since its an older design.

The flimsier racks won't have a problem with the Norco rails... and as you mentioned, a shelf is a practical solution too.

The actual fan swaps that I made were always planned. The various plugging and airflow optimizations I made were to improve airflow management, and were relatively simple, especially now that I've documented them ;)

So, forgetting about the rackmounting issues, everything else was fine about the case... budget for swapping out the fans.

 

[Mugiwara]

Good to know. I'll keep an eye out for a sale on the case and then I may take the plunge. Thanks for the advice everyone.

 

[Alan W. Smtih]

@Mugiwara - Question for you on your drives. Assuming it's 12 of the 8TB (not gb) WD Reds, where did you get them for $2300?

I see them on Amazon for $274.99 which would be $3,299.88.

Do you have a connection you can share, or did I just catch them after a price hike?

Thanks!

 

[Mugiwara]

@Mugiwara - Question for you on your drives. Assuming it's 12 of the 8TB (not gb) WD Reds, where did you get them for $2300?

I see them on Amazon for $274.99 which would be $3,299.88.

Do you have a connection you can share, or did I just catch them after a price hike?

Thanks!

I was actually able to pick them up as part of this deal at best buy for 8TB externals, but seems to be over now I will be shucking and installing bare.

 

[Alan W. Smtih]

Nice!

I missed it, but loving knowing that folks made that work.

I'll keep an eye out for that next time I'm in drive buying mode.

 

[Mugiwara]

Nice!

I missed it, but loving knowing that folks made that work.

I'll keep an eye out for that next time I'm in drive buying mode.

Heads up, they are on sale again for $180 if you dont mind shucking the the 2 year warranty. http://www.bestbuy.com/site/wd-easy...-3-0-hard-drive-black/5792401.p?skuId=5792401

 

[Mugiwara]

So, sorry to bump my own thread, but after waiting 2 weeks, Sueprbiiz comes back and tells me that the Norco is out of stock and they don't have a restock date. So I am back to chassis hunting. I made a couple changes (I picked up an additional 2i/8 port HBA) and 4 more 8TB drives, along with 2 500GB samsung EVOs for things like booting and jails. So, my thoughts are to connect the 16 drives to both the onboard and PCIe HBA. I went ahead and bought a full set of fans and CPU coolers as Alan recommended, but can return if the new chasis cant be modified to accept them.

So, at this point, I figured why not bite the bullet and try to get a supermicro for a couple hundred more. Unfortunately, after looking for several days, I cant seem to find any 4u servers with SAS2 or SAS3 backplanes for under $1500, ebay or otherwise. I am under the impression that I need to be looking for backplanes like BPN-SAS2-846EL[1,2] . Is this correct? Anyone have any link to any chasis that they can recommend that would work for my build now? Thanks.

 

[danb35]

Don't remember where you're located, but if you're in the U.S., this looks like a fine deal. Supermicro 4U, 36-bay chassis, motherboard, 2 x E5-2660 CPUs, 128 GB RAM, 9211-8i, 64 GB SATA DOM for a boot device, SAS2 backplanes. Pretty much my system except that it has 2660s instead of 2670s, and the HBA is an add-on card rather than integrated on the motherboard.

Edit: Update the firmware on the HBA (maybe), add drives, and go--it's pretty much turnkey, from the listing.

 

[Mugiwara]

Thanks I had taken a look at that one. I wish he was offering the chassis barebones without all the extra hardware. Im out in Arizona, and its hot in the summer. so was looking for something more efficient / cooler running with less power, and some more optimized CPU instruction sets. Maybe I can sell off the RAM and mobo after, but may keep my eyes out a bit longer to see if he has something more barebones I can drop my own mobo into.

 

[Mugiwara]

After doing some research, I think I understand a bit more about the back planes, but would appreciate someone double checking my work before I byte the bullet and place an order. I stumbled across this configuration when searching, and I think that it should work for me?

Essentially, it looks like its a Supermicro 846 server chassis, with an SAS846TQ backplane. From reading, it appears that this is a simple passthrough, so it shouldn't matter what speed its rated for, right? I can simply pick up 4 of these cables from SM, plug the minisas-hd connector into each of the 4 ports on the HBA, and connect the SATA connectors to the back of the TQ backplane and be good do go? I will probably have to do some minor modifications to replace power supplies and fans to be a bit more quiet, but as far as I can tell, I think this would be a workable solution? Once again, I really appreciate all the help. Thanks.!

 

[Ericloewe]

You really want to avoid TQ backplanes, as the mess of cables quickly becomes unmanageable. The ones that take SFF-8087 are much better.

 

[Mugiwara]

Even with the breakout cables? Since its only one connection to the HBA itself, is it really that ugly on the backplane side? I guess after spending $4500 on hardware stuff, I am hesitant to want to drop about 900+ on any of the sueprmicro 4u chasis that have a sas2 or sas3 expander. If the general consensus is that its worth the extra $700, I guess I'll consider it, I was just hoping a few extra cables might save me some extra money :(