BUILD 60TB Supermicro Xeon-D build - comments are welcome

[Steve Salier]

Howdy folks - so here is my latest NAS build - any and all comments are welcome!

1. Case: LIAN LI PC-Q26B
2. Fans: 5 x COUGAR CF-V12HB 120mm, 1 x Vantec Stealth SF8025L 80mm
3. Board: SUPERMICRO MBD-X10SDV-TLN4F (Xeon-D 1541 8-core)
4. RAM: 4 x SAMSUNG 16GB 288-Pin DDR4 ECC RDIMMS (64GB)
5. SAS HBA: LSI LSI00244 (9201-16i)
6. Breakout: 4 x Monoprice 8190 1m 28AWG Internal Mini SAS 36pin (SFF-8087)
7. DDMs: 10 x HGST Deskstar NAS 6TB 7200 RPM 128MB Cache SATA 6Gb/s
8. ZIL: 2 x Intel 730 Series 2.5" 240GB SATA 6Gb/s (mirrored)
9. ZFS OS: SUPERMICRO SSD-DM032-PHI SATA DOM
10. PSU: CORSAIR RMx RM1000X 1000W full modular

I will be connecting this via GigE until i can get a few 10GBASE-T switches ....

Thanks for looking!

 

[Ericloewe]

LIAN LI PC-Q26B

Careful: Those drive backplanes DO NOT support hot-plugging. DO NOT to insert drives with the server running.

• ZIL: Intel 730 Series 2.5" 240GB SATA 6Gb/s

• SLOG: Intel 730 Series 2.5" 240GB SATA 6Gb/s

Wut. An SLOG is a Separate Log device - the log in question is the ZIL - ZFS Intent Log.

PSU: CORSAIR RMx RM1000X 1000W full modular

Bit of an overkill. An 850W unit would probably be a better choice, for 10 drives. Consider a Seasonic Platinum 850 or X-850.

 

[anodos]

Howdy folks - so here is my latest NAS build - any and all comments are welcome!

1. Case: LIAN LI PC-Q26B
2. Fans: 5 x COUGAR CF-V12HB 120mm, 1 x Vantec Stealth SF8025L 80mm
3. Board: SUPERMICRO MBD-X10SDV-TLN4F (Xeon-D 1541 8-core)
4. RAM: 4 x SAMSUNG 16GB 288-Pin DDR4 ECC RDIMMS (64GB)
5. SAS HBA: LSI LSI00244 (9201-16i)
6. Breakout: 4 x Monoprice 8190 1m 28AWG Internal Mini SAS 36pin (SFF-8087)
7. DDMs: 10 x HGST Deskstar NAS 6TB 7200 RPM 128MB Cache SATA 6Gb/s
8. ZIL: Intel 730 Series 2.5" 240GB SATA 6Gb/s
9. SLOG: Intel 730 Series 2.5" 240GB SATA 6Gb/s
10. ZFS OS: SUPERMICRO SSD-DM032-PHI SATA DOM
11. PSU: CORSAIR RMx RM1000X 1000W full modular

I will be connecting this via GigE until i can get a few 10GBASE-T switches ....

Thanks for looking!

I'd go with an LSI 9211-8i for the HBA and use a couple of the motherboard's sata ports for the remaining drives. I'm not sure why you have two SSDs (ZIL and SLOG), but I'd probably not add those unless actually needed (note that this isn't a particularly good SSD for a slog device). I'll let someone else comment about other parts. :D

 

[Steve Salier]

Thanks for the comments all - so that was a typo - I have 2 Intel 730 series SSD laying around - which I believe use SuperCaps - so I was going to use these as a mirrored ZIL .....

 

[Ericloewe]

Question is, do you really need an SLOG?

 

[Steve Salier]

Careful: Those drive backplanes DO NOT support hot-plugging. DO NOT to insert drives with the server running.

Yeah looks like they are passive - since I cannot locate these anywhere to purchase ill remove the single 2 x backplane and just directly connect the drives to the breakout cables.

 

[Steve Salier]

Question is, do you really need an SLOG?

Yeah good point - not sure yet - I have the SSDs laying around so I will have to test with/without.

 

[Ericloewe]

There's no problem with using the backplanes, just don't try to mess with the drives while the server is running.

 

[wreedps]

What happens if you hot plug a drive into a non hot plug backplane?

 

[Ericloewe]

What happens if you hot plug a drive into a non hot plug backplane?

Best case? Your PSU is fantastic and keeps the voltage stable and you don't notice a thing.

General case: Your PSU can't handle the intense transient and several other drives spin down due to the low voltage, creating all sorts of fun things.

 

[jgreco]

Best case? Your PSU is fantastic and keeps the voltage stable and you don't notice a thing.

General case: Your PSU can't handle the intense transient and several other drives spin down due to the low voltage, creating all sorts of fun things.

Consider it another good reason to have a sufficiently large PSU ;-)

 

[Evan Richardson]

Howdy folks - so here is my latest NAS build - any and all comments are welcome!

1. Case: LIAN LI PC-Q26B
2. Fans: 5 x COUGAR CF-V12HB 120mm, 1 x Vantec Stealth SF8025L 80mm
3. Board: SUPERMICRO MBD-X10SDV-TLN4F (Xeon-D 1541 8-core)
4. RAM: 4 x SAMSUNG 16GB 288-Pin DDR4 ECC RDIMMS (64GB)
5. SAS HBA: LSI LSI00244 (9201-16i)
6. Breakout: 4 x Monoprice 8190 1m 28AWG Internal Mini SAS 36pin (SFF-8087)
7. DDMs: 10 x HGST Deskstar NAS 6TB 7200 RPM 128MB Cache SATA 6Gb/s
8. ZIL: 2 x Intel 730 Series 2.5" 240GB SATA 6Gb/s (mirrored)
9. ZFS OS: SUPERMICRO SSD-DM032-PHI SATA DOM
10. PSU: CORSAIR RMx RM1000X 1000W full modular

I will be connecting this via GigE until i can get a few 10GBASE-T switches ....

Thanks for looking!

Where did you find the D-1541? and what did you pay for it, if you don't mind me asking. all I can find is the 1540's, or 1520s. I was hoping to snag a 1531 6 core proc for my next file server,

 

[Steve Salier]

Haven't bought the board yet - I was waiting a bit longer in anticipation that the "storage optimized" (and supposedly with functioning SR-IOV and fewer errata) X10SDV-TLN4F 1541's have made their way into the channel. I may gamble and just get this http://www.amazon.com/dp/B010W32T62...e=df0&creative=395093&creativeASIN=B010W32T62 and see what happens. I can always return it.

SM never changed the SKU to reflect the new v2 CPU stepping - so its more or less a gamble as to whether you will get the 1540 or the 1541 (which is profoundly annoying) ....

 

[Evan Richardson]

....

Ahh ok, yeah it's frustrating, as the sku's have been out for some time now, yet SM and othe vendors aren't rapidly releasing the various parts to market yet. Everything so far seems to be 1520 or 1540 from SM, even though they quote 1540/41 , etc) I'm hoping in ~6 months or less that'll change, and we'll see a buncha boards from asrock and SM, if not others, and the price comes down, as you can still easily get a E3 capable board + e3 v4 for < 500$, vs these 800-900$ ones. Yeah it might draw more power, but it's cheaper in the long run (unless you want the XE ports)

nice looking build btw, don't suppose you'd be interested in a PC-A79 case, with 4 5in3 supermicro drive holders? Looking to part out my old file server, not sure what to do with the case :)

 

[Steve Salier]

As an alternative build, since my requirements are ITX form factor with 64GB ECC support - this board looks great: http://www.asrockrack.com/general/productdetail.asp?Model=EPC612D4I#Specifications

Paired with an E5 2620 V3 makes a great combo, however, 16GB ECC SoDIMMs are not available as yet. As per this:

http://www.intel.com/content/dam/ww...y/ddr4-2133-sodimm-ecc-validation-results.pdf

They are apparently in production, but not yet available anywhere.

So the waiting continues .....