BUILD [UPDATE] X10SLR-F with 8/16 HDD -> Please review the result

[Hobbel]

Hey guys and girls,

a customer of mine wants to increase his storage by replacing the current multiple solutions.
I need to build a system with ~8-10 TB for now, with the possibility of adding more easily.

This is what I thought:

Chassis
16x 3.5" bays front + 2x 2.5" bays rear, 3U
Supermicro SC836A-R1200B replaced by
SC836BA-R920B

Motherboard
Supermicro X10SRH-CLN4F replaced by
X10SRL-F

CPU
E5-1620 v3

CPU heatsink
2U, passive
SNK-P0048PS
(a little bigger than the SNK-P0048PSC, if CPU upgrade is needed.)

SAS-Controller
1x Supermicro AOC-2308L-L8E

RAM
2x 16GB, ECC, DDR4
M393A2G40DB0-CPB

HDD
8 (for now), SATA3, WD Re, 2TB
WD2000FYYZ

Boot-Device
mirrored
2x SSD-DM016-PHI


I would use the chassis with 8 HDDs and expand them later up to 16 if needed. This setup costs ~3500 €. It shouldn't exceed 4000, if you have some improvements.

Any additional SSDs for caching?

The drives should get connected to the internal SATA-Ports with breakout cables (perhaps the upcoming 8 drives will be SAS). The main use (90%) is iSCSI-storage for ESXi.
Replaced by the SAS-Controller.

Assuming, that the hardware specs are good:
If I understand most of the threads in this forum right, a pool of mirrored vdevs would be the best?
What about backup? Go with software solutions inside the virtaul machines or some kind of replication? (The old storage systems could be used for backup)

EDIT #1:
Forgot to ask, which boot devices you would use? USB sticks mirrored?
Replaced Kingston memory with Samsung, as recommended by Supermicro.

Thanks in advance.

EDIT #2:
added CPU heatsink

EDIT #3:
changeb chassis, Motherboard, SAS-Controller

EDIT #4:
no SAS onboard, dual GbE instead of quad, chassis replaced

EDIT #5:
added Boot-Device

 

[Ericloewe]

Hey guys and girls,

a customer of mine wants to increase his storage by replacing the current multiple solutions.
I need to build a system with ~8-10 TB for now, with the possibility of adding more easily.

This is what I thought:

Chassis
16 bays, 4U
Supermicro SC836A-R1200B

Motherboard
Supermicro X10SRH-CLN4F

CPU
E5-1620 v3

RAM
32GB, ECC, DDR4, Kit of 4
M386A4G40DM0-CPB

HDD
8 (for now), SATA3, WD Re, 2TB
WD2000FYYZ


I would use the chassis with 8 HDDs and expand them later up to 16 if needed. This setup costs ~3500 €. It shouldn't exceed 4000, if you have some improvements.

Any additional SSDs for caching?

The drives should get connected to the internal SATA-Ports with breakout cables (perhaps the upcoming 8 drives will be SAS). The main use (90%) is iSCSI-storage for ESXi.

Assuming, that the hardware specs are good:
If I understand most of the threads in this forum right, a pool of mirrored vdevs would be the best?
What about backup? Go with software solutions inside the virtaul machines or some kind of replication? (The old storage systems could be used for backup)

EDIT #1:
Forgot to ask, which boot devices you would use? USB sticks mirrored?
Replaced Kingston memory with Samsung, as recommended by Supermicro.

Thanks in advance.

Looks good.

As for backups, there are several strategies. I believe ESXi can synchronize its snapshots with ZFS snapshots, which goes a long way towards getting good snapshots. Still, it's rather likely that the underlying filesystem will not be clean. A typical approach is to brute-force it and do more snapshots. Snapshots can then be replicated to a second ZFS pool, on a remote server.

 

[marbus90]

You may want to consider a 24bay 4U chassis with an SAS 12Gbps expander (just 4U instead of 3U but 50% more bays -> more IOPS) and Intel S3710 200GB for SLOG in the rear. Cabling will be more straightforward as well compared to 2x 4x SATA -> SFF-8087 reverse breakout and 2x SFF-8644-8087.
For VMs striped mirrors is the way to go. Note the 50% space reservation due to fragmentation.

846BE1C-R1K28B -> 24bay 4U 12Gbps Expander
MCP-220-84606-0N -> rear 2x2.5" SATA hotswap bays for SLOG/L2ARC/whatever
2x SSD-DM064-PHI -> 64GB SATADOMs for FreeNAS boot (USB sticks may be fine, but SSDs/SATADOMs are considerably faster at rebooting. since the FreeNAS is the SPOF for the VM landscape..)

Which coolers do you have planned for the system? 4U compatible ones are easier to come by.

 

[Hobbel]

@Ericloewe
thx for your comment. Will read further on this topic...

@marbus90
the 4U is nice, but also more expensive. Including the SATADOMS, it would exceed the 4000 €. Some of the parts are hardly available in Germany so they can cost more than customers in other countries are used too. I need to use the 3U chassis and if someone wants to expand the storage in future, 4 GB drives are an option.

What do you mean with the coolers?! I thought the Supermicro chassis have them pre-assembled?

 

[Ericloewe]

@Ericloewe
thx for your comment. Will read further on this topic...

@marbus90
the 4U is nice, but also more expensive. Including the SATADOMS, it would exceed the 4000 €. Some of the parts are hardly available in Germany so they can cost more than customers in other countries are used too. I need to use the 3U chassis and if someone wants to expand the storage in future, 4 GB drives are an option.

What do you mean with the coolers?! I thought the Supermicro chassis have them pre-assembled?

They don't seem to be included. Their stuff is listed here:
http://www.supermicro.com/ResourceApps/Heatsink_Matrix.aspx

Noctua also has some 3U-compatible coolers that might just be cheaper than Supermicro's stuff.

 

[Hobbel]

They don't seem to be included. Their stuff is listed here:
http://www.supermicro.com/ResourceApps/Heatsink_Matrix.aspx

Noctua also has some 3U-compatible coolers that might just be cheaper than Supermicro's stuff.

I thought, I only would need some fan for CPU?

what about this table?

 

[marbus90]

That's chassis fans, not CPU heatsinks.

Check with sona.de, they can build the server as well and would see which parts and cables are missing. The hardware is worth thousands, so add another couple bucks to get a working system, not something which goes up in flames the instant you connect the A/C cord.

 

[Bidule0hm]

I would recommend 3 TB drives over the 2 TB ones, it's probably cheaper in TB/$ and add a bit of future proofing ;)

 

[Hobbel]

That's chassis fans, not CPU heatsinks.

Check with sona.de, they can build the server as well and would see which parts and cables are missing. The hardware is worth thousands, so add another couple bucks to get a working system, not something which goes up in flames the instant you connect the A/C cord.

:D true story.
But I won't forget about the CPU ;)

I think Ericloewe was asking for chassis fans?

 

[Hobbel]

I would recommend 3 TB drives over the 2 TB ones, it's probably cheaper in TB/$ and add a bit of future proofing ;)

adds 60 bucks per HDD (0,06 €/GB for 2, 3 or 4 TB disks). with 8 HDDs this would be 480 € more, knocking at the limit. Would like to have some room for money to spend, perhaps for adding some other stuff. There are also 8 more bays to fill to get future proof ;)
Thanks for your opinion.

 

[Ericloewe]

:D true story.
But I won't forget about the CPU ;)

I think Ericloewe was asking for chassis fans?

No, the chassis fans are included. It's just the CPU cooler that isn't (in part because it depends on the motherboard).

 

[Hobbel]

Regarding CPU cooler, I thought about the SNK-P0048PS or SNK-P0048PSC

 

[Ericloewe]

Either one should be ok.

 

[marbus90]

adds 60 bucks per HDD (0,06 €/GB for 2, 3 or 4 TB disks). with 8 HDDs this would be 480 € more, knocking at the limit. Would like to have some room for money to spend, perhaps for adding some other stuff. There are also 8 more bays to fill to get future proof ;)
Thanks for your opinion.

No, more and faster spindles are needed for a VM workload. Do not use bigger, slower/fewer disks.

 

[Hobbel]

No, more and faster spindles are needed for a VM workload. Do not use bigger, slower/fewer disks.

Thanks for this hint. I learned this lesson before - from cyberjock ;)

This is why I go with the 16 bay chassis, but only use 8 for now. So there is no need of increasing storage by replacing the used 2 TB with 4 TB disks. I would also recommend SAS HDDs when adding more disks in future.

 

[marbus90]

thing is, you can't replace the 8 SATA disks with SAS disks. I'd rather go with an expander from the start or use a board without onboard SAS -> adding 2x LSI 9207-8i or Supermicro AOC-S2308L-L8E (same chipset) later on.

CSE-836BA-R920B + X10SRL-F = 1,420.36EUR
CSE-836A-R1200B + X10SRH-CLN4F = 1,632.13EUR (initial combination)
CSE-836BE1C-R1K03B + X10SRH-CLN4F = 1,854.98EUR

which means initial cost is down by 211.77EUR as well. The 836BA-R920B includes the option for rear hotswap SSDs as well: MCP-220-83605-0N this time around.

 

[Hobbel]

thing is, you can't replace the 8 SATA disks with SAS disks. I'd rather go with an expander from the start or use a board without onboard SAS -> adding 2x LSI 9207-8i later on.

I don't see any reason (from the customer point of view), why the SATA disks should be replaced by SAS. But in this unlikely case, you are right. The LSI 9207-8i costs double of the (often recommended) IBM ServeRAID M1015. Is it worth the money?

EDIT:
I will stick to the board, because it has 4 GbE-Ports and is only 40 € more expensive than the X10SRH-CF.

 

[marbus90]

I edited the posting above with some new/better SKUs. For a business DO NOT EVER buy something from ebay and crossflash it to other branded firmwares. The Supermicro AOC-S2308L-L8E already comes with the IT mode firmware -> less hassle with LSI/Supermicro support.

My POV on SATA -> SAS: if you're going to replace the existing 2TB disks with bigger ones, why not go SAS. it's a small markup for enterprise drives anyway and you have more SATA ports free for DOMs, SLOG SSDs, L2ARC, whatnot.

Also, if you consider a TrueNAS later on: it only accepts SAS drives ;) I'd also only use 15 HDDs max. -> Striped 7x Mirrors, one hot spare plus one slot free for a high end SLOG device like the ZeusRAM.

I mentioned the X10SRL-F, not the SRH-F. The SRL-F does not come with a SAS controller -> cheaper.

 

[Hobbel]

I edited the posting above with some new/better SKUs. For a business DO NOT EVER buy something from ebay and crossflash it to other branded firmwares. The Supermicro AOC-S2308L-L8E already comes with the IT mode firmware -> less hassle with LSI/Supermicro support.

My POV on SATA -> SAS: if you're going to replace the existing 2TB disks with bigger ones, why not go SAS. it's a small markup for enterprise drives anyway and you have more SATA ports free for DOMs, SLOG SSDs, L2ARC, whatnot.

Also, if you consider a TrueNAS later on: it only accepts SAS drives ;) I'd also only use 15 HDDs max. -> Striped 7x Mirrors, one hot spare plus one slot free for a high end SLOG device like the ZeusRAM.

I mentioned the X10SRL-F, not the SRH-F. The SRL-F does not come with a SAS controller -> cheaper.

I see... U know what u are talking about ;)

For me, the problem is, that the CSE-836BA-R920B is listed for SAS and SATA drives by Supermicro, but the distributors just mention SAS drive bays?! Who is right? (if it accepts SATA, it's the one I go with)
The X10SRL-F is cheaper, but also lacks of 2x 1 GbE-Ports. To get around this issue with an INTEL Dual-Port Adapter, it costs the same.
I like your idea of using the AOC-S2308L-L8E from the beginning. But I have a better feeling about the LSI 9207-8i. (I would never buy used parts on eBay. I thought about buying new hardare and reflash it. Don't have that much concernes about reflashing firmware)

 

[marbus90]

The AOC-S2308L-L8E and the LSI 9207-8i are based off the same chipset. IBM and Dell use a custom firmware on their cards whereas the Supermicro cards don't need to be /cross/flashed, just flashed to the right version. 836BA and A are build for SATA and SAS drives alike since SAS bays are always compatible to SATA HDDs.

10Gbe is the future -> rather plan for that instead of using LAGG/MPIO. Single faster interfaces are easier to handle and implement than multiple smaller interfaces.