BUILD Full flash build - RFC

[David E]

Chassis and MOBO: SSG-2028R-E1CR48L https://www.supermicro.com/products/system/2U/2028/SSG-2028R-E1CR48L.cfm
CPUs: (2) 8C/16T E5-2620V4 2.1G 20M 8GT 85W R3 2011 R0
RAM: (8) MEM-DR432L-CL01-ER24 32GB DDR4-2400 2Rx4 LP ECC REG DIMM
NIC: AOC-MTGN-I2S-O AOC-MTGN-I2S RETAIL PACK
SLOG: Intel DC P3700 400GB, NVMe PCIe3.0,HET,MLC AIC 20nm
L2 ARC: Intel DC P3600 800GB, NVMe PCIe3.0,MLC AIC 20nm
FLASH: (24) Samsung 850 EVO,1TB,SATA 6Gb/s,VNAND,2.5",7.0mm,19nm
BOOT: Samsung 850 Pro,128GB,SATA 6Gb/s,VNAND,2.5",7.0mm,19nm

Usage is as primary storage for multiple VMware servers.

Planning to run it as 8 vdevs that are 3 way mirrored, for 8TB of usable capacity. This is probably overkill in the IOPS department, but to get the chassis through Supermicro you have to buy 24 drives minimum.. my ideal would have been to get 12 2TB drives to maximize future capacity expansion. But as is I still have 24 more drive bays that can be expanded to.

I'm curious on any comments anyone has, particularly if anyone has used this kind of chassis before. I'm also considering running dedupe, which (I assume) should be viable with 256GB of RAM, but I'm curious if anyone has documented how dedupe affects performance.

Thanks!

 

[Robert Smith]

Multiple VMware servers as in more than two? Have you looked into clustering with vSphere HA and Virtual SAN?

 

[Mlovelace]

Two things:
1. The E5-2620V4 CPU isn't the best choice for a storage server. The E5-2643V4 is going to be a much better CPU for the filer and give you better performance overall.

2. The AOC-MTGN-I2S-O AOC-MTGN-I2S uses the Intel X520 chip and isn't the best option for freeNAS. You can get decent speeds with tuning but a Chelsio card would be better.

 

[Mlovelace]

Multiple VMware servers as in more than two? Have you looked into clustering with vSphere HA and Virtual SAN?

This is a great option if your hosts have enough local storage and you get some NVMe SSDs for them.

Edit: and 10Gbe management network.

 

[depasseg]

There aren't many all flash builds around.

I don't think you will get much, if any, use out of the dual procs (except extra RAM slots).

The SLOG is a great device, but you could easily get away with a fifth of that size (assuming you could find some NVMe SSD's smaller).

And I would mirror the boot drives (it makes dealing with drive failures so much easier).

 

[David E]

Multiple VMware servers as in more than two? Have you looked into clustering with vSphere HA and Virtual SAN?

Two currently, but probably more on the way in the next 12mo

Considered it, the things I didn't love were:

• Cost as it scales out
• No native access via protocols
• I'd have to buy a backup solution since I couldn't just send ZFS snapshots around, which tends to be Veeam, and isn't particularly cheap, and would require more hardware and a learning curve
• And frankly I have tons of problems with ESXi anytime I do anything outside of creating/deleting VMs, often requiring multiple reboots, which would make me nervous for coupling the integrity and availability of my data to an already troublesome system

What I did like

• Really low latency to data if it is stored on the server doing the computation
• HA from a controller/hardware perspective

 

[David E]

Two things:
1. The E5-2620V4 CPU isn't the best choice for a storage server. The E5-2643V4 is going to be a much better CPU for the filer and give you better performance overall.

Agreed, the problem is the ~$1100 price difference per cpu! Intel is putting a heavy premium on the higher clocked versions of their E5 CPUs, even at low core counts.. pretty annoying, but I'm guessing they are doing this due to binning.

2. The AOC-MTGN-I2S-O AOC-MTGN-I2S uses the Intel X520 chip and isn't the best option for freeNAS. You can get decent speeds with tuning but a Chelsio card would be better.

Yeah unfortunately the SIOM for that motherboard only has Intel options. We've got a couple of these cards and haven't noticed any issues, is there anything in particular I should expect to see misbehaving from them? Iperf shows them able to hit >9Gbps in each direction. I could burn one of the 3 PCIE slots on a Chelsio card, but would the speed difference really warrant that?

 

[Mlovelace]

What I did like

• Really low latency to data if it is stored on the server doing the computation
• HA from a controller/hardware perspective

You should look into a Nimble array, maybe a CS215, looking at what you're currently spec'ing.

 

[Mlovelace]

.Yeah unfortunately the SIOM for that motherboard only has Intel options. We've got a couple of these cards and haven't noticed any issues, is there anything in particular I should expect to see misbehaving from them? Iperf shows them able to hit >9Gbps in each direction. I could burn one of the 3 PCIE slots on a Chelsio card, but would the speed difference really warrant that?

The Intel chip has the occasional driver issue pop up, usually associated to off-load (last one was tso specific) that will make them go catatonic.

 

[David E]

There aren't many all flash builds around.

I don't think you will get much, if any, use out of the dual procs (except extra RAM slots).

Yeah I'd tend to agree. The only other thing in this case is PCIE lanes, all 3 of the external slots are fed by the second CPU, so if I dropped that I would have practically no connectivity.

There aren't many all flash builds around.
The SLOG is a great device, but you could easily get away with a fifth of that size (assuming you could find some NVMe SSD's smaller). :)

I know right! Major overkill in size terms, but I couldn't find a better alternative since speed is so crucial.

There aren't many all flash builds around.
And I would mirror the boot drives (it makes dealing with drive failures so much easier).

FreeNAS can't do this natively, correct? I'd have to find out what SATA ports are hooked up to those two backplane bays since the onboard LSI chip is running in IT mode.. if it was motherboard ports then theoretically I could use Intel's chipset raid?

 

[Mlovelace]

FreeNAS can't do this natively, correct? I'd have to find out what SATA ports are hooked up to those two backplane bays since the onboard LSI chip is running in IT mode.. if it was motherboard ports then theoretically I could use Intel's chipset raid?

It can natively, on install you select the two drives as install targets and it will mirror them.

 

[David E]

You should look into a Nimble array, maybe a CS215, looking at what you're currently spec'ing.

I looked at them, but I want off of mechanical disks completely, and they quoted me north of 70k for an AF3000, when I could build the above system for 15-20k depending on support level. It would also require me to invest at least another 20k into another array for DR, and/or a backup vendor like Veeam.

 

[David E]

It can natively, on install you select the two drives as install targets and it will mirror them.

Nice! Didn't know this, I'll definitely get another disk, thanks!

 

[Mlovelace]

We have a Nimble CS500 and a CS300 which are their hybrid arrays and the CS500 will do 90,000+iops (real-world I've seen it) and the CS300 will consistently break 30,000+iops both with sub-milisecond latency. Mechanical drives aren't a problem.

Now the AF3000 will do 300,000+iops but I'm not sure how that will compare to the array you're building.

For me, and for that kind of money, I would want dual controllers and enterprise support.

 

[David E]

We have a Nimble CS500 and a CS300 which are their hybrid arrays and the CS500 will do 90,000+iops (real-world I've seen it) and the CS300 will consistently break 30,000+iops both with sub-milisecond latency. Mechanical drives aren't a problem.

Now the AF3000 will do 300,000+iops but I'm not sure how that will compare to the array you're building.

For me, and for that kind of money, I would want dual controllers and enterprise support.

What is the price range on the cs300? And those IOPS numbers are random data non-sequential writes?

 

[CheckYourSix]

The 850 EVOs are warrantied for what, 150 TBW? I know the tests that people have done show they'll far exceed that amount, but I'm curious how much data you'll be writing to each per day and how long you expect the drives to last?

 

[David E]

The 850 EVOs are warrantied for what, 150 TBW? I know the tests that people have done show they'll far exceed that amount, but I'm curious how much data you'll be writing to each per day and how long you expect the drives to last?

Anandtech believes the flash is good for ~2000TB worth of writes. I checked my array and over the last month it sunk around 60T of writes. So if that was evenly sprayed across 8 vdevs I'd be good for 22 years. I'm sure the write level will increase, but if I turn dedupe on it should also help to keep amplification down, and realistically with tech like 3D xpoint on the horizon from intel, this array will almost be guaranteed to be replaced with a faster better primary array in the next year or two.

 

[Mlovelace]

What is the price range on the cs300? And those IOPS numbers are random data non-sequential writes?

Yes random and the CS300 came in around $45k. I had a CS215 quoted for a branch office around $12k

 

[Robert Smith]

[*]I'd have to buy a backup solution since I couldn't just send ZFS snapshots around, which tends to be Veeam, and isn't particularly cheap, and would require more hardware and a learning curve

Unless you are only interested in crash-consistent snapshots (unlikely) do not forget to configure FreeNAS VMware aware snapshots.

Also consider the speed and storage requirements for sending snapshots around. In many [though not necessarily all] cases this means building two [or more] FreeNAS servers. FreeNAS breeds more FreeNAS, LOL.
HTH

 

[wreedps]

How did this build go?