BUILD Enterprise Build Critique

[mgb]

Greetings humans,

I'm in the "requirements gathering" stage of a new enterprise build and I'd like some feedback / recommendations on the hardware I'm looking at purchasing.

Use Case: Serve around ~10 ESXi VMs over iSCSI and provide SMB shares for ~40 Win7 workstations (corporate file share + workstation backups).

I've narrowed my options down to the following:

Chassis (option 1): SM CSE-826BE2C-R920LPB 2U 12-bay (Dual Expanders, 920W Redundant PSUs)
Chassis (option 2): SM CSE-846BE26-R920B 4U 24-bay (Dual Expanders, 920W Redundant PSUs)
Motherboard (UP): SM X10SRL-F
Motherboard (DP): SM X10DRi
CPU(s): 1x (or 2x) Intel Xeon E5-2620v3 (Hex-core, 15M Cache, 2.40 GHz)
HBA: 1x LSI SAS 9207-8i
RAM: (..to be determined...)
OS: 2x (or 4x) SM SuperDOM Flash (64GB)
ZIL: 1x Intel DC S3700 100GB
L2ARC: 1x Intel DC S3500 480GB
Data: 4x HGST Ultrastar 7K400 SAS2 4TB (1x3-deep mirror + 1 hot spare)

My thoughts...

Chassis: As of now, we're using 2x3-deep mirrors with 2TB WD RE4 HDDs (~2TB consumed out of ~4TB usable). If I opt for the 12-bay system, I could get a maximum ~12TB (3x3-deep mirrors + 3 hot spares). I'm of the paranoid type so I prefer 3-way mirrors + 1 hot spare / vdev. A 24-bay system would give me lots of room to grow.

Motherboard: Since our workload is quite lite, I'm leaning towards the UP board. The overall price difference of the DP board + an extra CPU is ~$730. The only benefits I can think of is extra RAM slots + 12-cores (24-threads) over 6-cores (12-threads). I don't think our workload warrants this price increase.

ZIL: I've considered mirroring the ZIL but since the chassis would use SAS2 drives + dual expanders, my thoughts are to keep the front bays reserved only for data drives and use the optional 2x2.5" hot-swappable cage in the rear for the ZIL + L2ARC.

Data: I understand the benefits of dual-ported SAS drives, but is this really necessary unless your using dual expanders?

An alternative would be to opt for the 4U / 24-bay with either a single expander (or direct cabling) and use SATA3 data drives. Then I would put all drives (OS + ZIL + L2ARC) in the front bays and I could mirror the ZIL + have the option of concatenating multiple L2ARC drives. I've heard that expanders can go bad, so maybe I'd be better off with direct cabling + an extra HBA (I have no experience with expanders).

Am I going overboard here with this CPU + Motherboard combo?

Any feedback is greatly appreciated!

Thanks,
--mgb

 

[Darren Myers]

I would contact iX systems....This sounds like a job for them, i dont think i would FreeNAS it....the fact youre using iSCSI means your total useable pool should be no more then 50% due to fragmentation.

 

[Arwen]

Greetings humans,
...

<humor>Speciest! How dare you assume we are all human. I know for a fact that not
all of us members are human. Certainly not me, (I'm an Elven Princess!).</humor>

Back to reality, Darren Myers has a point.

Other options include checking out pre-assembled servers from HP, Dell, Oracle and
other vendors that have roughly what you need. You may pay more up front, but pay
less in your time supporting a custom build.

Putting together an Enterprise file server can be tricky for lots of reasons. For example,
warranty on the components may come from different vendors. Or compatibility issues
may arrise long after the server has past it's initial burn it at site. Like performance issues
when you get a year or 2 into service, with no one to call.

Last, the SuperMicro DOM Flash 64GB is a bit much. The 32GB should be more than
suitable and has similar speed to the 64GB one. Where the SM 16GB is noticably slower...
Mirrored 32GB DOMs should allow you to move the system dataset over, off your main
Zpools. Even 32GB is pretty larger, though I have not seen a 16GB SATA DOM with both
reliability in reviews, and decent speed.

 

[mgb]

Thanks for the feedback guys.

@Darren Myers

As much as I would love to purchase from iX (and further support the community), I'm afraid the additional costs of shipping + import taxes (I'm in Europe) would be upwards of +20% of the cost of the build. Saying that, I did reach out to iX and unfortunately reached a dead end (... I'll leave it at that).

I was not aware of the 50% iSCSI recommendation (although it's spelled out in the FreeNAS docs). This is going to bump my price up a little as I'm afraid I'm going to need 4TB drives instead of 2TB drives to be able to fulfill this.

@Arwen

I never considered purchasing pre-assembled servers. Are you suggesting purchasing one of these and using their proprietary storage software? I don't have any experience using one of the name brand appliances. For the last 4 years we've been running a custom build SuperMicro system with OpenIndiana serving iSCSI, NFS and CIFS to our small/medium business LAN (and SAN). Now that FreeNAS has progressed leaps and bounds since the 2011 days, I would like to switch.

About the warranty, I'm fortunate enough to have a local vendor that can source all the individual parts and provide the warranty directly to me - I deal with them, they deal with the suppliers / manufacturers. This makes it a little easier for me. As far as having no one to call, I guess that's the risk I take using FOSS as our storage platform.

Lastly, I'm leaning away from putting DOMs in the case - I'd prefer not to have to open the case if one of the OS DOMs kicks the bucket. My solution for this is to install the 2x2.5" rear drive carrier and install 2x Intel DC S3500 80GB drives for FreeNAS. I know this seems like overkill, but in my area, the 32GB DOM is $62 and the 80GB SSD is $106 - only $88 more for 2 drives + $42 for the cage. A small price to pay considering these SSDs will be much more reliable than the DOMs and I can use them for the system dataset as well.

Now my build looks like this:

Chassis 1x CSE-846BA-R920B* + 1x MCP-220-84606-0N
Motherboard 1x X9DR7-LN4F
CPU 2x Xeon E5-2620v2
RAM 8x8GB DDR3
HBA 2x 9207-8i
OS 2x Intel DC S3500 80GB
L2ARC 2x Intel DC S3500 240GB
ZIL 2x Intel DC S3700 100GB
HDD 20x 4TB 7K4000 SATA3

* (edit): Do you think the 920W would be sufficient or I should opt for the 1280W PSUs?

Using 3-deep mirrors with 2 hot spares, I should have a total usable capacity of ~22TB. Right now I'm in the danger zone of 3.21TB used with only 366GB free (~90% capacity). I have to do something quick as I don't have enough slots in my existing system to add another 3-deep mirror (poor planning on my part).

Thanks again for the feedback, and if you have any comments about the new configuration, I would greatly appreciate it.

--mgb

 

[Arwen]

Now that you have explained, I understand your situation. Yes, a SuperMicro with FreeNAS is an option.
Might be better in someways than OpenIndiana, since FreeBSD is possibly more active for updates than
OpenIndiana.

I agree with your desire to use 2.5" flash drives for the OS. One thing you can do to make them more reliable,
is reduce their over-all storage. I am not sure of the exact method, but basically turn a 80GB drive into a
70GB drive. That gives you 10GB more spares and wear leveling. Thus, they may last the entire life of your
storage server. Especially so because you are going mirrored.

Most people here who have had there FreeNAS for a while, have had to replace their OS flash drives at least
once. Never having to do it after initial purchase, saves money.

 

[mgb]

@Arwen

I think you're talking about "over-provisioning" the SSDs. More than likely I will do this and reduce them to about 40GB (~50% over provisioned). Considering how little FreeNAS touches these drives, I shouldn't have any issues.

After thinking some more about my storage & performance requirements, I think I'm going to scale back on the CPU and go with a single socket X10 board and a 2U case.

This is what I'm looking at now:

Chassis CSE-826BA-R920LPB
Motherboard X10SRL-F
CPU Xeon E5-2620v3
RAM 8x8GB DDR4 *
HBA 1x 9207-8i
OS 2x Intel DC S3500 80GB
L2ARC 1x Intel DC S3500 240GB
ZIL 1x Intel DC S3700 100GB
HDD 10x HGST 7K4000 4TB SATA3 (3x3-way mirrors + 1 hot spare)

* RAM: Since this board only has 8 slots, maybe I'm wiser to start with 128GB of RAM from the get go?
** HBA: I intend to wire 8 bays to the board and 4 to the HBA. This way if I loose the HBA, I won't loose the pool.

Overall this will give me ~12TB of storage (which is ~3x what I have now) with 3-way mirrors + 1 hot-spare, which I think is more than enough redundancy.

Any thoughts / comments / feedback is greatly appreciated.

Thanks!

--mgb

 

[Arwen]

Yes, "over-provisioning" is the term I was looking for.

Another thing you can do with mirrored, reliable OS flash, is move the system dataset to that pool.
My OS pool is mirrored with 2 x SATA DOMs, (I did not have any more 2.5" disk bays). With the
system dataset on the OS pool, in theory I can have my data pool's disks spin down. Then if data is
needed, they spin up.

I don't understand how you plan to setup your disks, but if you do, great.

 

[HoneyBadger]

This is what I'm looking at now:

Chassis CSE-826BA-R920LPB
Motherboard X10SRL-F
CPU Xeon E5-2620v3
RAM 8x8GB DDR4 *
HBA 1x 9207-8i
OS 2x Intel DC S3500 80GB
L2ARC 1x Intel DC S3500 240GB
ZIL 1x Intel DC S3700 100GB
HDD 10x HGST 7K4000 4TB SATA3 (3x3-way mirrors + 1 hot spare)

* RAM: Since this board only has 8 slots, maybe I'm wiser to start with 128GB of RAM from the get go?
** HBA: I intend to wire 8 bays to the board and 4 to the HBA. This way if I loose the HBA, I won't loose the pool.

Overall this will give me ~12TB of storage (which is ~3x what I have now) with 3-way mirrors + 1 hot-spare, which I think is more than enough redundancy.

Any thoughts / comments / feedback is greatly appreciated.

Thanks!

--mgb

I like the look of your updated build, but would like to suggest a few more tweaks.

Since you have only 8 RAM slots, go with 4x16GB DDR4 DIMMs or better yet go for 128GB out of the gate.

The "ZIL" (actually called an SLOG when it's a Separate LOG device) might not need to be that big in size, but the 100GB S3700 is significantly slower than the 200GB S3700 or its newer sibling the S3710 (which doesn't come in 100GB).

100GB S3700: 500/200 r/w MB/s, 75K/19K r/w IOPS
200GB S3700: 500/365 r/w MB/s, 75K/32K r/w IOPS
200GB S3710: 550/300 r/w MB/s, 85K/43K r/w IOPS

The S3710 has slower theoretical sequential write speeds, but SLOG performance is largely dependant on single-queued write IOPS at low latencies, so the 200GB S3710 is probably your best choice.

Similarly the L2ARC and boot SSDs can be replaced by the S3610 as they get a significant improvement in durability (3 drive writes per day vs 0.35 in the S3500) which will make them last longer. Not as important for the boot drives but can help if you get a bit of churn on your L2ARC.

I didn't see an additional NIC in the build but I assume you're going to run something more than just the dual-GbE onboard, or all that delicious performance will go to waste being bottlenecked on the network side.

And finally, I identify as Badgerkin, so check your robot privilege. ;)

 

[mgb]

@Arwen

Here's a picture of the chassis layout I'v come up with so far. I was thinking of keeping things simple and just using 1 pool that will be divided into iSCSI + SMB datasets. iSCSI for the ESXi VMs and SMB for the corporate file shares / backups.

@HoneyBadger

Thanks for the info about the S3610! I did not realize it used High Endurance Technology (HET). I just asked my supplier for a price and I think I can still keep it in budget.

For the RAM, I've decided might as well get 128GB out of the gate. Will never hurt to have more ARC.

So now I'm looking at these non-data drives:

OS 2x Intel DC S3500 80GB (I think these are the best for the price considering they will rarely be touched and severely over-provisioned)
L2ARC 1x Intel DC S3610 400GB (might as well opt for 400GB with 128GB RAM = >500GB cache)
ZIL 1x Intel DC S3710 200GB

Regarding the extra NIC, I'm going to drop a 2x1 GbE NIC dedicated for the SAN (iSCSI multipath) between the ESXi servers and the NAS. There will also be an additional 1x1 GbE NIC directly connected to the backup NAS for ZFS replication.

Looking over my build sheet, my other 2 options are as follows:

Add $550, X9DR7-LN4F + 2x E5-2620v2 @ 2.10 GHz (bonus: 4x1 GbE on-board)
Add $1000, X10DRi + 2x E5-2620v3 @ 2.40 GHz (the X10DRi-LN4+ won't fit in the 2U case with the rear 2x2.5" SSD cage installed)

Obvious advantage of dual vs. single socket is 12 cores (24 threads) > 6 cores (12 threads). The advantages of the v3 over the v2 would be clock speed (2.4 > 2.1) and QPI speed (8GT/s > 7.2 GT/s).

What do you guys think... more CPU or stick with my current build?

Thanks!

--mgb

 

[HoneyBadger]

I'd think a single E5 should be more than enough. What I'd look at as another option would be buying the dual-socket X9 board but only populating one socket to start with. You can fill it later if you need more CPU/RAM.

Considering what you're hoping to serve I'd look at 10Gbps networking if your switch has the uplink capabilities. If not then I'd split your MPIO across the board and card (as in "one port on each") so you've got an extra piece of redundancy, then LACP the link you use for SMB. Give me a bit and I'll make an ugly diagram of what I'm thinking.

 

[HoneyBadger]

Ugly diagram ahoy. I think you'll want a quad-port NIC so that you have redundant links for SMB and replication as well as two for MPIO.

This is assuming you have to share switches between data and IP SAN. Obviously if you have a separate IP SAN just use those.

 

[mgb]

@HoneyBadger

Thanks for putting in the effort to draw a diagram! This is exactly the setup I'm using now - splitting the LAGG across the on-board + add-on NICs. Here's how I intend on wiring the servers (sharing the switches for access + SAN):

I'm not splitting the iSCSI networks on either of the FreeNAS boxes because if the motheboard dies, redundancy is out the window.

As far as the build goes, I think I'm going to opt for the dual socket X9 and just get the 2nd CPU from the get go. My plan is to convert most of the Linux VMs into jails and only use one ESXi machine for the remaining Windows VMs. Maybe one day (FreeNAS 10?), when bhyve supports Windows, I can do away with ESXi all together and run a true "all-in-one" box hosting everything from FreeNAS. That'll be the day!

So the final build looks like this:

Chassis 1x CSE-826BA-R920B
Motherboard 1x X9DR7-LN4F-JBOD
CPU 2x Xeon E5-2620v2
RAM 8x16GB Samsung DDR3
HBA 1x 9207-8i
OS 2x Intel DC S3500 80GB
ZIL 1x Intel DC S3710 200GB
L2ARC 1x Intel DC S3610 400GB
HDD 10x 4TB 7K4000 SATA3 (3x3-way mirrors + 1 hot spare)
NIC 1x Intel I350-T2

Any other thoughts before I pull the trigger?

Thanks!

--mgb

 

[HoneyBadger]

Glad my ugly diagram helped. (Yours looks much nicer.)

Only other thing that I might suggest would be mirrored SLOG devices (because "corporate") and don't run too many or depend too much on jails. Especially if the software you're running has vendors that insist on a specific hypervisor (most won't support anything but VMware). 128GB is a lot of RAM for a fileserver but can run out faster than you think for a VM host.

Anyone else want to critique?

 

[mgb]

I thought about mirrored SLOGs, and I could only come up with 2 reasons to use one:

1. Data Loss (rare): there exists a minuscule edge case where you could incur data loss using a non-mirrored SLOG. If SYNC writes are pending in RAM and on the SLOG and you incur an immediate power loss, theoretically the data should be safe as it exists on the SLOG. However, when the machine comes back online and the SLOG has failed, the data has been lost. This can easily be solved by using a UPS and a proper shutdown sequence. In my environment, every server has redundant PSUs which are each connected to separate UPSes.
   
2. Performance: if the SLOGs fails when your depending upon it for SYNC writes, the pool performance will suffer (and therefore the ESXi VMs will suffer). This is all assuming you force SYNC writes for iSCSI (which is not true by default)

I'm not too concerned with the data loss as I have the proper UPS and shutdown sequence configured (my rack will shut itself off and power itself on if there's ever an extended power outage that depletes the batteries). Regarding performance, since iSCSI is async by default (I intend to force it SYNC for sanity), if the SLOG ever fails, I can simply revert back to async until the SLOG is replaced. This would leave me in a "vunerable" state while the SLOG is offline, which is the default anyways with async iSCSI. I think this is a risk I can accept.

That is by no means an exhaustive list, it's just what I've found from the research I've done.

When I say "corporate", I mean SMB (small/medium business). The only software vendor we have requires Windows Server + MSSQL (hence the Windows ESXi VMs). Everything else is currently FOSS I configured to run in Linux VMs: apache (want to switch to nginx), postfix, dovecot, mysql (want to switch to mariadb), owncloud, etc. If I can move these into jails on the NAS, I wouldn't have to worry about the SAN connection as everything's running from the same host.

About the RAM, I agree 128GB is quite a lot. I might scale back to 4x16GB (or 8x8GB since I have 16 slots) to save a little money. It all depends on the mood of the boss when I finally share the total cost of the build.

 

[HoneyBadger]

You're right in that it's a very, very minor edge case; you'd need to have a double UPS failure and your SLOG die, all simultaneously. You've definitely done your homework here. My thought on it at build time was "well, I can reduce/eliminate even those edge cases with another small investment, so I might as well." But hey, see signature line for my thoughts on overkill. ;)

Looks like everything is set personally. If you do cut back on the RAM I'd say stick with the bigger 16GB DIMMs as they appear to be cheaper per-GB (at least from my North American distributors, might be different on the other side of the pond) and leave you with more slots down the road.

 

[mgb]

I agree with you in wanting to reduce / eliminate even the smallest of edge cases. However, if I were to use a mirrored SLOG, it would throw my whole chassis configuration off:

Either I sacrifice the L2ARC or Spare, or I'd have to get a bigger chassis (would opt for 4U over a 3U) + HBA + SSD . It would cost roughly another $1K just to cover that edge case. I'm not sure it's worth it.

On the flip side, another argument for the larger chassis would be flexibility. It would give me options down the road: Maybe I need more storage. Maybe I need a pure SSD pool. etc.. Going with a 2U chassis and filling it from the get go, I'm kind stuck with this configuration for the long haul. Once I hit capacity, I'm going to have to spring for 10x 6TB drives just to get a 50% space increase (albeit ~12TB to ~18TB which is a gain of ~6TB which isn't too shabby).

I think I've gone and almost talked myself into getting the 4U.