BUILD Build from replaced enterprise equipment

[Turgin]

Hello,

My background is a senior level network engineer for a large energy company so I feel I am reasonably tech savvy but I don't get to "play" with storage and servers at work. I just want to build a decent home lab mostly for fun, but I do have about 1TB of the usual family data of pictures, movies, etc that I'd like to keep safe as well.

We are doing a hardware refresh in one of our test/dev environments at work and I will be acquiring the
following parts from Cisco UCS blades and NetApp filers:

4 x Xeon X5570 4c/8t @ 2.933 Ghz LGA1366
2 x Xeon X5670 6c/12t @ 2.933 Ghz LGA1366
18 x 4GB DDR3-1333 ECC RAM
12 X 8GB DDR3-1333 ECC RAM
8 x 16GB DDR3-1066 ECC RAM
2 x Intel 710 series 100GB SSD
14 x Hitachi DK7SAA200 2TB SATA2 7200 RPM drives

I want to build at least one FreeNAS box and an ESXi host. Maybe build an additional FreeNAS for backups. We're about 2 months away from being done with the refresh so I'm using this time to read, read, read and plan, plan, plan.

I'm looking for the best deal I can find on two (or three) Supermicro X8DTE-F motherboards (or similar) on eBay. Also researching cases, power supplies, and CPU heatsink/fans.

My questions:

1. Are the Hitachi drives acceptable for now? I can't find much information on them but can I assume that since they came from enterprise class storage they are up the the task? Assuming they are and I plan to use the 6 on-board SATA ports for now I think the best design is 3 pairs of mirrored vdevs. I plan to replace the Hitachis with WD Reds as time goes on since the drives are so old, but I will have 8 on-shelf spares for now if I don't build 2 FreeNAS.

2. How should I divide up my available RAM? Is there a point of diminishing returns? If so, should 72GB (6*8GB + 6*4GB) get me there or is more always better? I ask because I assume I shouldn't mix the speeds (1066 and 1333) and that would give me enough chips for 2 FreeNAS boxes of my fastest RAM with way more than enough left for my ESXi host. I could also put the 8x16GB for 128GB in the primary FreeNAS, 12x8GB for 96GB in the secondary, and that would leave me with 48GB for the ESXi box which is plenty I am sure for my needs.

3. Are dual X5570 a decent enough fit for this? I figure the ESXi box would benefit more from the 6 cores in the X5670.

4. Would the Intel 710 series 100GB SSDs be suitable for SLOG? I've read some on SLOG and ZIL and I'm certainly no expert and I'm unsure if I'll even benefit from it but it might be something to play with further down the road.

TIA for any advice.

 

[Mirfster]

Are the Hitachi drives acceptable for now?

Sure, just note that these are 3gbps and not 6gbps; as well they are 7200 RPM (I use 7200 RPM drives, but others may say there is not any real value in it). Also, make sure to do proper burn in testing (some links are in my sig)

I think the best design is 3 pairs of mirrored vdevs.

If you let us know what the use-case is for FreeNas, suggestions could be made.

How should I divide up my available RAM? Is there a point of diminishing returns?

FreeNas loves RAM, if you are getting up in total though (~100 GB) you may want to think L2ARC. Honestly though depending on your actual use-case for FreeNas 48GB may be plenty and then you would not need to worry about L2ARC.

Are dual X5570 a decent enough fit for this? I figure the ESXi box would benefit more from the 6 cores in the X5670

I would think so as well, best to leverage the hex core for ESXi.

Would the Intel 710 series 100GB SSDs be suitable for SLOG? I've read some on SLOG and ZIL and I'm certainly no expert and I'm unsure if I'll even benefit from it but it might be something to play with further down the road.

If your using only 48 GB of ram, no need to be concerned with this.

 

[Turgin]

Thanks for the reply.

Sure, just note that these are 3gbps and not 6gbps; as well they are 7200 RPM (I use 7200 RPM drives, but others may say there is not any real value in it). Also, make sure to do proper burn in testing (some links are in my sig)

I figured since the X8 on-board SATA ports were also 3GBps it didn't matter. What's the concern about 7200RPM? Burn in testing will be done for sure.

If you let us know what the use-case is for FreeNas, suggestions could be made.

As I see it now to start:

• ~2 TB for a CIFS share for 4 or 5 Windows clients to use as storage including backing up their local disks. I have just under 1TB of stuff to put there initially.
  
• 2 or 3 TB for iSCSI targets for my ESXi host. Guests will be Linux and Windows. Probably a Minecraft server for my son. Probably internal DNS, root CA, web and ftp server, etc. Nothing super high bandwidth.
  
• Some space for snapshots

This article I thought had some interesting pros to using striped mirrored vdevs. I don't have a problem with the 50% storage penalty given my very limited storage needs at the moment. Copy and paste from the summary:

for a given number of disks, a pool of mirrors will significantly outperform a RAIDZ stripe.
a degraded pool of mirrors will severely outperform a degraded RAIDZ stripe.
a degraded pool of mirrors will rebuild tremendously faster than a degraded RAIDZ stripe.
a pool of mirrors is easier to manage, maintain, live with, and upgrade than a RAIDZ stripe.

A pool of mirrored vdevs feels like RAID10 of which I am a big fan.

FreeNas loves RAM, if you are getting up in total though (~100 GB) you may want to think L2ARC. Honestly though depending on your actual use-case for FreeNas 48GB may be plenty and then you would not need to worry about L2ARC.

This could be a testing point for me as part of build out and burn in. Baseline at say 48GB then add memory in 8 or 16 GB chunks and retest.

Really what I want is to do everything I can, within reason and my budget, to be as close as possible to the NICs being the bottleneck with the hardware I have available.

I'm so ready to get started I can't wait for those server dudes to finish their work so my fun can begin.

 

[Robert Trevellyan]

interesting pros to using striped mirrored vdevs

Yes, the article makes a good case for striped mirrors. The big downside is that if you lose both drives from one vdev (assuming 2-way mirrors), you lose the entire pool.

 

[gpsguy]

One could go with 3 way mirrors to minimize the risk. jgreco is doing that on one of his recent builds.


Sent from my phone

 

[Mirfster]

What's the concern about 7200RPM?

Some say they run hotter and/or louder with not much performance increase. But, I still have them. ;)

 

[Yatti420]

Yaaa 7200RPM! They will definitely run hotter..