Possible Multi-Server Configurations?

[JWTech]

I have a few new enterprise servers, and while I have a few ideas as to how I might end up configuring them, I would love to hear some ideas from the community as I value everyone's input. If this is too off-topic for H&S we can move this over to whichever section is most appropriate.

The scenario:
I currently have a consumer-grade FreeNAS server (hereby known as Mark) configured as a "deep-storage" backup for four dedicated servers. Mark is connected via a single port to the server network, and also to my personal vlan with a separate port. Access to various shares are restricted via physical port, and subsequent server IP. Speed is currently bottlenecked by the ports, and the chassis doesn't allow for easy hard drive access, but that has so far been fine for its intended use. Specifications listed below.

I have recently purchased two new Xeon servers that I intend to use for virtualization. Specifications listed below.

The question:
How would these three servers best be configured to allow for greatest possible use of available components? The Xeon servers have ECC RAM, which is obviously preferred for FreeNAS, but if I were to use an entire Xeon server for FreeNAS I would potentially lose a good portion of hardware for virtualization.

Server specifications:
2 x Xeon E3 1225v3

40GB of ECC RAM (total) (2x16GB + 2x4GB)
2 x 4-bay hotswap tray (can have two more added)
2 x GbE ports​

Mark (current FreeNAS server)

16GB non-ECC RAM
Internal HDD access only
2 x GbE ports​

Post your thoughts, let me know how you might configure a similar setup with its various pros and cons, given the provided hardware.

 

[JWTech]

A few configurations I'm considering:

• Keep Mark as it is, used for deep-storage, and setup the two Xeon servers with onboard RAID storage and use both Xeons solely for virtualization.
• Use one of the two Xeon servers as the sole FreeNAS server, with 16GB of ECC RAM, and the other Xeon for virtualization, with 24GB of RAM, and find something else to re-purpose Mark for.
• Setup a smaller FreeNAS on one of the Xeon servers under virtualization (not recommended), and use it for shared storage, while keeping Mark as the backup storage unit, and allowing for more systems to be configured with remaining virtualization space.
• Optionally, though obviously not preferred, purchase a third E3 server solely for FreeNAS, and completely scrap Mark for being consumer-grade.

 

[Robert Trevellyan]

if I were to use an entire Xeon server for FreeNAS I would potentially lose a good portion of hardware for virtualization

Have you considered running VirtualBox on top of FreeNAS? Probably fewer pitfalls than virtualizing FreeNAS.

 

[jgreco]

Have you considered running VirtualBox on top of FreeNAS? Probably fewer pitfalls than virtualizing FreeNAS.

That's a complicated compromise, because it probably means that you will not be using the same virtualization technology on the other hypervisor(s). VirtualBox is pretty good at running a VM or three, but you're also robbing resources from the underlying FreeNAS system.

We advise against the use of non-ECC RAM for reasons well documented elsewhere, though, so my suggestion is this:

Consider using Mark as a hypervisor. Swap in one of the Xeons as your FreeNAS box.

 

[toadman]

Consider using Mark as a hypervisor. Swap in one of the Xeons as your FreeNAS box.

Yes, that. (You want ECC with FreeNAS.)

Or depending on the rest of the Xeon HW specs and your choice of hypervisor, virtualize Freenas on one of the Xeons and use all three servers as hypervisors.

 

[jgreco]

virtualize Freenas on one of the Xeons and use all three servers as hypervisors.

That'd be MY solution (as in "it's actually what we do here") but there's a fair amount of danger in virtualizing if you don't know what you're doing. See the stickies on the topic.

 

[JWTech]

I appreciate all the input so far. I've been weighing the pros and cons in my head for each various scenario.

That'd be MY solution (as in "it's actually what we do here") but there's a fair amount of danger in virtualizing if you don't know what you're doing. See the stickies on the topic.

I quite like the thought of this so far. I'm certainly not new to virtualization, but there's always added risk and performance variations when taking a system like FreeNAS and making it virtual.

One of the larger concerns in that scenario, however, is the increase in overhead with the addition of the hypervisor. The ability to utilize the virtualized FreeNAS for network VM storage is a definite plus, though.

 

[jgreco]

Yeah, the ability to "utilize the virtualized FreeNAS for network VM storage" is the poster child reason not to do it.

1) it creates a circular dependency, causing your hypervisors to hang for NN minutes while they figure out the datastore isn't available while booting

2) Virtualized instances are almost always under-resourced. A VM datastore FreeNAS probably needs 64GB++ RAM to be effective at more than very light duty VM storage.

3) Virtualization pretty much requires a primo VT-d implementation, which is sketchy on a lot of even server grade gear. We know certain things work, but also that lots of other things don't.

Some plonker wrote up a bunch of documentation designed to scare off newbies because there are a lot of caveats. If you can read all the things mentioned in this message and either go "well that makes perfect sense" and/or "well duh of course" to each thing said in the message, then you may actually have some reasonable chance of success (plus I give you my blessing to go ahead and try, FWIW).

Again, see the stickies on the topic. There is substantial risk involved if you choose to do ghetto virtualization of FreeNAS, especially if you're not an experienced virtualization guru.

 

[JWTech]

I'm legitimately considering buying another Xeon just for FreeNAS to make things simpler, and just scrap Mark rather than trying to get the most out of what I have. Though that benefit alone doesn't quite justify the purchase.

it creates a circular dependency

I was thinking more along the lines of utilizing network virtual storage in the form of a virtual backup drive for virtual machines in the network rather than mapping a network location, essentially encapsulating it in a simpler package for the VMs. Not to be used within the local machine, which I would likely designate entirely for various network services.

I'm terrible at explaining my thoughts.

Though I'm curious as to #2. Do you mean that any VM disk utilization will suffer from performance issues under that scenario, or just running guest operating systems stored to FreeNAS? In the situation I mention previously in this post, would you assume similar performance issues?

 

[jgreco]

A ZFS system used for block storage usually gets hit with fragmentation as a significant performance issue. The standard ZFS answer to this is to heavily resource a filer serving block data. "Heavily resource" means a bunch of things, including throwing lots of memory at the problem. ZFS used for block storage on a system without a lot of memory will generally spiral into lower and lower performance as fragmentation increases.

If I had a dollar for every time I've seen an end user show up here with "I want to make an all-in-one box out of a Xeon E3 running ESXi and FreeNAS as a VM," thinking that it'd be okay to run FreeNAS on the "well the manual says it will work on 8GB RAM!" minimal RAM configuration, I could probably go buy a nice E5 hypervisor with the resulting cash.

 

[JWTech]

Duly noted. Looks like network storage will need to remain without virtual encapsulation then.

If I had a dollar for every time I've seen an end user show up here with "I want to make an all-in-one box out of a Xeon E3 running ESXi and FreeNAS as a VM," thinking that it'd be okay to run FreeNAS on the "well the manual says it will work on 8GB RAM!" minimal RAM configuration, I could probably go buy a nice E5 hypervisor with the resulting cash.

Well you won't be having a dollar from me.

 

[JWTech]

My largest concern with FreeNAS entirely consuming a Xeon to be used for "deep storage" backups is that the processor is subsequently underutilized. Though given the various performance concerns and additional overhead of virtualizing the server to allow for additional services, I'm going to take an alternate approach and ask the question, how can we get more use out of FreeNAS itself?

My immediate thought is to add a few jails and slapping in owncloud and plex for the personal network, either via plugin or manually configured (likely plugin due to ease of use).

 

[jgreco]

The jail route is a fairly well-understood quantity and works in close alignment with the design intent of the system.

Your FreeNAS is unlikely to consume all of a Xeon all by itself unless you do some really unusual things. Even with 10GE on the dedicated VM filer here, I don't see the CPU (E5-1650v3) hit more than ~10% utilization. I could probably do more damage if it was all SSD storage but I'm pretty confident that I could downsize it to an E5-1620v3 without any impact. If you plan to add jails/plugins, do remember that your host system is doing all the functions of a RAID controller in software, and therefore you might want to consider "two cores as burned" or something like that when figuring on how big to go.

 

[JWTech]

If there aren't many performance concerns, I could reduce VM resource requirements on the other servers by designating a network MariaDB/MySQL server in a jail on the FreeNAS server. This would help to better utilize the CPU, but would also reduce the available RAM with increased usage.

 

[jgreco]

Tradeoffs, tradeoffs...! The one good part is that you can potentially change your mind at a future date if it doesn't work out. It's super hard to make predictions about how "X" will work when X is as poorly defined as "a MySQL server" - sorry, that may sound harsh but it's true - but my general opinion is that if you can avoid running stressy things on a VM using a ZFS block datastore (even if the alternative is running them directly on the filer using database files) that's usually a better choice. It may also mean you just need to add RAM to your filer to fix performance/memory starvation issues, rather than trying to sleuth out multiple tiers of issues that arise when you do it from within a VM.

 

[JWTech]

sorry, that may sound harsh

Pointlessly in fact, considering the idea is currently in the form of a concept and cannot be further defined at the moment.

Regardless, I'm feeling as though this thread may have reached its conclusion for the moment. I'll make another post later to add final information on how the servers ended up being configured based on provided input and overall evaluated performance.

 

[Pheran]

A few configurations I'm considering:

• Keep Mark as it is, used for deep-storage, and setup the two Xeon servers with onboard RAID storage and use both Xeons solely for virtualization.

You haven't given us very specific info about these servers, and the above bit raises some concerns. I'm not sure what controllers you have on these servers but FreeNAS isn't meant to run on top of a hardware RAID controller. It's possible that you meant you would only do that in the case of using them solely for virtualization, but it's kind of ambiguous.

 

[JWTech]

You haven't given us very specific info about these servers, and the above bit raises some concerns. I'm not sure what controllers you have on these servers but FreeNAS isn't meant to run on top of a hardware RAID controller. It's possible that you meant you would only do that in the case of using them solely for virtualization, but it's kind of ambiguous.

The latter, though detail was not provided to remain relatively on-topic, as that scenario didn't really include doing much of anything with FreeNAS on the Xeons.

After considering various possible scenarios, with the inclusion of the input discussed throughout this topic, I have elected to purchase a third and fourth identical Xeon server to act as an Active/Backup storage solution. The servers being used in this case are TS440 systems, which are currently at an appropriate price point to justify a proper configuration. Mark will likely be re-purposed for virtualization.

I appreciate all the helpful advice and ideas.