TrueNAS scale - ESXI datastore

[bloodyskullz]

Hi all,

Been thinking about this a bit for the office production servers. Current setup is as follows:

2 standalone ESXi servers with the VMs split across the two. Current usage is about 14TB across the two.

What I was thinking about doing:

Take two legacy Dell servers we have lying around and throw a bunch of disks in, install TrueNAS Scale and use it as a shared datastore among the two esxi servers and enable HA between the two NAS scale units to eliminate a single point of failure. Doing so would allow us to utilize the essentials plus license we want to get from our vendor and enable HA plus utilize other features like vMotion.

Does TrueNAS Scale work with such a plan in mind? Can't do TrueNAS core because it doesn't support HA.

Should I look at other ideas instead?

Thanks

 

[jgreco]

What makes you think that SCALE supports HA?

 

[Ericloewe]

What makes you think that SCALE supports HA?

There's a marketing slide from iX with various options and it's not as clear as it sounded to whoever made it (not blaming them, it's clear to me, but we've had multiple users show up with the same question).

Hi all,

Been thinking about this a bit for the office production servers. Current setup is as follows:

2 standalone ESXi servers with the VMs split across the two. Current usage is about 14TB across the two.

What I was thinking about doing:

Take two legacy Dell servers we have lying around and throw a bunch of disks in, install TrueNAS Scale and use it as a shared datastore among the two esxi servers and enable HA between the two NAS scale units to eliminate a single point of failure. Doing so would allow us to utilize the essentials plus license we want to get from our vendor and enable HA plus utilize other features like vMotion.

Does TrueNAS Scale work with such a plan in mind? Can't do TrueNAS core because it doesn't support HA.

Should I look at other ideas instead?

Thanks

Simple answer, you do not. You could try something along these lines with Gluster and three servers (the absolute minimum for a minimally-reliable Gluster setup), but it would be slow.

HA requires buying from iX because of the tight integration requirements of the HA architecture (two hosts sharing a single pool) and it doesn't really address many failure scenarios. Rather, it focuses on maintaining uptime of the host (which makes some sense, as a sensible design of the ZFS pool(s) is a given in this scenario) in scenarios like:

• Host hardware failureds
• OS upgrades

 

[bloodyskullz]

What makes you think that SCALE supports HA?

Compare TrueNAS Editions - Powerful Storage Platform

TrueNAS is the ultimate Open Source storage platform. Choose from CORE, Enterprise, or SCALE to get the best performance and reliability.

www.truenas.com

It says HA is available in Scale. Not sure why this wouldn't be available if they clearly documented it there.

 

[Ericloewe]

Compare TrueNAS Editions - Powerful Storage Platform

TrueNAS is the ultimate Open Source storage platform. Choose from CORE, Enterprise, or SCALE to get the best performance and reliability.

www.truenas.com

It says HA is available in Scale. Not sure why this wouldn't be available if they clearly documented it there.

See above. Think of it as "TrueNAS Enterprise, but on Linux". HA is not available in general.

 

[bloodyskullz]

There's a marketing slide from iX with various options and it's not as clear as it sounded to whoever made it (not blaming them, it's clear to me, but we've had multiple users show up with the same question).


Simple answer, you do not. You could try something along these lines with Gluster and three servers (the absolute minimum for a minimally-reliable Gluster setup), but it would be slow.

HA requires buying from iX because of the tight integration requirements of the HA architecture (two hosts sharing a single pool) and it doesn't really address many failure scenarios. Rather, it focuses on maintaining uptime of the host (which makes some sense, as a sensible design of the ZFS pool(s) is a given in this scenario) in scenarios like:

• Host hardware failureds
• OS upgrades

Sorry what do you mean "you do not"? As in this plan is not a good idea or it just flat out won't work because this isn't the intended use for TrueNAS Scale?

We don't have many options here and this is probably the only scenario I can come up with that could make the most sense operationally. We would have another server dedicated to backing up the VM's.

 

[Ericloewe]

Sorry what do you mean "you do not"?

I mean that it is not possible within the set of constraints you defined. As I said, the only option that comes close would be Gluster, but it needs a third server and performance is a problem.

 

[jgreco]

It says HA is available in Scale. Not sure why this wouldn't be available if they clearly documented it there.

HA is limited to gear sold by iXsystems in part because it requires a particular design. It is not a "you get to pick your own architecture" sort of thing. Someone screwed up on the slide, or maybe misunderstood that there MIGHT be a SCALE version capable of doing HA on iXsystems-provided platforms; I would not be shocked if you could run SCALE in HA mode on a TrueNAS Enterprise hardware platform.

"Clearly documented" is a bit of a snicker. There's a bunch of conflicting documentation. My favorite is that they list the minimum memory for Core as 16GB on the download page but 8GB elsewhere. I actually suspect that they meant to bump Scale to 16GB (because that would give you 8GB of ARC) because having Scale with an 8GB minimum memory requirement is really starving the system harshly.

 

[bloodyskullz]

Few questions at this point then:

1) Is the minimum requirements for GlusterFS 3 servers or is your experience stating 3 servers would be best?
2) Would TrueNAS still be best used having the ZFS pool and GlusterFS on top or is it a filesystem on it's own that doesn't require ZFS at all?
3) What hardware requirements are needed to avoid performance issues that you speak of? Our servers have 10GBe cards so hopefully that helps.

We really don't have a bunch of additional servers we can use for such a task so trying to make this work as best as possible. I am open to ideas that can accomplish what I am looking for in the most simplistic manner

 

[HoneyBadger]

There's an updated article regarding clustering in SCALE:

TrueNAS SCALE Clustering Overview

Discover the power of TrueNAS SCALE Clustering, enabling scalable and reliable unified storage solutions for modern data environments.

www.truenas.com

But there is a difference in latency and performance between a scale-out cluster, and scale-up storage. Since virtual machines tend to thrive on low-latency, for the purpose of a highly-available ESXi datastore, the solution I'd recommend is a TrueNAS Enterprise setup.

But you could certainly try out TrueNAS CORE on a single Dell server (provided that it meets the hardware requirements) to get a feel for the UI, and become familiar with the process of assigning shared storage resources; and we'd be more than happy to get you started with that.

 

[bloodyskullz]

I have done the setup before with FreeNAS at my old employer and the VM's run fine without issues. I have setup the datastore and hooked it up into ESXi without issues. I know this works without issues but again it would be a single point of failure which we are trying to avoid.

If the scale only works with iXsystems then its a useless solution for me unless glusterfs can allow me to achieve the same outcome but we only have two servers that could be used for this.

 

[jgreco]

No, Scale works on whatever you would like. However, Scale does not do HA. Further, Gluster probably doesn't do what you want either.

 

[Ericloewe]

1) Is the minimum requirements for GlusterFS 3 servers or is your experience stating 3 servers would be best?

A bit of both. 2 servers will work, but only until the slightest network interruption - then you get to deal with a split-brain situation. Arbiters are also more of a joke than a real solution, in part because they only work as 2+1 and in no other configuration.
3-way mirrors are the minimum viable configuration.

2) Would TrueNAS still be best used having the ZFS pool and GlusterFS on top or is it a filesystem on it's own that doesn't require ZFS at all?

Despite the FS in the name, it's more of a shim on top of standard POSIX filesystems. So ZFS is still in play.

3) What hardware requirements are needed to avoid performance issues that you speak of? Our servers have 10GBe cards so hopefully that helps.

The main driver, according to my current understanding, is latency. That includes disk and network. I've seen performance as catastrophic as listing files at 1 per second, albeit with 2.5" SMR HDDs (with dedicated 10GBase-T links).
Bulk transfers should be fine (and we're fine for me, even in the above nightmare scenario), but any IOPS-sensitive workload is likely to work poorly.
If you absolutely want to try this for VM storage, do not skimp on the disks, do not skimp on memory (for ARC, to reduce read latency) and do not skimp on networking.

 

[Ericloewe]

Scale does not do HA

Asterisk: the version of Scale you can install on arbitrary hardware does not do HA.

 

[bloodyskullz]

In summary, I am screwed and I need to find alternatives because we aren't looking to spend more money as we are trying to get other things in the office that have been prolonged.

TrueNAS Enterprise isn't an option.

 

[Patrick M. Hausen]

You can do "poor man's HA" and accept that you will have a single point of failure for storage. Then snapshot and replicate all of your storage at least nightly to the second identical system.

That means return to operation is a matter of an hour or two of manual intervention.

A certain time frame of data loss is of course to be expected, too. But that is the case with any backup solution. Local ESXi datastore probably even worse in case the host crashes.

In addition to ZFS replication - which will introduce lost open transactions for live virtual disk images - you could additionally use e.g. GhettoVCB to get a daily guaranteed consistent archive copy of each VM at night.


We as a small (~35 people) software development company have come to terms with this form of "HA" and it served us amazingly well and saved us a lot of money. Orders of magnitude better than no redundancy and disaster recovery plan at all.

 

[bloodyskullz]

The only other thing I can think of at this point that could also work is ZFS replication between the two TrueNAS units which I believe Patrick is referencing in his post.

Question is, at this point does it make sense to go with Scale or stick with core for such a task?

 

[Patrick M. Hausen]

As a datastore for ESXi I would recommend CORE for performance and stability reasons.

Yes, I implied ZFS snapshots and replication.

 

[jgreco]

does it make sense to go with Scale or stick with core for such a task?

Does it make sense to go with the thing that is crappy for your use case, or "stick with" core for the thing it does swimmingly well?

Explain to me: "stick with" in this context. Are you just so anti-FreeBSD or desperate for a subpar solution that you need to imply that Core is a crummy solution? It's been explained to you in unambiguous terms that Core is the solution you're looking for. If you are just looking for a blessing to run Scale regardless, we can certainly provide that, but it'll be (at least) a little white lie.