Multiple servers for VMWare primary and secondary datastore

Status
Not open for further replies.
Joined
Mar 2, 2015
Messages
22
Hello all.

If you allow me, here is a bit of my background and a not-so-short description of my goal with FreeNAS.

About 2o years ago I decided that learning some variation of Unix would be a good idea, and after some research, over a 9600 BPS modem, I found out about FreeBSD, then version 2.0.x. Excited, I ordered my first set of CDs, had them apprehended by Brazilian customs, had to pay import taxes (which, for goods like books, music CDS, etc, is illegal in Brazil, but, go figure), etc. Beautiful start.

Since then I've been building simple servers, FTP, file, print, and other not-so-simple, FreeBSD and Linux based. Along with that, an extensive experience with Windows-server products, and a good deal of IT infrastructure work.

Due to some professional reasons I had to start using CentOS for a while, working with Gluster and a few other applications, but a twist in the plot brought me back to FreeBSD and finally FreeNAS.

Enough about me, let's talk about the "problem".

I manage about 20 virtualized server spread between two (or among three) physical servers (R710s, diskless, 96 GB RAM). Two years ago I was going through the ordeal of having local storage on those servers (actually back then they were about 10 physical servers) and never having enough disk space on any of them.

All of a sudden I became the proud (???) admin of a 3-node Isilon 6000x, with its 15 TB of disk space, which allowed me to break free from the local disk prison. With time the Isilon became a liability, and due to budget constrains, I had NO BACKUP or contingency plan whatsoever. (please, no comments on this. I inherited that situation).

The solution ? A couple of decommissioned Dell PE 2900 populated with all of the disks from the physical servers, multiple disk controllers, running CentOS and REGULAR (NFS for simplicity) storage. Once a week I would suspend all VMs (custom script) and copy each VM over, from the Isilon to the 2900s. Good performance on those 2900s, soon i was tempted to invert their roles: 2900s as the main production datastore(s) and the Isilon as the LIVE backup.

If _it_ happens, I can afford the luxury of "mapping" the VMs from the Isilon and resume production, which would take literally a couple of hours. For our case, this is perfectly acceptable.

Now the Isilon is giving in, and my adventures with ZoL (ZFS on Linux) were very pleasant, but I needed more. Needless to say, this is the reason I am here.

Right now I was able to allocate two Dell T610s, each with 22 GB of RAM, and I plan on moving the storage from the 2900s with CentOS to a much more manageable FreeNAS structure. Even importing the ZFS volumes (already tested) works like a charm, BUT my next steps would require your input.

Despite the fact I've spent a considerable amount of hours reading manuals, blogs, posts, etc, I still cannot get my head around a couple of concepts, or, more to the point, how to apply them to my project.

So, in short, here is the idea:

- FreeNAS (9.3) working as a NFS datastore for two or three ESXi servers (vCenter SMB, if it is important).
(btw, yes, I've been though a lot of threads, and the manual, and other lists, about NFS and ESXi and iSCSI issues/performance.)
- Second FreeNAS server that will host a COPY of all VM images from the official FreeNAS number 1. Like in the Isilon before it, in case something happens, I'll just start using the images hosted on the second server.

Both of those server will be (well already are) Dell T610, 22 GB RAM (ECC, yes sir). The main box will have 8 or 10 450 GB @ 15K RPM SAS disks (already in production on my current Linux box), and on a separate controller ONE SSD disk for LARC2. I am currently using it for SLOG, but according to my readings, it is almost not used, so, on the new configuration I'll use it as a LARC2 device.

(Yep, I am aware of the risks of not using a mirrored device for SLOG).

I am using a LAGG with both NICs for the datastore, a third NIC for management only (well, and to isolate NFS traffic) and will have a 10 GB fiber NIC soon on those servers.

Hardware-wise, PERCs and disks are configured to handle single volumes, and on the FreeNAS side disks are configured as RAIDZ-2. SLOG on the main server only.

So, "what is the problem ?", you ask. My problem is that copying all of the VM images between the two servers takes 6 hours. I am trying to come up with a smarter way of copying filesystems across servers, in a way I will end up with TWO LIVE datastores, one on each server.

For some, or maybe most of you, this may sound trivial, and I am pretty sure the answer is "SNAPSHOT", but so far, I was not able to devise a way to make the initial copy, and then go on exporting subsequent snapshots from the main server to the second, in a way the second server will always have a live filesystem.

Quite frankly, I admit that I do not fully master the concept of snapshot, but on the other hand, I was not able to find an implementation for this solution I proposed here. I often see snapshots or copies from one volume to another on the same box, and even another box being the repository of many snapshots, but never another live filesystem.

Rsync is out of the picture, unless if used over NFS mounts, which is what I already do nowadays, but my true goal is taking full advantage of ZFS' snapshot (not VMWare's, for some other reasons).

I don't really need compression over SSH, or even cryptography whatsoever, since I have an isolated network for the datastores.

The initial steps of taking a snapshot of the main system and send it to the second are clear (well, the GIU makes it look simple, and the CLI scripts I've seen are not too bad either), but integrating the snapshot to the file system of the second server is something that I still don't understand, and, as I said, could not find so far.

I did not elaborate too much on the hardware or software configuration because I didn't think it would be that important for this initial discussion, but if needed, I can go into details.

Thanks for your time.

Carlos
 
Last edited:
Joined
Mar 2, 2015
Messages
22
No, not at all. I am going to start experimenting with snapshots, just because this seems to be the way to go, but my fear is that, according to some documents, snapshots are file-related, more or less like Apple's Time Machine.

If this is true, this is a major no-go. The amount of disk space needed would be ridiculous.

Should that be the case, exploring VMWare's snapshot and then managing from FreeNAS will be the option.

But this is all theoretical for now. I am still stuck with my six-hour rsync over NFS.

Thanks for the interest.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
- FreeNAS (9.3) working as a NFS datastore for two or three ESXi servers (vCenter SMB, if it is important).
(btw, yes, I've been though a lot of threads, and the manual, and other lists, about NFS and ESXi and iSCSI issues/performance.)

I wouldn't sweat it. People like iSCSI for the performance, but a lot of experienced ESXi administrators strongly prefer an NFS environment because it allows you to manipulate things outside the vmfs3/vmfs5 environment. It's a matter of throwing the right stuff at the design.

- Second FreeNAS server that will host a COPY of all VM images from the official FreeNAS number 1. Like in the Isilon before it, in case something happens, I'll just start using the images hosted on the second server.

Both of those server will be (well already are) Dell T610, 22 GB RAM (ECC, yes sir). The main box will have 8 or 10 450 GB @ 15K RPM SAS disks (already in production on my current Linux box), and on a separate controller ONE SSD disk for LARC2. I am currently using it for SLOG, but according to my readings, it is almost not used, so, on the new configuration I'll use it as a LARC2 device.

That might be too small a system for L2ARC to be beneficial.

(Yep, I am aware of the risks of not using a mirrored device for SLOG).

I'm not aware of any substantial risk. The likelihood that a SSD will fail at almost the same time as an unexpected panic or power loss event is vanishingly small.

I am using a LAGG with both NICs for the datastore, a third NIC for management only (well, and to isolate NFS traffic) and will have a 10 GB fiber NIC soon on those servers.

Hardware-wise, PERCs and disks are configured to handle single volumes, and on the FreeNAS side disks are configured as RAIDZ-2. SLOG on the main server only.

So, "what is the problem ?", you ask. My problem is that copying all of the VM images between the two servers takes 6 hours. I am trying to come up with a smarter way of copying filesystems across servers, in a way I will end up with TWO LIVE datastores, one on each server.

For some, or maybe most of you, this may sound trivial, and I am pretty sure the answer is "SNAPSHOT", but so far, I was not able to devise a way to make the initial copy, and then go on exporting subsequent snapshots from the main server to the second, in a way the second server will always have a live filesystem.

Snapshots are always snapshots; by definition, not "live". You are taking a picture of your ZFS pool at a given point in time and then (possibly) replicating it to some other machine. The replication is a process that takes additional time.

It sounds to me like what you're hoping for is some form of realtime replication...? If so, FreeNAS doesn't do that.

FreeBSD does have the ability to do HAST. This seems a lot closer to what you want. You can set up two machines, with a virtual storage device that is stored on both nodes, and you can NFS export from one of the nodes to act as a NFS server. If that fails, you can reconfigure the other node to serve up NFS and take over. You still can't (or at least shouldn't) have both them serving NFS simultaneously because FFS is not a cluster-aware filesystem and you cannot have two servers both mounting the HAST drive without causing corruption and despair.

You could find a cluster-aware filesystem to cram on top of HAST, or it may make more sense to export the HAST drive as an iSCSI device, which then lets VMware's VMFS take care of a lot of that. The downside there might be that you could run into substantial problems trying to import a disk that looks like it is a multipath view of an existing datastore... this could work or it might not, I don't know.

Otherwise, with ZFS and snapshots, the best you can do is to take frequent snapshots and propagate them to the remote host.
 
Joined
Mar 2, 2015
Messages
22
"It sounds to me like what you're hoping for is some form of realtime replication...?"

No, not really. I can afford the luxury of pausing the servers for sometime (e.g. Saturday midnight) and running my backup. Also means that, on my case, if I have to revert to the backup, it will be from a week ago and that should be acceptable (well, you get what you pay for, so no one can complain. Trust me, they did not even have a backup before...)

So, back to the point, what would be good sense here ?

Snapshot -> export the snapshot to the other system -> incorporate the exported snapshot on the second system ? ? Sounds counter-intuitive if the snapshot is read-only.

All of a sudden snapshots no longer look like a plausible option.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Well, you can certainly clone a snapshot to make a writeable copy of the snapshotted filesystem on the second system, but there's no structure to do this automatically within FreeNAS.
 

RegularJoe

Patron
Joined
Aug 19, 2013
Messages
330
this is an old thread I know. Did you get your system migrated and figured out? What did you do? If you have the cheapest version of VMware with vCenter the new snapshots can be coordinated between VMware and FreeNAS so the virtual machines with OpenVMTools know they were backed up. No pausing virtual machines. FreeNAS and VMware work great together. If your running Linux or FreeBSD machines they seem to work really well on NFS/ZFS.
 
Status
Not open for further replies.
Top