Carlos Capriotti
Dabbler
- Joined
- Mar 2, 2015
- Messages
- 22
Hello all.
If you allow me, here is a bit of my background and a not-so-short description of my goal with FreeNAS.
About 2o years ago I decided that learning some variation of Unix would be a good idea, and after some research, over a 9600 BPS modem, I found out about FreeBSD, then version 2.0.x. Excited, I ordered my first set of CDs, had them apprehended by Brazilian customs, had to pay import taxes (which, for goods like books, music CDS, etc, is illegal in Brazil, but, go figure), etc. Beautiful start.
Since then I've been building simple servers, FTP, file, print, and other not-so-simple, FreeBSD and Linux based. Along with that, an extensive experience with Windows-server products, and a good deal of IT infrastructure work.
Due to some professional reasons I had to start using CentOS for a while, working with Gluster and a few other applications, but a twist in the plot brought me back to FreeBSD and finally FreeNAS.
Enough about me, let's talk about the "problem".
I manage about 20 virtualized server spread between two (or among three) physical servers (R710s, diskless, 96 GB RAM). Two years ago I was going through the ordeal of having local storage on those servers (actually back then they were about 10 physical servers) and never having enough disk space on any of them.
All of a sudden I became the proud (???) admin of a 3-node Isilon 6000x, with its 15 TB of disk space, which allowed me to break free from the local disk prison. With time the Isilon became a liability, and due to budget constrains, I had NO BACKUP or contingency plan whatsoever. (please, no comments on this. I inherited that situation).
The solution ? A couple of decommissioned Dell PE 2900 populated with all of the disks from the physical servers, multiple disk controllers, running CentOS and REGULAR (NFS for simplicity) storage. Once a week I would suspend all VMs (custom script) and copy each VM over, from the Isilon to the 2900s. Good performance on those 2900s, soon i was tempted to invert their roles: 2900s as the main production datastore(s) and the Isilon as the LIVE backup.
If _it_ happens, I can afford the luxury of "mapping" the VMs from the Isilon and resume production, which would take literally a couple of hours. For our case, this is perfectly acceptable.
Now the Isilon is giving in, and my adventures with ZoL (ZFS on Linux) were very pleasant, but I needed more. Needless to say, this is the reason I am here.
Right now I was able to allocate two Dell T610s, each with 22 GB of RAM, and I plan on moving the storage from the 2900s with CentOS to a much more manageable FreeNAS structure. Even importing the ZFS volumes (already tested) works like a charm, BUT my next steps would require your input.
Despite the fact I've spent a considerable amount of hours reading manuals, blogs, posts, etc, I still cannot get my head around a couple of concepts, or, more to the point, how to apply them to my project.
So, in short, here is the idea:
- FreeNAS (9.3) working as a NFS datastore for two or three ESXi servers (vCenter SMB, if it is important).
(btw, yes, I've been though a lot of threads, and the manual, and other lists, about NFS and ESXi and iSCSI issues/performance.)
- Second FreeNAS server that will host a COPY of all VM images from the official FreeNAS number 1. Like in the Isilon before it, in case something happens, I'll just start using the images hosted on the second server.
Both of those server will be (well already are) Dell T610, 22 GB RAM (ECC, yes sir). The main box will have 8 or 10 450 GB @ 15K RPM SAS disks (already in production on my current Linux box), and on a separate controller ONE SSD disk for LARC2. I am currently using it for SLOG, but according to my readings, it is almost not used, so, on the new configuration I'll use it as a LARC2 device.
(Yep, I am aware of the risks of not using a mirrored device for SLOG).
I am using a LAGG with both NICs for the datastore, a third NIC for management only (well, and to isolate NFS traffic) and will have a 10 GB fiber NIC soon on those servers.
Hardware-wise, PERCs and disks are configured to handle single volumes, and on the FreeNAS side disks are configured as RAIDZ-2. SLOG on the main server only.
So, "what is the problem ?", you ask. My problem is that copying all of the VM images between the two servers takes 6 hours. I am trying to come up with a smarter way of copying filesystems across servers, in a way I will end up with TWO LIVE datastores, one on each server.
For some, or maybe most of you, this may sound trivial, and I am pretty sure the answer is "SNAPSHOT", but so far, I was not able to devise a way to make the initial copy, and then go on exporting subsequent snapshots from the main server to the second, in a way the second server will always have a live filesystem.
Quite frankly, I admit that I do not fully master the concept of snapshot, but on the other hand, I was not able to find an implementation for this solution I proposed here. I often see snapshots or copies from one volume to another on the same box, and even another box being the repository of many snapshots, but never another live filesystem.
Rsync is out of the picture, unless if used over NFS mounts, which is what I already do nowadays, but my true goal is taking full advantage of ZFS' snapshot (not VMWare's, for some other reasons).
I don't really need compression over SSH, or even cryptography whatsoever, since I have an isolated network for the datastores.
The initial steps of taking a snapshot of the main system and send it to the second are clear (well, the GIU makes it look simple, and the CLI scripts I've seen are not too bad either), but integrating the snapshot to the file system of the second server is something that I still don't understand, and, as I said, could not find so far.
I did not elaborate too much on the hardware or software configuration because I didn't think it would be that important for this initial discussion, but if needed, I can go into details.
Thanks for your time.
Carlos
If you allow me, here is a bit of my background and a not-so-short description of my goal with FreeNAS.
About 2o years ago I decided that learning some variation of Unix would be a good idea, and after some research, over a 9600 BPS modem, I found out about FreeBSD, then version 2.0.x. Excited, I ordered my first set of CDs, had them apprehended by Brazilian customs, had to pay import taxes (which, for goods like books, music CDS, etc, is illegal in Brazil, but, go figure), etc. Beautiful start.
Since then I've been building simple servers, FTP, file, print, and other not-so-simple, FreeBSD and Linux based. Along with that, an extensive experience with Windows-server products, and a good deal of IT infrastructure work.
Due to some professional reasons I had to start using CentOS for a while, working with Gluster and a few other applications, but a twist in the plot brought me back to FreeBSD and finally FreeNAS.
Enough about me, let's talk about the "problem".
I manage about 20 virtualized server spread between two (or among three) physical servers (R710s, diskless, 96 GB RAM). Two years ago I was going through the ordeal of having local storage on those servers (actually back then they were about 10 physical servers) and never having enough disk space on any of them.
All of a sudden I became the proud (???) admin of a 3-node Isilon 6000x, with its 15 TB of disk space, which allowed me to break free from the local disk prison. With time the Isilon became a liability, and due to budget constrains, I had NO BACKUP or contingency plan whatsoever. (please, no comments on this. I inherited that situation).
The solution ? A couple of decommissioned Dell PE 2900 populated with all of the disks from the physical servers, multiple disk controllers, running CentOS and REGULAR (NFS for simplicity) storage. Once a week I would suspend all VMs (custom script) and copy each VM over, from the Isilon to the 2900s. Good performance on those 2900s, soon i was tempted to invert their roles: 2900s as the main production datastore(s) and the Isilon as the LIVE backup.
If _it_ happens, I can afford the luxury of "mapping" the VMs from the Isilon and resume production, which would take literally a couple of hours. For our case, this is perfectly acceptable.
Now the Isilon is giving in, and my adventures with ZoL (ZFS on Linux) were very pleasant, but I needed more. Needless to say, this is the reason I am here.
Right now I was able to allocate two Dell T610s, each with 22 GB of RAM, and I plan on moving the storage from the 2900s with CentOS to a much more manageable FreeNAS structure. Even importing the ZFS volumes (already tested) works like a charm, BUT my next steps would require your input.
Despite the fact I've spent a considerable amount of hours reading manuals, blogs, posts, etc, I still cannot get my head around a couple of concepts, or, more to the point, how to apply them to my project.
So, in short, here is the idea:
- FreeNAS (9.3) working as a NFS datastore for two or three ESXi servers (vCenter SMB, if it is important).
(btw, yes, I've been though a lot of threads, and the manual, and other lists, about NFS and ESXi and iSCSI issues/performance.)
- Second FreeNAS server that will host a COPY of all VM images from the official FreeNAS number 1. Like in the Isilon before it, in case something happens, I'll just start using the images hosted on the second server.
Both of those server will be (well already are) Dell T610, 22 GB RAM (ECC, yes sir). The main box will have 8 or 10 450 GB @ 15K RPM SAS disks (already in production on my current Linux box), and on a separate controller ONE SSD disk for LARC2. I am currently using it for SLOG, but according to my readings, it is almost not used, so, on the new configuration I'll use it as a LARC2 device.
(Yep, I am aware of the risks of not using a mirrored device for SLOG).
I am using a LAGG with both NICs for the datastore, a third NIC for management only (well, and to isolate NFS traffic) and will have a 10 GB fiber NIC soon on those servers.
Hardware-wise, PERCs and disks are configured to handle single volumes, and on the FreeNAS side disks are configured as RAIDZ-2. SLOG on the main server only.
So, "what is the problem ?", you ask. My problem is that copying all of the VM images between the two servers takes 6 hours. I am trying to come up with a smarter way of copying filesystems across servers, in a way I will end up with TWO LIVE datastores, one on each server.
For some, or maybe most of you, this may sound trivial, and I am pretty sure the answer is "SNAPSHOT", but so far, I was not able to devise a way to make the initial copy, and then go on exporting subsequent snapshots from the main server to the second, in a way the second server will always have a live filesystem.
Quite frankly, I admit that I do not fully master the concept of snapshot, but on the other hand, I was not able to find an implementation for this solution I proposed here. I often see snapshots or copies from one volume to another on the same box, and even another box being the repository of many snapshots, but never another live filesystem.
Rsync is out of the picture, unless if used over NFS mounts, which is what I already do nowadays, but my true goal is taking full advantage of ZFS' snapshot (not VMWare's, for some other reasons).
I don't really need compression over SSH, or even cryptography whatsoever, since I have an isolated network for the datastores.
The initial steps of taking a snapshot of the main system and send it to the second are clear (well, the GIU makes it look simple, and the CLI scripts I've seen are not too bad either), but integrating the snapshot to the file system of the second server is something that I still don't understand, and, as I said, could not find so far.
I did not elaborate too much on the hardware or software configuration because I didn't think it would be that important for this initial discussion, but if needed, I can go into details.
Thanks for your time.
Carlos
Last edited: