Hot wo Backup Ubuntu Server not running ZFS using FreeNAS?

[consultant]

#1
I have an Ubuntu Server being used as a file server (SAMBA) and I want to use FreeNAS as a backup solution. The storage device would be an external RAID Array. Since I'm not running ZFS file system on Ubuntu, what would be the best system architecture for using FreeNAS as a backup. Would I basically installed FreeNAS on the NAS RAID box and then NFS mount the Ubuntu file folder and "pull" the files from the Ubuntu server?

I was thinking it might be better to attached the storage via USB 3.1 so the backup doesn't run over the network, but that would mean trying to treat the Ubuntu file server as a storage device and my knowledge of Linux isn't deep enough to determine if that's even possible. As long as the LAN is fast and not congested, considering I need to give access to the backup file to users via CIFS, I suppose it's okay to do the backup data transfer over the network.

If anyone could confirm the recommended architecture I'd be grateful.

 

[Chris Moore]

I would need a better understanding of this situation because some of what you are saying just doesn't make sense to me.

I have an Ubuntu Server being used as a file server (SAMBA) and I want to use FreeNAS as a backup solution.

No problem so far.

The storage device would be an external RAID Array.

What is this? Exactly. Also, how does it attach currently and what does it attach to? Is it hardware RAID or some type of SAS JBOD that is connected to the Ubuntu Server with the disks controlled by mdadm or something else. Details matter.

Since I'm not running ZFS file system on Ubuntu, what would be the best system architecture for using FreeNAS as a backup.

You could setup a simple rsync between the Ubuntu Server and the FreeNAS server. It wouldn't be super fast, but you didn't say how much data you are dealing with or what type of network you have.

Would I basically installed FreeNAS on the NAS RAID box and then NFS mount the Ubuntu file folder and "pull" the files from the Ubuntu server.

This whole thing is just broken. How would you, "install FreeNAS on the NAS RAID box"? What are you calling a NAS RAID box? Do you have a server that is running FreeNAS and a separate server that is running Ubuntu and a separate "RAID box" of some kind? You can't NFS mount a directory in FreeNAS, but you could NFS mount a share from FreeNAS in the Ubuntu server. That might be a way to initially copy the data. To keep the backup up to date, you would want to setup a cron task to run on the Ubuntu server to send data to the FreeNAS server on a schedule. You could setup a rsync on the FreeNAS side that would pull data from the Ubuntu system. The rsync software is quite flexible and was designed for this purpose.

Or is the only way for it to work property to convert the Ubuntu server to ZFS and install FreeNAS on the Ubuntu server and designate the external attached NAS as the backup target?

Again, this makes no sense. Ultimately, you might want to change the Ubuntu server to be a FreeNAS server, I suppose, but you keep talking about an external attached NAS. What is that? Is it some hardware appliance that connects to the network directly or is it a disk array connected to the Ubuntu server? If there is some other brand of NAS on the network, it may be that you should be looking at what it is able to do and the Ubuntu server could be right out of the conversation.

I was thinking it might be better to attached the storage via USB 3.1 so the backup doesn't run over the network

Again, is this some kind of hardware appliance that has a USB port on it? What are you talking about, because you can't do that with any computer I am familiar with.

As long as the LAN is fast and not congested

How fast is it?

I need to give access to the backup file to users via CIFS

Then why are you talking about NFS?

If anyone could confirm the recommended architecture I'd be grateful.

You need to answer some questions before we can even know what you are looking at.

 

[danb35]

I was thinking it might be better to attached the storage via USB 3.1

No, it wouldn't. FreeNAS doesn't work that way.

What do you understand FreeNAS as being, and how do you see it working in this scenario?

 

[consultant]

Let's start over.

The data files to be backed up are on an Ubuntu server NOT running ZFS. This cannot change.

I need to store the backup of the files on a storage device other than the Ubuntu server of course. The Ubuntu server hard drive slots are already filled with existing drives for the servers own RAID array.

I need to take hourly snapshots to grab any files changes that have happened in the last hour and I need to retain all the versions for the last 30 days.

I need an end user to be able to copy a version of the backup file(s) to their local Windows workstation using Windows Explorer.

My initially research turned up FreeNAS as being a potential solution due to the CIFS capability with the Windows File Explorer 'restore previous versions' functionality. Everything else requires using the backup software's interface to restore the files, not an option for my goals, or, write my own scripting using rsync and --append-suffix flag to version the filenames (a pretty crude solution this would be.) The end users need to restore files WITHOUT running Linux command line commands or running special software on their workstation (a simple end-user [not sys admin] web interface to browse and download files would be an option.) Hence the requirement to restore via Windows File Explorer and a mapped network drive to the backup location seems the only feasible solution given the requirements.

Is FreeNAS not capable of meeting these requirements in some sort of configuration? Are the source files required to be stored on ZFS? If that's the case, then that would be the hangup here and I made a false assumption that only the TARGET files system had to be ZFS and the source files could be from any source the server can read?

The thing that always hangs people up on making an appropriate recommendation for this situation is everyone operates from the mindset that the Sys Admin is responsible for restoring files and therefore the user restoring the files can use the special software or run Linux commands to restore the files. That's NOT the case here.

 

[Chris Moore]

I need to take hourly snapshots to grab any files changes that have happened in the last hour and I need to retain all the versions for the last 30 days.

This is easily done on ZFS, but I don't know if it is possible using some other file system.

I need an end user to be able to copy a version of the backup file(s) to their local Windows workstation using Windows Explorer.

I don't think this would be reasonably achievable using any system that I am familiar with. This is something that would be done by a backup administrator. You would not want a user mucking about in the backup server trying to find a file.

My initially research turned up FreeNAS as being a potential solution due to the CIFS capability with the Windows File Explorer 'restore previous versions' functionality.

That might be a possibility, but it is a feature I have not implemented, so I am not sure how that would work for you.

Is FreeNAS not capable of meeting these requirements in some sort of configuration?

You would need to make a copy of the files on the FreeNAS server so the files could be included in hourly snapshots. The problem with that is, you would need to update from the Ubuntu server to the FreeNAS server on an hourly basis so that any change would be included in the next snapshot. Since you are not using ZFS on the Ubuntu server, you could use cron and rsync to push updates from the Ubuntu server to the FreeNAS server and setup a snapshot task on the FreeNAS server to keep hourly snapshots. It wouldn't be a perfect solution, but it could work. It would be better if the Ubuntu system were running ZFS for the file system, then it could make a snapshot and send that snapshot to the FreeNAS server. I think it would be less overhead than using rsync, but it would take a lot upfront to convert to ZFS.

Are the source files required to be stored on ZFS?

It would be better because then the snapshot could be at a block level and only changed blocks would be included in the next hourly snapshot. With rsync, you are copying the whole file every time.

I made a false assumption that only the TARGET files system had to be ZFS and the source files could be from any source the server can read?

You don't have to change from Ubuntu to FreeNAS to get ZFS. You can add ZFS (a file system type) to Ubuntu, the problem with that is, you need to move the data to another storage location, convert the file system then move the files back. There are also some hardware requirements that need to be met, but I have a server at work running RedHat Enterprise Linux with ZFS installed as the file system and it is toring almost 280TB of data on a pool of 60 drives. How much data are you working with here?

 

[consultant]

So to summarize what you are saying:

- The source files for the backup have to be on the same server as FreeNAS which uses the ZFS files system.

- Essentially to make this work I need to mirror the Ubuntu files on the FreeNAS server. So I will essentially end up with two sets of the same data on the same server -- the set of files copied over from Ubuntu and the most recent backups which would be identical which does me no good for backup purposes (disregarding versioning) as if the local storage of the FreeNAS server fails, you lose both the source copy and the target copy of the most recent versions of the files. Granted that is not a catastrophe because there will be the original source copies on the Ubuntu server.

But as you describe it, even though I'll have two sets of the same versions of the files on the FreeNAS server it won't be double the storage due to deduplication?

So basically FreeNAS, as a SERVER backup solution is really only appropriate for backing up a server running on FreeBSD with ZFS (won't run on Ubuntu - a very obscure operating system ;) To accomplish what I want, I basically will need to waste some storage space to mirror the Ubuntu files locally on the FreeNAS server.

I've got about 4TB of data. But the data doesn't change that much. It's a 20-person company and each person probably modifies no more than 20 files a day, so 400 files change, about 4GB worth of data changing per day. I can also get away with daily snapshots, hourly would give more granular versioning but it's not an absolute must.

** What I need to be clear on for determining the storage capacity to provision, is that if I have 4TB of source data on the FreeNAS, and let's say 25% of it has changed, then I would have 4TB source (mirror copy of the Ubuntu server), 4TB backup "seed" data + 1TB changed data = 9TB?

 

[consultant]

I keep forgetting FreeNAS is a storage solution and not a backup solution. It looks like it fits my needs. From reading the documentation it appears the snapshots will only contain the changed data. So 4TB of data with 25% of changed versions of files would only take up 5TB. So essentially my solution is a two-step process. Rsync does the actual backup using the FreeNAS as the backup storage destination. And then FreeNAS handles the file versioning via the Snapshot function.

 

[droeders]

I've used rsnapshot in the past (rysnc based snapshots) for daily, hourly, etc backups under Linux.

https://rsnapshot.org/
https://wiki.archlinux.org/index.php/Rsnapshot

You could use FreeNAS as the destination for your rsnapshot backups, then provide this as a read-only share for users to copy files from as needed. I certainly prefer ZFS snapshots whenever I can use them now, but rsnapshot served me well for several years. Just another option if ZFS snapshots don't work in your use case for some reason.

 

[consultant]

Yes, I already ran across rsnapshot. It would be a simpler solution the downside being the inability to use the 'Restore Previous Version' function in Windows Explorer. Instead users would need to navigate a less simpler directory structure. eg: /.snapshots/daily.0/server/directory/file

 

[Chris Moore]

So I will essentially end up with two sets of the same data on the same server

No. You would have the originals on the Ubuntu server and a copy in the initial state on the FreeNAS server. Then the FreeNAS server could track changes with ZFS snapshots. There must be an initial state for snapshots to track changes against.

the set of files copied over from Ubuntu and the most recent backups which would be identical which does me no good for backup purposes

ZFS would only make a 'copy' if there was a change. No change, no copy, no duplicate file.

if the local storage of the FreeNAS server fails, you lose both the source copy and the target copy of the most recent versions of the files.

No, because the most recent version of the file would be on the Ubuntu server, the FreeNAS server is just getting a copy of that when you rsync it over from the Ubuntu server and keeping track of the versions through snapshots.
I see what you are getting at after reading your next sentence, but a server outage is only going to be temporary as long as you don't let your pool fail. ZFS is very robust if you do it correctly.

But as you describe it, even though I'll have two sets of the same versions of the files on the FreeNAS server it won't be double the storage due to deduplication?

ZFS uses compression by default and it will only increase storage consumption by the amount of change from one snapshot to the next.

So basically FreeNAS, as a SERVER backup solution is really only appropriate for backing up a server running on FreeBSD with ZFS

FreeNAS is an appliance operating system that is built using FreeBSD. It is not an application that runs on top of an operating system.

ZFS (won't run on Ubuntu - a very obscure operating system ;) To accomplish what I want, I basically will need to waste some storage space to mirror the Ubuntu files locally on the FreeNAS server.

I don't understand the way you are looking at this. ZFS is a file system that can be installed in any version of Linux, even Ubuntu. You are not 'wasting' space when you make a backup. A backup is something you should have already. You are attempting to create a system where users are able to restore their own files without an administrator needing to be involved. Did you think it would have no cost at all?

I've got about 4TB of data. But the data doesn't change that much. It's a 20-person company and each person probably modifies no more than 20 files a day, so 400 files change, about 4GB worth of data changing per day. I can also get away with daily snapshots, hourly would give more granular versioning but it's not an absolute must.

You could implement a server with that little storage for (probably) under $2000, less if you are willing to consider used components.

** What I need to be clear on for determining the storage capacity to provision, is that if I have 4TB of source data on the FreeNAS, and let's say 25% of it has changed, then I would have 4TB source (mirror copy of the Ubuntu server), 4TB backup "seed" data + 1TB changed data = 9TB?

You need to project the amount of data growth over the expected life of the server. If you anticipate the server will be in service for five years, and your data grows by 400GB per year.