Best way to backup freenas data of Mac files

[jonandermb]

I have a particular scenario where I store all the data on a freenas server from multiple Mac computers via Afp.
When I want to make the backup, I'd like to avoid doing through the network because it takes a lot of bandwidth. I'd like to know if there's some solution involving plugging a disk via USB.
I currently rsync everything to a USB disk attached to a Mac machine (I don't want to lose the file permissions and attributes) and at the time the backup is being run, the performance decreases for the rest of the users.
This is a small company and the backup is an off-site system, so, leaving the disk plugged overnight to perform the task is a no-no....
What ideas could you give me?
Thanks.

 

[cyberjock]

Not really. What you are asking for seems almost impossible. Can't use the network, and can't leave a disk to run overnight. What options *do* exist that would let you do a backup that wouldn't need time or bandwidth? Seems like you are asking for the impossible to me. Am I confused?

This sounds like the common dilemma with any given technology. You either wait for it to finish (time) or make it faster (more throughput). Throughput isn't an option and you are saying time isn't.. so what are the options?

 

[Alvin]

Why not attach the USB disk to the FreeNAS box and rsync from there? The method is no different from doing it on the Mac itself. It'll be faster. (Also, the version of rsync on FreeNAS supports extended attributes, while the Mac version does not. This can be important for some Mac files.)

 

[aufalien]

Well, something stands out here. You said perfamance slows for all during backup. But is this a function of your pool or network? In other words are you disk bound or nic bound?

If disk, then it is what it is. However what warez are you using for backup, CCC, SuoerDuper, etc...

And is your scheme grandfather etc... in terms of incrementals and fulls?

Just trying to get an idea of your scenario.

There may be some ways, even if disk bound, to minimize impact on user experience.

 

[aufalien]

I just re read your post and you are using rsync, however are you doing full rysncs every day and do you need to?

Delta rsyncs should not be a burden to disk or network as much as full rsyncs and you can, through nice, lower your rsync priority to run really low as well as telling rsync to only run at a certain very low kb/s etc...

I think your issue is workflow related so tweaking your approach is the best bet. And yes Alvin has a very good point in rsync and extended attributes. However if this has not been an issue so far, then its a non issue. However it would be nice to keep those attributs as you may need them in the future.

 

[jonandermb]

Thanks for the replies :)
About the performance impact, it's the network that suffers most. Even though I created a bond device with two nics, the network is 1Gb (It's a small company, not a datacenter, or microsoft or google, meaning: We have to recycle and extend the lifetime of the infrastructure and hardware :) ) so while performing the rsync backup, other users might notice a certain performance drop.
After reading this forums, I came across lots of comments disregarding plugging usb devices, creating pools with them and detaching them from the machines but I always wondered why is it so much trouble to implement a small tool that could let you attach a device and rsync data via the internal bus of the machine. For my setup, this sounds like the easiest, most straightforward system. However, the way I have to do things now is more complicated and seems kind of like a patch, a workaround.
Lately I have been trying to use a lacie BigDisk I have here. It's 4Tb and network attached, so, I'd plug it to the lan, give it an IP and enable a module for acting as a rsync receiver, but Lacie products suck, or at least their firmware, which is outdated, and the rsync version on it gets in conflict with freenas rsync version and I have been coming across a certain amount of problems, like the "delete" option not working or even permissions problems.
My intention here is to see if someone has a way of simply attaching a USB device to the freenas machine, for the reasons I explained before, becauseit's the only thing that came to my mind, but seems not doable.
I'm all ears: If you need any more information about my setup and the resources or the things I can try, just tell me.
Alvin: You suggest I just pllug the device, enter via console, format the external disk with something like mkfs.ext4 on the USB, mount it and rsync? can that be done?

Thanks

EDIT: I perform incremental rsyncs, but besides the network impact, I also have some permissions issues on the lacie device.... :)

 

[burnmai]

K, im going to try something here, but others will need to help me out; im still to new to make recommendations.

Your current situation:

Backup<--usb-->Applecomputers<--network--AFP-->Freenas
.

(Did i understand that right?)

What you could maybe do is remote_replication trough a second NIC in the freenas box.
this zonesout the load on the primary network and creates a backup on a second server.

I'd look something like this: (scematicly that is)

Backup<--usb-->Applecomputers<--network--AFP-->Freenas<--Network(2)--remote replication-->Freenas(2).
.

i might be wrong here, so just wait until others rate my idea.

 

[diedrichg]

FTP will not saturate your LAN if you set a maximum transfer speed.

 

[jonandermb]

FTP will not saturate your LAN if you set a maximum transfer speed.

Wouldn't that mess the permissions and ownerships up?
Good idfea, though, because all I need is that: to tweak the bandwidth and maybe give priority to afp connections vs rsync

 

[Alvin]

Alvin: You suggest I just pllug the device, enter via console, format the external disk with something like mkfs.ext4 on the USB, mount it and rsync? can that be done?

Yes, with the exception of ext4. You'll need to format as UFS with newfs(8). UFS is supported on OS X, so reading your backups will not be a problem.

 

[aufalien]

Would it be over kill to format as ZFS and then use replication? If needing to mount on OSX, install maczfs on it. This way there are zero issues with attribute mangling etc... Plus you have the benefits of ZFS albeit a single disk. You then have a workflow that can scale well etc...

 

[diedrichg]

Wouldn't that mess the permissions and ownerships up?
Good idfea, though, because all I need is that: to tweak the bandwidth and maybe give priority to afp connections vs rsync

To edit my comment, you could limit the (S)FTP speed on the client ftp program.

 

[jonandermb]

Quick question: Correct me if I'm wrong but, regarding the option of formatting the external disk on UFS and using it as backup, I think I read somewhere FreeNAS was dropping UFS somewhere in the future, am I right?

Currently performing some tests on a small disk: I'll report with the results :)


UPDATE: I tested creating a ZFS filesystem on the external drive, copied some stuff on it, dettached without destroying data and then, went to my mac. Installed OpenZFS for mac and when trying to import I get an error because, I guess, different versions of ZFS

com.delphix:hole_birth

Hmmm....... almost perfect plan, I think I'll give a try to UFS again

 

[Alvin]

Quick question: Correct me if I'm wrong but, regarding the option of formatting the external disk on UFS and using it as backup, I think I read somewhere FreeNAS was dropping UFS somewhere in the future, am I right?

I believe they're thinking about that, yes. But possibly only in the GUI storage manager. Since UFS is used to format the USB stick itself, it needs to be supported in the kernel. And newfs(8) is barely 33k.
Removing it entirely would be too much effort for no benefits.

 

[cyberjock]

UFS is *dead*. The code is removed from FreeNAS. Not because of its size but because the developers are tired of dealing with bugs that involve UFS when only an extremely small fraction of users use UFS. The benefits of the many outweigh the benefits of the few.

So no, UFS is not recommended for this and if you are in a position where you need to use something besides ZFS there are plenty of other OSes out there that you are welcome to try.

But, UFS is *dead*. It was removed months ago from the master branch. It won't be coming back. Even the developers are glad to see it gone. :)

 

[Ericloewe]

I believe they're thinking about that, yes. But possibly only in the GUI storage manager. Since UFS is used to format the USB stick itself, it needs to be supported in the kernel. And newfs(8) is barely 33k.
Removing it entirely would be too much effort for no benefits.

FreeNAS 9.3 will move to a ZFS system partition, so UFS is truly dead around here.

 

[burnmai]

Then why no replication?
trough a different NIC that is.

 

[Dennis.kulmosen]

Yes, with the exception of ext4. You'll need to format as UFS with newfs(8). UFS is supported on OS X, so reading your backups will not be a problem.

UFS is no longer supported in OS X either so as CJ says UFS is truely dead as in dead.
I would go with attaching a usb drive directly to the freenas box using ZFS, put on ZFS on the mac using openZFSformacosx it works very well with some small manual commands.


Sent from my iPhone using Tapatalk

 

[aufalien]

Yea agreed, local replication and in times of DR/BCP, move external media off to CrApple box as an option.

 

[jonandermb]

Hi guys :) I'm just testing now what I finally went for:

4tb USB disk attached to the freenas machine.
I created a volume via the web ui, formatted with ZFS, mounted it on /mnt/Backup
Then, I ssh'd to the machine. Now, I'm currently running rsync -aEv --progress /mnt/STORAGE/* /mnt/Backup/

The process will take a lot of time, because there are lots of .ai, .idd, .pdf, .zip, .jpeg, .png,..... etc small files, and some other huge ones.
So far, no one complained about the system performance, so, so far, so good.

The only negative thing I can find is that I sometimes like to get the backup disk and be able to mount it on any device, so, just in the scenario where the machine crashes, I can plug the disk elsewhere and just extract what I need at the moment.
The other day I tried to do so, but seems like zfs for mac is in a different version than freenas, so, I guess I'll only be able to plug it on the original machine.


When I'm done copying to the USB, I plan on removing the Backup volume (without data deletion) and rsync again in a couple of days with the --delete option, and see how it behaves, but so far, I like this way.

Thanks


EDIT: ZFS for mac complains about com.delphix:hole_birth (Unsupported version or feature) and does not let me mount the volume. Any ideas?