RSync to USB issue

[SpreadFoxtrot]

Hi,

I am new to FreeNAS, so be gentle with me.

I use FN as a storage for an ESXi 5.1 through iSCSI, that part works GREAT.

Then I decided to replicate the VM files (.vmdk files) to a USB external drive connected to the FN.

I used RSync, pushing the "Data" partition to the usb drive, it seems to work but after a few hours, I came back, the 2 rsync processes were still running.

So I checked the USB drive usage and it's way too much data (check pictures)

Picture: first partition is the internal drive holing the running VM, second is the USB drive


As you can see, too much data is backed up compare to the origin drive.


Since the data is 1 running VM, I thought maybe RSync what trying to sync the VM in real time.

So I stopped the VM, but the RSync processes are still running and the data on the external HDD is still growing as I write this post.


I am not familiar with either RSync or FreeNAS, so maybe I am not getting something right.

I don't know what file/configuration I should add to this post, so tell me what you need.

If you have a better solution to backup to USB, please tell me, I just need it to be automatic on a daily basis!

 

[cyberjock]

Why do you want to backup to USB? FreeNAS has its own builtin method for restoring from a failed boot disk. The config file. If your boot drive became unusable it would take about 15 minutes tops to recover.

 

[jgreco]

I'm confused. You've got ESXi hooked up to FreeNAS via iSCSI, right? So you have some iSCSI extents on the FreeNAS server.

So then you're replicating the vmdk files to a USB disk on the FreeNAS system.

Um, how? Be specific.

Are you NFS-exporting the USB disk to the ESXi system? And doing the rsync on the ESXi? Because really, if you're using iSCSI, only the ESXi box is going to have access to vmdk files. And those files cannot be safely copied or rsynced without using VMware's vmdk manipulation tools (I don't recall specifics offhand) - you can use tools like ghettoVCB (?) to make copies to an NFS datastore but there's no place for rsync in that picture.

See, I get lost when you say "pushing the Data partition" because it's totally unclear what you're referring to, or where that is, or what it implies, so I think you're not meaning what I'm talking about above - but I can't figure out what you ARE doing or why you expect it to work a certain way.

 

[cyberjock]

I thought he was trying to backup the OS. But looking closer at the pictures included, it shows 1TB, not 1GB. So yeah, I don't know either.

 

[SpreadFoxtrot]

Sorry about my english, it is in fact my second language.



The ESXi is using FN as a iSCSI target.
I want to back up FN DATA (extent file) to a USB drive (external backup is essential in my case)

I used RSync to backup the main-storage of the FN to the usb drive connected to the FN.

As it turns out, when RSync is syncing the extent file, it backs up the entire extent file (which I configured to be 200gB).

So I now know why RSync was syncing such a big file.


Sorry for the misunderstanding.

Is there anyway to delete this post, since there are actually no issue with my rsync?

 

[jgreco]

No, but as long as you understand what's going on, that's just fine. Your initial description was a little hard to decipher, because you said the "VM files" and went on to even say ".vmdk" when you were describing what you were copying. You meant that you were copying a file extent served up by FN as iSCSI, which coincidentally (and irrelevantly) held .vmdk files. It could have just as easily held DOS .exe files on an NTFS filesystem and you'd have the same problem.

The answer, then, to your question is that the extent file may be 200G but it may not be using all of that on-disk. However, copying it will cause it to use all that space on the backup, unless you tell rsync to use "--sparse" mode (see command line switch --sparse). This will have the side effect of being MUCH faster for the time being, until your extent starts getting closer to full.

Enjoy!