RSYNC with PC

[eqartimus]

I had been using RSYNC to pull backups off of my PC (running DeltaCopy). I have reciently upgraded my computer -- which required me to reinstall everything. At the same time, I went through several updates with my FreeNAS server. In any case, the rsync tasks don't work anymore and I have spent days trying to solve something that would probably take someone with experience about 5 mins to diagnose.

I have most of it working -- except that permissions are ownership/groups are getting messed up in a way that I can't seem to figure out. I don't know if something happened on the windows side to mess with the ownership -- or if something happened on the FreeNAS side :(

Originally, I had the rsync Task set with user; David == That's me :) uid 1001. David is in Homegroup (1001) and in Wheel, and a few others ... but wheel seems significant.

Now, when I run the task now, the first thing it does is set the permissions on the backup directory to 0. Then it proceeds to fail out because it can't read or write into it.
If I run the task as user:root, it works sorta ok, except I get a ton of errors about not setting uid and gid.

I am backing up my computer into the directory: /mnt/Deerhaven1/Users/David/backup
This directory is set with permissions to 775 before I start the process. After the process fails out, the directory is set with permissions to 0.

Any ideas? I don't know what you want me to show you...

Here is the cron log file entry kicking off the process:

Code:

Jan  9 02:27:00 freenas /usr/sbin/cron[76484]: (David) CMD (PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /usr/bin/lockf -s -t 0 -k "/mnt/Deerhaven1/Users/David/backup" /usr/local/bin/rsync -r -t -a --delete-delay --delay-updates -v -L --exclude-from '/mnt/Deerhaven1/Users/David/exclude-list.txt' "David"@192.168.1.7::"backup" "/mnt/Deerhaven1/Users/David/backup" 2>&1 |/usr/bin/logger -t rsync)

Here is a listing from the /var/log/messages file:

Code:

Jan  9 03:28:11 freenas rsync: receiving file list ... done
Jan  9 03:28:11 freenas rsync: rsync: failed to modify permissions on "/mnt/Deerhaven1/Users/David/backup/.": Permission denied (13)
Jan  9 03:28:11 freenas rsync: rsync: opendir "/mnt/Deerhaven1/Users/David/backup/." failed: Permission denied (13)
Jan  9 03:28:11 freenas rsync: rsync: recv_generator: failed to stat "/mnt/Deerhaven1/Users/David/backup/Masterobiwan.jpg": Permission denied (13)
Jan  9 03:28:11 freenas rsync: rsync: recv_generator: mkdir "/mnt/Deerhaven1/Users/David/backup/DDrive" failed: Permission denied (13)
Jan  9 03:28:11 freenas rsync: *** Skipping any contents from this failed directory ***
Jan  9 03:28:11 freenas rsync: ./
Jan  9 03:28:11 freenas rsync: DDrive/
Jan  9 03:28:11 freenas rsync: rsync: recv_generator: mkdir "/mnt/Deerhaven1/Users/David/backup/Freenas" failed: Permission denied (13)
Jan  9 03:28:11 freenas rsync: *** Skipping any contents from this failed directory ***
Jan  9 03:28:11 freenas rsync: rsync: recv_generator: mkdir "/mnt/Deerhaven1/Users/David/backup/HDrive" failed: Permission denied (13)
Jan  9 03:28:11 freenas rsync: *** Skipping any contents from this failed directory ***
Jan  9 03:28:11 freenas rsync: Freenas/
Jan  9 03:28:11 freenas rsync: HDrive/
Jan  9 03:28:12 freenas rsync:
Jan  9 03:28:12 freenas rsync: sent 346 bytes  received 6,639,460 bytes  577,374.43 bytes/sec
Jan  9 03:28:12 freenas rsync: total size is 565,167,531,856  speedup is 85,118.08
Jan  9 03:28:12 freenas rsync: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1664) [generator=3.1.2]

In the listing above, DDrive, Freenas, and HDrive are existing directories under the 'backup' directory. RSYNC seems to be trying to create these directories since it can't find them after it has set the backup directory to 0.

Thanks in advance. This is totally driving me nuts.

 

[eqartimus]

I seems like user David has permissions to set the permissions to 0. But then lacks the permissions to do anything else. Don't know if this is true or not. By the way, all files and directories have user:group set as David:Homegroup.

 

[eqartimus]

Anyone?

 

[Spearfoot]

Hello, @eqartimus -- would you post your system information, please, per the forum rules? The more detailed information you provide, the more likely it is that an expert will be able to help out. ;)

EDIT: In particular: I get the impression you're running Windows on your PC. If so, what version of Windows? Also, what version of FreeNAS did you start with, when things were working? And what version are you running now, that isn't?

Rsync can be a little quirky. Here are a couple of threads were I delved into some of the darker aspects of it:

https://forums.freenas.org/index.ph...rmissions-support-for-windows-datasets.43973/
https://forums.freenas.org/index.php?threads/easy-rsync-task-creation.46373/

Of course, it's probably to do with permissions - 'cause that's probably the #1 root cause of all problems on Unix systems!

 

[pirateghost]

If FreeNAS is PULLING the rsync across, it is unlikely to be a permissions issue, assuming your dataset is read/writable by root.

 

[JackShine]

I would put all datasets on nobody, nogroup, with full access, then work out the permission on the Windows side.

rsync is fickle.

 

[pirateghost]

The beauty of using rsync, pulling the backup, is that there are no permissions to deal with

 

[eqartimus]

Spearfoot, my current Freenas system Information is listed in the section beneath my message. I thought people could see it. The system *was* working with FreeNAS-8bc815b059fa92f1c8ba7c7685deacbb.

On the windows side, I am working with Windows 10.

BTW, I apologize for reposting the message. I thought that I may have placed it in the wrong section of the forum in the first place.

I am going to try setting all nobody, nogroup, full access. Though like I said before, it's strange how Freenas RSYNC on the Pull is setting the privileges to 0.

I *thought* that this is what pirateghost is saying ... that since I am pulling, I shouldn't have to worry about permissions. I guess I just don't get why Freenas is setting permissions on a folder inside the dataset to 0.

 

[Spearfoot]

Spearfoot, my current Freenas system Information is listed in the section beneath my message. I thought people could see it. The system *was* working with FreeNAS-8bc815b059fa92f1c8ba7c7685deacbb.

On the windows side, I am working with Windows 10.

BTW, I apologize for reposting the message. I thought that I may have placed it in the wrong section of the forum in the first place.

I am going to try setting all nobody, nogroup, full access. Though like I said before, it's strange how Freenas RSYNC on the Pull is setting the privileges to 0.

I *thought* that this is what pirateghost is saying ... that since I am pulling, I shouldn't have to worry about permissions. I guess I just don't get why Freenas is setting permissions on a folder inside the dataset to 0.

I can't make out which FreeNAS version you used that was working, 'cause I've never seen one with that name (FreeNAS-8bc815b059fa92f1c8ba7c7685deacbb). I assume it was 9.3?

Anyway... In one of the rsync-related links I posted above, I discuss the problem of permissions getting dorked when working with Windows. Things don't work. Might apply here. In particular, you're using the '-a' rsync option in your chron job. This is the equivalent of -rlptgoD, where the 'p' tells rsync to preserve permissions. I've had to avoid this option in my scripts to get rsync to work. I filed a bug report about this, referenced at the link above, several months ago.

My suggestion is to modify your chron job, changing the '-a' option to '-rltgoD' (i.e., leaving out the 'p'). This skirts around the permissions conundrum and works well in my experience.

Give it a shot!

Another thing to look at is user account 'David' on your Windows 10 box and the share named 'backup'. How is the share configured? What does it look like when you check its Security properties?

In the meantime, I'm going to try to duplicate your problem here on my systems...

Good luck!

 

[Spearfoot]

Or, if you want to ditch Deltacopy & rsync altogether, consider using the free version of Macrium Reflect to backup your PC directly to an SMB share on your FreeNAS box. It's what I use.

 

[eqartimus]

I like the sound of that ... Macrium that is ... RSYNC has been a real PITA ... I got it to work once before -- and I still have it working to Push a backup of system to a separate Linux box ... but the connection to Windows has always been challenging ... and as I haven't been a computer person for nearly 20 years, I have to relearn everything every time I need to work on my systems.

 

[eqartimus]

So can you walk we through the high level of this ... how you use Macrium. I create an SMB share on Freenas and 'Mount' it in windows. I use Macrium to 'backup' my normal files to that SMB share? Is that the idea?

 

[Spearfoot]

So can you walk we through the high level of this ... how you use Macrium. I create an SMB share on Freenas and 'Mount' it in windows. I use Macrium to 'backup' my normal files to that SMB share? Is that the idea?

You're exactly right.

I created a FreeNAS dataset called... wait for it... 'backups'! I set it up as an SMB share. Then I create a folder in this share for each Windows PC on my network. I direct Macrium Reflect to store its backup images there. Reflect has a scheduler, but I prefer to run it from Task Scheduler. Reflect stores backup jobs as XML files in your user folder; you'll need to run it and set up appropriate backup job(s) for your needs. In my case, I run two jobs: a daily full system backup of my Windows boot disk, and a weekly full backup of my larger 'data' disk (which contains mostly VirtualBox VM images that don't change often 'cause I hardly ever use them). Here's a screenshot of the Task Scheduler setup for my main Windows workstation, 'Onyx':

...where you can see I call Reflect and pass it an XML file to run a backup. The 'file-groom.exe' program is a utility I wrote that's similar to rollogs, i.e., it deletes older backup images so that they don't fill up my NAS!

I can't recommend Macrium Reflect highly enough: it's saved my bacon at least once. It makes it simple to restore a system image to a new drive. It let's you create a bootable ISO/CD/USB disk custom-tailored for your system that will let you restore an image over your network. The only thing the free version doesn't do that I wanted is the file maintenance, which is why I wrote the file-groom utility I mentioned.

Basicallly, 'It Just Works', as Jerry Pournelle used to say.

 

[eqartimus]

How come your freenas doesn't automatically delete older backups? I'm unclear about that part. My presumption is that your backup is copying to your dataset --so why would your Freenas be making snapshots with a stated shelf life? Am I missing something?

 

[Spearfoot]

How come your freenas doesn't automatically delete older backups? I'm unclear about that part. My presumption is that your backup is copying to your dataset --so why would your Freenas be making snapshots with a stated shelf life? Am I missing something?

FreeNAS only provides storage in this scenario, not any kind of file management. All modern file servers: FreeNAS, FreeBSD, Linux, Windows, etc., can be configured to provide SMB storage. But we can't expect them to automatically manage our files for us. In this case, we're writing backup files to an SMB share: how is the SMB server supposed to know which files we want deleted, and when to delete them?

Reflect runs on your Windows PC and creates a backup of your system drive(s) as a single file. It creates another backup file each time you run it, so these files will accumulate over time, and they're large, so we need to regularly delete older versions. But FreeNAS, or any other server, doesn't have anything to do with managing these files; that's up to us. Now to be clear, FreeNAS does provide tools allowing us to manage files: in this case we could set up a scheduled chron task to delete the older backup files. But the burden is on us to set this up; not on FreeNAS or indeed any other server operating system.

Snapshots don't have anything to do with this. They are a FreeNAS feature, used to keep archival snapshots of the ZFS filesystem, which incidentally can include our backup dataset, depending on how the snapshots are configured. This allows us to retrieve past versions of existing files, and with replication, allows us to backup the FreeNAS server itself. But they don't have anything to do with directly managing our backup files.

For my Windows PC backup strategy, I chose to use the file grooming program I mentioned above to take care of deleting older backup files, like this :

• Every night at 11:35PM: run Macrium Reflect to create a new "Drive C:" backup, then run file-groom.exe to delete "Drive C:" backups more than 8 days old.
• Every Sunday night at 8:00PM: run Macrium Reflect to create a new "Drive D:" backup, then run file-groom.exe to delete "Drive D:" backups more than 15 days old.

 

[eqartimus]

Your SMB share is already a ZFS filesystem. Creating a snapshot is rather trivial. The 'grooming' you do looks a lot like what I have going on with my snapshotting. Now, all I have to do is consistently reflect the changes from my current computer to the SMB share and all my backing up and grooming will be managed by Freenas. Am I missing something?

 

[Spearfoot]

Your SMB share is already a ZFS filesystem. Creating a snapshot is rather trivial. The 'grooming' you do looks a lot like what I have going on with my snapshotting. Now, all I have to do is consistently reflect the changes from my current computer to the SMB share and all my backing up and grooming will be managed by Freenas. Am I missing something?

Yes, sir, you're completely missing my point. :)

FreeNAS isn't backing up your PC, and it's not automatically grooming your backup files.

Yes, SMB is a service provided by FreeNAS; it delivers access to storage; storage which is based on ZFS. And that's all we're using FreeNAS for here: storage.

The file administration I'm talking about has nothing to do with snapshots. It's simply file management. In this case, deleting old files which we no longer need or want because they're outdated.

Here's the sequence of events:

• Reflect creates a backup image of your windows PC, which ends up as a single file, stored on a dataset accessed via the SMB service on your FreeNAS server. This backup file contains a full image of your Windows PC disk(s).
  
• Each succeeding time you run Reflect, it creates another backup file, stored on your FreeNAS server alongside previous backup files. Again, each backup file contains a full image of your Windows PC disk(s). Over time, these rather large files will accumulate, using up vast amounts of storage. Which means that you must delete older backup files, else they will fill all of your storage. Which is why:
• I run the file grooming program, which deletes these older files.

But you can handle the issue of deleting these older files in other ways:

• Ignore the problem until your NAS is full; then delete the old files.
• Mark your calendar to periodically remind yourself to delete the old files.
• Set up a chron job running a script to delete the old files.

But again, snapshots are completely unrelated to any of this...

Hope this helps!

 

[eqartimus]

I think I understand what you are doing. And I have been unclear in my last two messages. I am trying to understand "for my purposes" whether I want to do 'as you are doing' or something similar. So, I have been asking questions about your setup and your decisions regarding using Reflect vs. Snapshot to understand WHY, not how. My bad, I wasn't too clear.

So I guess I am wondering why I would not let Freenas manage the pruning jobs with snapshots. If my SMB share was kept up-to-date with incremental backups from Reflect, AND that smb share was set to snapshot and delete old versions, what would I miss? What are you getting from making multiple large backups that I wouldn't also have from my snapshots?

BTW, thanks so much for your time and attention to this.

 

[Spearfoot]

I think I understand what you are doing. And I have been unclear in my last two messages. I am trying to understand "for my purposes" whether I want to do 'as you are doing' or something similar. So, I have been asking questions about your setup and your decisions regarding using Reflect vs. Snapshot to understand WHY, not how. My bad, I wasn't too clear.

So I guess I am wondering why I would not let Freenas manage the pruning jobs with snapshots. If my SMB share was kept up-to-date with incremental backups from Reflect, AND that smb share was set to snapshot and delete old versions, what would I miss? What are you getting from making multiple large backups that I wouldn't also have from my snapshots?

BTW, thanks so much for your time and attention to this.

You're very welcome! But we're still talking at cross purposes... which is no doubt owing to my inability to explain things clearly.

It's not a question of Reflect vs. Snapshots. The two are completely unrelated.

Reflect is a Windows program. It runs on your Windows PC. It creates backup images of your Windows PC disk or disks. Where you store these images is up to you; it can be on a FreeNAS server, a Windows server, a Linux server, a NAS, somewhere on the cloud, etc.

FreeNAS can't back up your Windows PC and FreeNAS snapshots have nothing to do with backing up your PC. However, as noted above, you can store your Windows backup files on FreeNAS. FreeNAS can be configured to delete old snapshots as they age out, but this applies only to snapshots - not to specific files on your FreeNAS server, and that's all the Macrium Reflect backup images are; files on your FreeNAS server.

The FreeNAS snapshot system can't reach down to your Windows PC and back it up - that's something you must do somehow. In my case, I use Macrium Reflect. I make a full backup each evening, as I have never - in over 30 years - had any luck restoring a system from incremental backups, whether from CD or DVD or DAT tape or images files or whatever. Because I make these backups on a daily basis, these large backup files accumulate, potentially using up all the space on my FreeNAS system. And, again, let me point out that these are simply files, not snapshots. That's why I delete the older files; I don't need them any more - they're outdated; and if I don't delete them, they will eventually consume all of my storage capacity. And again, snapshots have nothing to do with this; the snapshot system doesn't delete your files for you, it only deletes older snapshots.

Phew! Hope this helps.

 

[eqartimus]

Ok, so the first part of 'cross-purposes' is that you are working with the constraint of 'incremental backup/restore is problematic' which means that for you, a full backup file needs to be created daily for your level of security. I get that now -- so for you, you need to create a huge 'file' that is your entire system - daily. I get it, its a big hammer, and it works flawlessly every time! :) Second constraint is that you want a full system backup daily -- such that you can restore and bring your systems up immediately. For me, I am less interested in having the entire system backed up -- I want just data really. If the system crashes, or I have to reinstall, I don't mind so much as long as all work product has been retained. I do have a system backup -- but I don't care about backing that up regularly.

When I started using Freenas, I fell in love with snapshots :) Being able to take shots a couple times a day, weekly and monthly -- then automatically prune them -- and know that they didn't take extra space besides what changed -- appealed to me on some emotional level :P
And I did mount those snapshots from time to time to resurrect changed files and such. So I am inclined to keep using that as my backup.... but how *best* to get my PC files to the Freenas share has been the issue.

For my wife, I made the SMB share the drive where all her downloads and documents are stored by default. So her data is snapshot daily, weekly and monthly without anything for me to worry about.
For me, I need more speed and access, so I don't want to work directly off the share and need to sync between my data and my share -- and then let Freenas do it's thing.

RSYNC has been annoyingly finicky about settings ... when It was working, it was fabulous. Backups took minutes -- but whenever I change something -- it fails, and I'm just not technical enough to keep that stuff in my mind -- and have to search the net and get reeducated any time something happens.

So for me what I want to do is find a way to either 'mirror' or routinely sync my local data to my SMB share. Then freenas just does it's thing. Seems to me that I could use Reflect for that and get the best of all worlds...

I hope this makes sense.