rsync - permission problem FN8-B4-AMD64

[fisheater]

Hi all,

Have been following FN for some time, took hardware recommendations from all your posts, and now have a FN8 setup in my basement working well.

Problem: using rsync from linus to FN8 gives permission problems.

Setup: FN8-B4-AMD64, windows 7 (backup working ok), ubuntu 11.04 (rsync backup fails).

Background:
I have mounted the FN8 HDs by editing the fstab = permanent mount. They are working to copy/edit/delete files/folders by explorer and nautilus.

I have turned on the rsycn service on the FN8 box, but have not changed or added anything to the Auxiliary parameters box.

When I try to rsync from linux to FN8 see the error log below.


Where do I have to set the permissions? If I understand rsync my FN box is the server, and my linux desktop is the client – please help me understand the basics in a basic way! (I have only my curiosity to guide me, and no formal computing background).

Please direct me to a simple guide to help set this up.

Thanks!

Code:

*** Launching RSYNC command:

pkexec rsync -r -t -v --progress --ignore-existing -s /media/Everest /home/goatmilk/FreeNAS2_Backup



sending incremental file list

rsync: readdir("/media/Everest/.systemfile"): Permission denied (13)

Everest/

Everest/.systemfile/

rsync: failed to set times on "/home/goatmilk/FreeNAS2_Backup/Everest": Permission denied (13)

Everest/.wd_tv/

rsync: recv_generator: failed to stat "/home/goatmilk/FreeNAS2_Backup/Everest/.synkron.syncdb": Permission denied (13)

Everest/Backup/

rsync: recv_generator: failed to stat "/home/goatmilk/FreeNAS2_Backup/Everest/.uid_***": Permission denied (13)

rsync: recv_generator: failed to stat "/home/goatmilk/FreeNAS2_Backup/Everest/ffp setup.txt": Permission denied (13)

rsync: recv_generator: failed to stat "/home/goatmilk/FreeNAS2_Backup/Everest/ffp.log.temp": Permission denied (13)

rsync: recv_generator: failed to stat "/home/goatmilk/FreeNAS2_Backup/Everest/fun_plug_temp": Permission denied (13)

rsync: recv_generator: failed to stat "/home/goatmilk/FreeNAS2_Backup/Everest/watermelon.txt": Permission denied (13)

rsync: recv_generator: mkdir "/home/goatmilk/FreeNAS2_Backup/Everest/.systemfile" failed: Permission denied (13)

*** Skipping any contents from this failed directory ***

rsync: recv_generator: mkdir "/home/goatmilk/FreeNAS2_Backup/Everest/.wd_tv" failed: Permission denied (13)

*** Skipping any contents from this failed directory ***

rsync: recv_generator: mkdir "/home/goatmilk/FreeNAS2_Backup/Everest/Backup" failed: Permission denied (13)

*** Skipping any contents from this failed directory ***

Everest/MP3s/

rsync: recv_generator: mkdir "/home/goatmilk/FreeNAS2_Backup/Everest/MP3s" failed: Permission denied (13)

Everest/Photographs/

*** Skipping any contents from this failed directory ***

rsync: recv_generator: mkdir "/home/goatmilk/FreeNAS2_Backup/Everest/Photographs" failed: Permission denied (13)

Everest/Temp/

*** Skipping any contents from this failed directory ***

rsync: recv_generator: mkdir "/home/goatmilk/FreeNAS2_Backup/Everest/Temp" failed: Permission denied (13)

Everest/Videos/

*** Skipping any contents from this failed directory ***

rsync: recv_generator: mkdir "/home/goatmilk/FreeNAS2_Backup/Everest/Videos" failed: Permission denied (13)



*** Skipping any contents from this failed directory ***

sent 3348106 bytes  received 8484 bytes  10951.35 bytes/sec

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1060) [sender=3.0.7]

total size is 673457405091  speedup is 200637.37

Rsync process exit status: 23

 

[ProtoSD]

Hi fisheater, welcome to the FreeNAS 8 forums. It looks like you've done a pretty good job getting things setup overall. The problem you are having is because the UID & GID, two different numbers that are assigned to users and groups are different on your linux system than on your FreeNAS system.

There are 2 ways (actually more) that you can fix this. You can either make them the same on your FreeNAS system as your Linux system, or the other way around. The numbers need to be unique, so if you make one system agree with the other, then those numbers shouldn't be in use by any another user on the system on which you change them (unless that user is also a member of the same group). The other thing you can do is create a group on FreeNAS and set the group number so it matches the group number on your linux system, and then change group on all your files on FreeNAS, which is really easy to do. Unfortunately on FreeNAS there's not any easy way to add an existing user to group that you add, but there's a way.

The first thing you need to do is find out what those values are on both systems, and what the default user & group is set to for your folders on FreeNAS.

From most Unix/Linux based systems, from the command line, you can type the command 'id' and it will tell you your UID & GID. So do that on your Linux system and write it down.

Now, if you 'cd' to the folder you are trying to copy files to and do an 'ls -l', you can see what the user and group is of the files in that folder, or 'ls -ld .' to find out the owner/group of the folder itself. You should do both just to confirm that they agree.

Below is a screenshot showing where you'll see the owner and group. If you are doing this from your Linux system and not the console of your FreeNAS system, you'll see the UID & GID numbers instead, unless by some chance they just happen to coincide with some other user and group on your Linux system.

<-- (this is a really tiny screenshot, looks like a line, click to enlarge it)

So lets take this approach:
On your FreeNAS system, from the GUI, on the left sidebar, go to Accounts -> Groups -> View All groups.
On the right side where all the groups are listed there is a column labeled ID. Verify that the GID you got from your Linux system DOES NOT exist in that list.
If it does, post back and we'll come up with another way.

Scroll back up to the top of that list and click on 'Add New Group'
Enter the GID you found on your Linux system and name the group something like "commongrp'' and click OK.

Now, back on the left click on Account -> Users -> Add User
Set the UID to the one you got on your Linux system when you did the 'id' command.
Set the username to anything you like or make it the same as your Linux account
Click the dropdown for Group and find the group you just created above 'commongrp'
Leave the home directory non-existent for now since we won't make this an active user
Change the Shell to 'nologin'
Click the checkbox for 'No Logins'
Set all the rest to whatever you like and click OK

I might be missing where to do this from the GUI, but from your FreeNAS console choose #9.
When you see the % prompt, type 'su' and enter the admin password which should be 'freenas' unless you've changed it.

'cd' to the mountpoint/folder you're trying to sync to this should be the folder you setup under Services -> Rsync on the left sidebar
for example 'cd /mnt/tank'

do 'chown -R username:group ./*'
(username is what you just created above & group is the group you created above)

Ok, back at the GUI.

You need to go back to the settings for the Rsync Module you setup originally.

On the left sidebar under Services -> Rsync -> Rsync Modules
Open the module you created

There are 2 drop-down menus, one for user, one for group
Make sure those are set to the user and group you created above
Click OK

I *think* that should do it, try your copy again. Sorry for all the details, I'm sure you're familiar with some of it, but I wanted to make sure others that might not be as familiar could understand it also.

 

[ProtoSD]

Fisheater, I'm still working on the post above because a lot of people don't understand permissions, and I think it could still be helpful, but I feel like an idiot because I'm not sure your problem is what I thought it was. I've had one of those days and I'm not sure it's over yet! ;-)

You have two folders, /home/goatmilk/FreeNAS2_Backup/Everest and /media/Everest.
/media/Everest is your pool on your FreeNAS system?
and the other is your Linux system?
and you are executing the rsync command from your Linux system?
maybe I am still right, but let me know those answers and I'll adjust my advice, or maybe ask some other questions.

Thanks

 

[ProtoSD]

Fisheater,

I looked at your original post again and think I might have answered my own questions. I might still be right about my original interpretation. Give it a try and let me know if you need anything clarified. I had a lot on my mind today, sorry it took me awhile to get through all that!

 

[fisheater]

Thanks for the great guide, the level of detail was just what I needed. I really am out of my depth with this, but am enjoying thinking my way through these puzzles. Here is my progress so far (spoiler: not good). Ill keep trying, and will read up about BSD command line syntax (including how to su). Thanks again for having a think about this.

FE

Step 1
id = 2000 2000 (my desktop matches the destination folder - i used terminal in linux to check both my home directory, and the FreeNAS directory)

Step 2
ls -l = drwxrwxrwx

Step 3
'On your FreeNAS system, from the GUI, on the left sidebar, go to Accounts -> Groups -> View All groups.'
linux GID is not in this list.

Step 4
Add New Group
-added my GID

Step 5
Add New User
-added my UID, followed the steps

Step 6
-console (option 9) on FreeNAS tower - chown -R username:group ./*

Step 7
-setup rsync module on FreeNAS. Path-to my FreeNAS HD, read and write, user-kidneystone, group-kidneystone

Final step: try rsync.
-same error.

But I think that it is user error. I am not fully understanding step 6.
- I set my UID to 2000, GID 2000, group kidneystone, user kidneystone.
Issue #1:
- On FN machine - su takes me from '%' to 'freenas#'
- I am unable to enter a password. When I tried su (password), it gave me unknown login.
- I have only changed my login/pw via GUI.
Issue #2
- When I ran chown -R username:group ./*, I used kidneystone:kidneystone
- Do I need to run 'chown -R 2000:2000 ./*' ?


- Tried 'sudo chown -R kidneystone:kidneystone ./*' from ubuntu terminal, but no luck.

To answer your other questions
"You have two folders": pkexec rsync -r -t -v --progress --ignore-existing -s /media/Everest /home/goatmilk/FreeNAS2_Backup
/media/Everest.
- this is my source folder (my old NAS)
/home/goatmilk/FreeNAS2_Backup/Everest
-this is destination (my FreeNAS box)

I am running my commands from my linux desktop (ubuntu 11.04)


Kind regards,

FE

 

[ProtoSD]

LOL Fisheaster! I was trying to make sure I got all the steps listed, but I didn't organize them as nicely as you did so I need to figure out which step of mine corresponds to your numbered steps. Let me try and answer some of your easy questions first.

- On FN machine - su takes me from '%' to 'freenas#'
- I am unable to enter a password. When I tried su (password), it gave me unknown login.

When your prompt changed from % to #, that means you have root privileges. If you just type su, it should prompt you for a password.

- When I ran chown -R username:group ./*, I used kidneystone:kidneystone
- Do I need to run 'chown -R 2000:2000 ./*' ?

If you setup the UID & GID in the GUI correctly then you should only need to do the first one because the number 2000 is now associated with the user kidneystone. Usually the UID isn't the same as the GID, but looking at my Ubuntu 11.04 system I noticed mine are both 1000. I suppose it doesn't really matter.

- Tried 'sudo chown -R kidneystone:kidneystone ./*' from ubuntu terminal, but no luck.

Yes, you need to do the 'chown' command from your FreeNAS system for this to work (without the sudo). Sudo is ummm, well it's like using su, but it's handled differently on Linux.

I'm not sure why you are using 'pkexec' this is also similar to su and sudo, but you shouldn't need it if your permissions are correct.

Whenever I usually use rsync it's either using ssh, or the destination is mounted with nfs on my source system. So doing it the way you're trying to do is a little different than I'm used to, but the idea should be the same.

Step 2
ls -l = drwxrwxrwx

This should probably be changed because the last 'rwx' gives the world read-write-execute to all the stuff in that folder. If that 'ls -l was for the folder on your FreeNAS system, then from the command prompt there, after doing 'su', do 'chmod 750 /mnt/your-pool/your-folder', then do the ls -l again and it should look like 'drwxrwx---'

Ill keep trying, and will read up about BSD command line syntax (including how to su)

If you use the commands I'm telling you, you should be fine, just replace the stuff like 'your-folder' with what you have. Of course reading up never hurts either!

So I guess recheck what I just explained and try giving it a go without the pkexec too. I'll try to duplicate your setup here and see if I have any problems using the Rsync Module instead of doing the other ways I mentioned.

Good Luck!

 

[Milhouse]

Assuming you have an rsync module created on your FreeNAS server, let's call the module "fnmodule", and the FreeNAS server has a DNS entry called "freenas" (or use your IP address if not), you would normally "push" your data from the Linux server to the FreeNAS server using the following rsync command executed on the Linux server:

Code:

rsync -r -t -v --progress --ignore-existing -s /media/Everest freenas::fnmodule

If you specify root:wheel as the rsync module user:group on the FreeNAS server you should avoid permission issues - any files will be created on the FreeNAS server with the same UID:GID as the Linux server, even if they're not valid UID's and GID's on the FreeNAS server (you'll just see numeric IDs when they don't match). Obviously this might be a potential security concern, in which case if all the files are owned by a single Linux user (or group of Linux users) then sorting out your UIDs/GIDs to be the same on both servers and specifying a more restrictive user:group account for the module on FreeNAS should resolve the issue.

In the module you specify the full path to the storage location on the FreeNAS server, eg. "/mnt/tank/backup" or whatever, and this way you don't need to mount the FreeNAS backup storage on your Linux server. You can "pull" the data from the FreeNAS server by reversing the source and targets in the rsync command (eg. "rsync <args> freenas::fnmodule /media/Everest" would restore your backup data to your local /media/Everest directory - again, without mounting any FreeNAS storage on your Linux server).

I normally use the following arguments on my rsync jobs - you should be able to work out what they do from "rsync --help", and if they'll be right for you:

Code:

rsync --delete --stats -tiprogl --progress /media/Everest freenas::fnmodule

 

[fisheater]

Great news, thanks for all your help! At the moment it seems to be working quite nicely, with some slight modifications to what you posted.

rsync --delete --stats -tiprogl --progress /media/Everest 192.168.1.299::fnmodule

It worked without mounting the HD, which was a new concept to me. I had previously struggled to mount my HDs and used grsync to do the work.

I will decode the <args> and try to set it up like my backintime incremental backup folder.

Thanks again for all your help, this is closed for now - barring disaster from tonights sync.

Kind regards,

FE

 

[fisheater]

Great news, it works! Thanks again everyone for your suggestions.

The rsync syntax is going to make my head explode, but now my files will backup with ease.

**this is great stuff, thanks to all the FN developers, testers, and all the posters for helping me with my NAS.

Kind regards,

FE

 

[ProtoSD]

Glad to hear you got it working Fisheater. Sorry I lead you down the wrong path. I hope it was still helpful in some way.

@Milhouse, thanks for helping me learn about a different method of using rsync. I wish it had just been a few posts earlier so I hadn't sent @Fisheater on such a wild goose chase! ;-)