Moving data between volumes using cp.. is this ok to do?

[ReggiePerrin]

Hi

I'm new to Freenas/Unix so please bear with me if this appears to be a really Noob question :)

Firstly this is a great product, my HP microserver arrived yesterday and within an hour I had Freenas up and running; no fuss, no problems... The longest part was waiting for the Freenas image to burn to the USB key.

Scenario/plan:-

I'm migrating a lot of DVD rips from a WHS v1 server into a temp 2TB volume I've created on Freenas, I then want to physically move a 2TB drive from the WHS server and put it into the Freenas server, create a single drive ZFS volume on this drive and then copy everything across from the temp volume.

Finally blitz the temp volume, setup a RAID1 ZFS mirror and selectively copy the video and nfo files from the former WHS drive (the single drive ZFS volume) to this new volume.

[Sorry if this is as clear as mud, but I want to move a lot of data :) ]

The question:-

After I move the data from the WHS box onto the Freenas box, and to remove the network as a bottleneck, can I then ssh onto the freenas box and use the UNIX cp command to copy data between volumes by referencing the relevant /mnt/'volume name'/'Dataset' directories?

Logically I don't see this as being a problem, but as my knowledge of Unix can best be described as dangerous, I thought I would just double check first.

thanks in advance
Reggie

 

[matthewowen01]

cp + ssh = scp

http://amath.colorado.edu/computing/software/man/scp.html

you may prefer to use rsync over ssh, it allows for partial transfers to be resumed more easily. very similar to scp but i generally use the following

rsync --partial --stats --progress -r -avz -e ssh [source] [destination]

the best part is, if something fails halfway through, you run the same command and it will figure the rest out.

 

[ReggiePerrin]

Thanks for the reply and info

Looks like rsync will do the trick, at least for the first local backup copy between freenas volumes.

I think I might have confused things by mentioning ssh, as after the data is moved onto the freenas box from the WHS server all copying will be between local volumes, but as the purpose of the post was to check that there is no problems with copying data between '/mnt/...' directories then cool

[I ask as things could go wrong with the storage pool in WHS if you tried being 'clever' :)]

 

[jim82]

Use cp or mv all you like, doesn't cause any problems. Just remember that copying or moving between datasets takes a long time, so make sure you actually copy the data from your WHS to the correct datasets/directories to begin with. If not you'll have to use the same amount of time moving and copying internally on the FreeNAS afterwards.

EDIT: You really should use rsync for this, although cp works also.

 

[ProtoSD]

I have to agree with @matt, rsync really makes a difference when copying large directory structures since it's able to resume where it left off. With cp, if something fails you have to figure out what didn't get copied. cp and mv work fine, but if a disk gets full or something fails, it can be a pain to start a partial copy.

 

[ReggiePerrin]

I had a play with rsync on a freenas VM and it's perfect for the job, so thankyou for the heads up.

It's also got me thinking about the final config of the NAS box I'm building and whether I really need a mirror'd volume.

I intend to use the box to store media, which doesn't change that often. So rather than go for a mirror'd volume, which appears to be difficult to expand when more space is required, e.g. you can't just add a single drive. I'll instead go for a striped volume with a nightly backup to copy additions/deletions/amendments to another heavily compressed striped volume cunningly called 'backup'.

So more testing on the VM, and it pays to ask questions :)

cheers

 

[matthewowen01]

I had a play with rsync on a freenas VM and it's perfect for the job, so thankyou for the heads up.

yay!

It's also got me thinking about the final config of the NAS box I'm building and whether I really need a mirror'd volume.


I intend to use the box to store media, which doesn't change that often. So rather than go for a mirror'd volume, which appears to be difficult to expand when more space is required, e.g. you can't just add a single drive.

redundancy is a good. nothing sucks more than loosing 2 TB of data over night. A mirror is much easier to expand then a than say a raidz(2) simply add a pair of drives to your existing pool. although dropping the cash for several large drives and tossing them in a raidz2 so you're done with it pays off in the end. you can pickup a 2 TB drive for $85, get 6 and you'll be setup with 8 TB, should be plenty.

I'll instead go for a striped volume with a nightly backup to copy additions/deletions/amendments to another heavily compressed striped volume cunningly called 'backup'.

media rarely compresses more than a few percent. it's already compressed using xvid, h.264 or whatever you fancied.

 

[ReggiePerrin]

This is all good stuff; you're correct, there is a lot to be said for RAID.

Cons of RAID

• Big'ish outlay to start with (family to feed, bills, teenage son needs a constant supply of beer tokens, etc )
• Drive capacity appears to double every couple of years, so maybe able to add 4Tb drives next year.
• Still need a backup [ideally remote].

Pros

Everything you list above.

'Luckily' the supplier lied to me when they had 2x2Tb drives in stock, so I have much time to mull this one over.

 

[EscapeVelocit3y]

Hi Folks,

I've done quite abit of research on rsync and how to use this tool with the various commands.

I'm have 2 volumes that I setup in the same zpool: Windows (share type is Windows) and jails (share type is Unix).

I continue to get an error regarding No Such File or Directory (2) when executing the rsync command. I'm simply trying to move files from the jail volume to the Windows volume.

I'm SSH into as root and I am able to mkdir on both the /tvshows and /TVSeries folders.

[root@freenas] /mnt# rsync --partial --stats --progress -r -av /mnt/Volume_1/jails/download_jail/mnt/media/videos/tvshows\ /mnt/Volume_1/Windows/TVSeries/
sending incremental file list
rsync: change_dir "/mnt/Volume_1/jails/download_jail/mnt/media/videos/tvshows /mnt/Volume_1/Windows/TVSeries" failed: No such file or directory (2)

Number of files: 0
Number of files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 3
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 12
Total bytes received: 12

sent 12 bytes received 12 bytes 48.00 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1053) [sender=3.0.9]
[root@freenas] /mnt#

All help is appreciated.

 

[fracai]

I think your only error is that you have '\' before a space in the paths you're trying to sync. The '\' escapes the space, treating it as part of the file path. In your case, it looks like that space shouldn't be escaped as it should just be a separator between the two paths. Note that the "No such file or directory" error references the entire path as if it were a single directory.

I have another question though, why are you transferring files from a mount point within a jail? The path ending with "videos/tvshows" is surely mounted from somewhere on your main pool. Why not just transfer directly from there?

What's the output of "zfs list"?

 

[EscapeVelocit3y]

fracai, the reason why I'm doing this transfer is because I want to consolidate my existing library with the one I download. I also don't ever want to do this manually, I just want this automated so I created a cron job. I got it working with CP command instead and then rm the directory after the copy...so far so good and the cron job also ran.

 

[fracai]

Yeah, I'm not questioning why you're doing the transfer. I'm questioning the source and destination directories that you're using. Specifically, by referencing one of them as being inside a jail, you're using a longer, more confusing path than you need to be.

Anyway, I'm glad you got something working.