Moving Files within the Same Share

[Zythyr]

I was successfully able to transfer all my data from my external hard drive to my freenas. Now I want to properly organize all my files and folder on my freenas server: this includes moving folders around within the same volume...

I have created a CFIS share for my freenas server so I can easily browse through my directories... When I try to "Move" files and folders within the same share, it takes a really long time. Why is this happening? The source and target are on the same freeans volume.

For example, I am trying move the Movies folder to Organized folder.
Source: //Share1//Storage/Unorganized/Movies
Destination: //Share1/Storage/Organized/

When I move the "Movies" folder by dragging it into the Organized folder, it takes a very very long time.

 

[cyberjock]

Are they in the same share? If you are mapping one folder as M: and the other as O:, when you "move" a file its literally reading from the M drive and writing to the O drive. If its the same share it should be pretty fast.

WHat's your hardware specs for the server? Is this ZFS or UFS? Is this CIFS or NFS?

 

[Zythyr]

Are they in the same share? If you are mapping one folder as M: and the other as O:, when you "move" a file its literally reading from the M drive and writing to the O drive. If its the same share it should be pretty fast.

WHat's your hardware specs for the server? Is this ZFS or UFS? Is this CIFS or NFS?

Yes, the folders are on the same CIFS share. I have mapped the share to only one drive letter (ex: W:).
The volume is a single hard drive 3TB ZFS (parity?)
The specs on teh hardware are:
Pentium 4 3.40 Ghz
2GB RAM
x86 Freenas v 9.1.1

I don't understand why its slow. The transfer is occurring on the same volume, and on the same share... I am not using "Copy/Paste". I am only doing "Drag and Drop (Move)".

I know many people say for ZFS, more RAM the better, but in my class, the RAM shouldn't matter that much... It is just a "move" function I am doing...

Edit:
One thing I noticed...
When I "Move" to a location that is on a different dataset, it takes a long time, but when I move to somehwere within the same dataset, then its almost instantaneous

For example:
Source: /Share1/Dataset1/Movies/
Destination: /Share1/Dataset2/
When I copy the Movies folder from Dataset1 to Dataset2, I have the issue of the slow transfer speed.

 

[paleoN]

Edit:
One thing I noticed...
When I "Move" to a location that is on a different dataset, it takes a long time, but when I move to somehwere within the same dataset, then its almost instantaneous

Datasets are separate filesystems. A "move" to a different dataset is actually a copy to the new dataset followed by a delete of the original.

 

[cyberjock]

Yes, the folders are on the same CIFS share. I have mapped the share to only one drive letter (ex: W:).
The volume is a single hard drive 3TB ZFS (parity?)
The specs on teh hardware are:
Pentium 4 3.40 Ghz
2GB RAM
x86 Freenas v 9.1.1

I don't understand why its slow. The transfer is occurring on the same volume, and on the same share... I am not using "Copy/Paste". I am only doing "Drag and Drop (Move)".

I know many people say for ZFS, more RAM the better, but in my class, the RAM shouldn't matter that much... It is just a "move" function I am doing...

Edit:
One thing I noticed...
When I "Move" to a location that is on a different dataset, it takes a long time, but when I move to somehwere within the same dataset, then its almost instantaneous

For example:
Source: /Share1/Dataset1/Movies/
Destination: /Share1/Dataset2/
When I copy the Movies folder from Dataset1 to Dataset2, I have the issue of the slow transfer speed.

And if you check out the manual it says you should use 8GB minimum with ZFS and that if you have 4GB or less prefetch is disabled which significantly hurt performance. Per the manual, since you have 2GB of RAM you should be able to get by with UFS.

 

[Zythyr]

Datasets are separate filesystems. A "move" to a different dataset is actually a copy to the new dataset followed by a delete of the original.

Does this apply for sub-datasets? Meaning, datasets within another dataset?

Is there any way to speed up the "Move/Copy" rate from one dataset to another? Am I experiencing slow transfer because of only 2GB RAM? If so, will the speed be significantly higher with 4GB RAM?

What are the benefits of having datasets instead of regular folders? I was panning to use datasets to separate my files from backup, storage, dump of unorganized files, public share (in home share with family), etc... How do I address the issue of my freenas structure since transfer from one dataset to another is slow?

 

[fracai]

A dataset is a dataset. Where it shows in the filesystem is just a matter of the mount point.

 

[warri]

What are the benefits of having datasets instead of regular folders? I was panning to use datasets to separate my files from backup, storage, dump of unorganized files, public share (in home share with family), etc... How do I address the issue of my freenas structure since transfer from one dataset to another is slow?

Snapshots and replication for example. But how you use datasets depends on your use case. If you often move from one set to another, maybe you should put both as folders in a single dataset. If you want to do separate backups of your data sets, it makes sense to split them, etc.

 

[Zythyr]

Is there a performance difference for moving/transferring/copying files over command line (shell) vs. GUI method using CIFS share method?

Using cp, mv, and rsync methods in command line, I have gettigns speeds of 50-100MB/s when trasnfering/moving/copying files between two different dataset or within the same dataset.

Using CIFS based method by GUI browsing through the directories in Windows explorer, I am getting transfer rates less than 1MB/s.

Why is GUI method slower? Is there solution to getting faster transfer rates using GUI based method?

 

[cyberjock]

If you do from the CLI you'll be limited to pool speeds. If you do it from CIFS, you'll be limited to whatever speeds you get over CIFS/network/etc.

 

[Zythyr]

If you do from the CLI you'll be limited to pool speeds. If you do it from CIFS, you'll be limited to whatever speeds you get over CIFS/network/etc.

Over CIFS, why am I limited to network speed? If I am organizing file within the freenas volume itself (server), there should be no file transfer between the client (my Widnows PC) and the server... The client only sends the commands to the server on what to do on the server itself???

 

[cyberjock]

I'm referring to moving files from one pool/dataset to another, since that's what started this thread.

Assuming all files on a pool/dataset are moved to the same pool/dataset, then there is negligible speed difference.

 

[Zythyr]

I'm referring to moving files from one pool/dataset to another, since that's what started this thread... If you are done with that topic you should have started a new thread.

Assuming all files on a pool/dataset are moved to the same pool/dataset, then there is negligible speed difference.

I understand that transfer of file between two different dataset are slower and are limited by pool speed.

Sorry I didn't clarify my previous question, and I think you were misunderstood.
Given the following structure:
ZFS Pool (Volume): ABCD
Datasets: Backup, Downloads, Storage
mnt/ABCD/Backup
mnt/ABCD/Downloads
mnt/ABCD/Storage

Using the command line shell in freeNAS's web interface, transfer of a file inside Downloads to Storage gives rate of around 50MB/s using the rsync command.

When I mount ABCD using CIFS share, and browse through the directories using Windows Explorer, and then drag/copy a file from Downloads to Storage I get a transfer rate less than 1MB/s.

 

[ewhac]

When you move a file within a dataset/filesystem, you're just altering where the file's data blocks link in to the filesystem's hierarchy, so it completes very quickly. When you move a file from one dataset/filesystem to another, you're copying data out of one dataset and writing it to another (then deleting the original), which takes much much longer.

Sorry I didn't clarify my previous question, and I think you were misunderstood.
Given the following structure:
ZFS Pool (Volume): ABCD
Datasets: Backup, Downloads, Storage
mnt/ABCD/Backup
mnt/ABCD/Downloads
mnt/ABCD/Storage

Using the command line shell in freeNAS's web interface, transfer of a file inside Downloads to Storage gives rate of around 50MB/s using the rsync command.

Yes. You're copying a bunch of data, but it's all happening completely inside the FreeNAS server through local memory over the local SATA bus(es). Also, 'rsync' is pretty well tuned for copying large amounts of data.

When I mount ABCD using CIFS share, and browse through the directories using Windows Explorer, and then drag/copy a file from Downloads to Storage I get a transfer rate less than 1MB/s.

You're still copying a bunch of data, except that now the copying "agent" is on the Windows side, so you're reading the file data across the network to the Windows client, who immediately writes it back across the network to the new location. And CIFS has never been noted for speed, and Windows Explorer even less so.

 

[cyberjock]

I'll never understand why someone would do an rsync from a ZFS source to a ZFS destination. It's like doing the checks twice. Add to that the performance killer that rsync can be and I see no "added value" if you could have just done a cp command and been done with it. I get over 200MB/sec when I do cp, but with rsync I'm lucky to hit 40MB/sec.

 

[ewhac]

This is getting off-topic, but there are little syntactical goodies in the 'rsync' command line that I've grown to appreciate. It also avoids re-copying files that are already present in the destination (and no, cp -u doesn't handle all the cases 'rsync' does). And rsync -a --delete --link-dest= is pretty nice for doing incremental backups.

Never really timed it against 'cp', though...

EDIT: Well, whaddya know. Learn something new every day...

 

[cyberjock]

This is getting off-topic, but there are little syntactical goodies in the 'rsync' command line that I've grown to appreciate. It also avoids re-copying files that are already present in the destination (and no, cp -u doesn't handle all the cases 'rsync' does). And rsync -a --delete --link-dest= is pretty nice for doing incremental backups.

Never really timed it against 'cp', though...

EDIT: Well, whaddya know. Learn something new every day...

Nice link. I didn't ever do a benchmark comparison, but if you think about how cp works and how rsync works, it just "made sense" to me. Most people do rsync in lieu of cp and don't intend to avoid recopying files(which is what I just don't get at all). For those that want to compare changes between folders, rsync works, but again, isn't record setting speeds. For me, it just made logical sense that rsync would be slow because of how it works internally. I didn't feel it necessary to "prove" my opinion as I saw it first-hand(granted in only one situation) and I didn't ever mention my opinion of rsync before.

I rsync from my cell phone to my server as a backup. Works great for that as my wifi would be a big limiting factor otherwise.
-----

A few months ago I chatted with someone in IRC that was doing copying from one local pool to another local pool(doing disk upgrades). He was using rsync and he estimated it at taking most of the weekend and Monday to complete. He wasn't happy because Monday morning he'd have to stop the rsync and wouldn't be able to continue it until the following weekend, when the whole thing would start over and would take the same amount of time since it was CPU bound. His speeds were something around 30-40MB/sec.

I told him to cancel the rsync and just do a cp or mv. He was skeptical since he had already had 14 hours of data copying done and like 36+ to go. He did some googling and decided that I might be right. He took the risk and decided to delete the data he had copied and recopy the pool from scratch. He started the cp command and thanks to zpool iostat he saw that he was now getting over 500MB/sec! That's a more than 10 fold increase!

I can see how rsyncing over a network for backup purposes is very useful, but when it comes to local copies the best answer is less clear and requires you to know what you are doing if you want maximum performance. Naturally, deleting your backup and doing a whole fresh cp is not smart as you don't have a complete backup until the cp finishes.

Just have to know your limitations and work with them. :P

 

[Zythyr]

Thank you guys for a very insightful reply. They were very helpful. I have one last questions:

I currently have all my content in the Storage dataset of my ABCD volume: mnt/ABCD/Storage
I want to move everything inside mnt/ABCD/Storage to mnt/ABCD.

Since the transfer will be occurring from one dataset to another, this will take a very very long time. How do I execute this transfer without GUI? The reason I ask is that I can do the "drag/drop" using my the GUI on my laptop, but this will take a long time. If I turn off my laptop, the transfer will also be aborted.

 

[warri]

That's a bit off topic here, you should familiarize yourself with basic linux commands

mv(1) should do the trick:

Code:

mv /mnt/ABCD/Storage/* /mnt/ABCD/

 

[Zythyr]

That's a bit off topic here, you should familiarize yourself with basic linux commands

mv(1) should do the trick:

Code:

mv /mnt/ABCD/Storage/* /mnt/ABCD/

Should I connect my FreeNAS to a monitor and execute that command directly on there? Or should I login to the web control panel and open the Shell and put the command in there? If I put the command in the web control panel shell, and if I close the browser, won't it abort the transfer?