Moving files

[Bever]

I have been playing around with a fresh FreeNAS 8.04 setup, and am trying to optimize performance. One thing I noticed is that it takes a long time simply to move a bunch of files (67GB in this case) from one directory to another on the same volume. The data is on a RAIDZ volume consisting of three 2TB disks.

Now I always thought that moving files on a file system was simply a matter of changing some pointers around, so it's unclear to me why it takes such a long time. Does it have something to do with the way ZFS (of RAIDZ) works? Is there a way to speed up this process?

 

[ProtoSD]

ZFS is Copy On Write (COW), so it actually rewrites everything instead of updating pointers like other filesystems. You didn't tell us much about your hardware so it's difficult to say what might make things faster. Usually more RAM is a plus, but maybe not in your case.

 

[sjieke]

Could it be that you are moving files between 2 different datasets? If that's the case then ZFS needs to copy every bit of data and isn't able to "just change pointers". In this context dataset are just like partitions on a windows disk. If you move files from your "C:" partition to your "D:" partition a complete copy is needed, even if it is the same fisical disk.

 

[sjieke]

@protosd

I should verify it, but I think that moving a file in the same dataset does not copy the entire file. I don't know how ZFS works, but from an engineering point of view, I would find it very strange that an 'enterprise level' filesystem like ZFS (with so much nice features) wouldn't be as efficient as old filesystems in moving files (if staying in the same dataset).

 

[Bever]

Wow, this is some really fast support guys :) Thanks! protosd: I am running FreeNAS VM on a core i3 ESXi 5 with 8GB RAM. Data is on a RAIDZ of 3 2TB WD's which are mapped to the VM as RDM. I don't think hardware is the bottleneck here. If I watch performance monitor in the VMware client, I see all three disks are doing an average of 225 iops, which looks fine to me.

So yes, it's probably by design, as you say. I'll probably have to learn to put my data on the correct spot from the start :)

 

[Bever]

sjieke: data is moved within the same dataset. Which made me wonder the very same thing as you just did :)

 

[sjieke]

I'm still not convinced... If you look at the explenation of COW (http://www.oracle.com/technetwork/s.../oraclesolariszfsstoragemanagement-360232.pdf and http://standalone.iteye.com/blog/487695) I conclude that it only re-writes data if you actually changed something. Moving a file inside a dataset doesn't change any data (I think).

I currently can't access my NAS, otherwise I would be testing it now.
Just to make sure, you do move the files from within an SSH session? And not through CIFS or NFS or ...?

If you do have snapshots on the dataset, maybe they are the cause of the COW to take effect?

I'm just dumping here what comes to my mind to try and understand the inner workings of ZFS better, so if I am completley hallucinating please correct all my misunderstandings :)

 

[Bever]

I moved the files from within an SSH session:

[bever@freenas /]$ time mv /mnt/Data/Movies/ /mnt/Data/Video/

real 70m28.197s
user 0m0.166s
sys 0m36.983s
[bever@freenas /]$ du -hs /mnt/Data/Video/
199G /mnt/Data/Video/

Now I must say that I'm also copying files from my old NAS to this new one at the same time, which will explain some of the delay. But not when moving would just be a case of changing pointers.

 

[sjieke]

Protosd will probably be correct, but I still can't understand it. If moving copies the file, even if it is moved to another location inside the same dataset then renaming a file should also be slow? Because if you want to rename a file from the commandline you also use the 'mv' command, so it should act the same way. And I know that renaming a file is faster than copying it...

The more I think about it, the less I understand it :S

 

[ProtoSD]

Sjieke,

I'm no expert, that was just my understanding of it, so I may be wrong. I know until recently moving file(s) between datasets was slower, but *something* I changed while playing with some tuning variables seems to have improved that a little. Now if I could only cut my scrub time down!

What you are describing makes perfect sense, so now I'm wondering also.

 

[Erwin]

I moved the files from within an SSH session:

[bever@freenas /]$ time mv /mnt/Data/Movies/ /mnt/Data/Video/

real 70m28.197s
user 0m0.166s
sys 0m36.983s
[bever@freenas /]$ du -hs /mnt/Data/Video/
199G /mnt/Data/Video/

Now I must say that I'm also copying files from my old NAS to this new one at the same time, which will explain some of the delay. But not when moving would just be a case of changing pointers.

Although both directories /mnt/Data/Movies/ /mnt/Data/Video/ resides on the same Zpool, they represent different ZFS datasets, or in other words different ZFS file systems. So a Move will end up in a Write on the target dataset/file system...
This is quite normal.
Try a move in /mnt/Data/Video/ to a subdirectory in this dataset. Then it will do just a change of the directory entry.

erwin

 

[Bever]

Hi Erwin, I think you are correct:

[bever@freenas /]$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
Data 439G 3.14T 80.5G /mnt/Data
Data/Install 28.0K 3.14T 28.0K /mnt/Data/Install
Data/Music 30.0K 3.14T 30.0K /mnt/Data/Music
Data/Pictures 13.5G 3.14T 13.5G /mnt/Data/Pictures
Data/Video 345G 3.14T 345G /mnt/Data/Video

I assumed it was the same dataset, but it is not.