To Compress or not to Compress - SLOW transfers

[Matt Tyree]

I just got my new NAS up and running. Still in a temp old PC for now, but it's running. i5, 8GB RAM, six 4TB WD Reds, and a 180GB Intel SSD for the OS.

Anyway, I started to copy my data over from my old external USB drives to the NAS and one of the operations said it was going to take 2 WEEKS (and climbing) to copy the data (1.4TB). Obviously, that can't be right over a gigabit network.

I did some testing copying a smaller chunk of that data (about 40GB). Still said it was going to take around 4 hours or more. So, I copied the same data to another PC Windows share on the network. Took about 30 minutes.
If I copy a single large file (1GB) to the NAS, it uses the full gigabit connection, roughly 111MBps, but if I'm writing 1000s of small files, it tanks.

Could the compression on the volume/data set be causing this? Or is it the RAIDZ2 across the six drives?

In my current backup scenario with the external USB drives, I backup this data nightly (the 40GB). It takes about 30 minutes to an hour, depending on what changed that day. I obviously can't go from it taking 30 minutes to 4 or more hours every night.

Any suggestions?

Thanks!

EDIT: The transfer of the 40GB is still running and now says "More than 1 day." So, 30 minutes vs 24 hours. :-/

 

[SweetAndLow]

Usb2 or usb3? Is everything using Ethernet?

Sent from my Nexus 5X using Tapatalk

 

[Ericloewe]

Could the compression on the volume/data set be causing this?

No. Any i5 will be way faster than any reasonable pool, making things faster overall.

 

[skyline65]

Can you attach the USB drive to another machine and try copying over?

Could the hard drive be corrupted?

Are the cables ok?

I have found that copying huge amount of small clipart files (50,000) took longer than a few large files however 30 mins still seems very slow for 40GB... isnt that about 20MB/s? It sounds like you have a USB2 drive for those speeds.

 

[Matt Tyree]

Here is a diagram of the setup. I'll address the other questions/suggestions when I get back.

Thanks!

 

[Matt Tyree]

I have found that copying huge amount of small clipart files (50,000) took longer than a few large files however 30 mins still seems very slow for 40GB... isnt that about 20MB/s? It sounds like you have a USB2 drive for those speeds.

True, that lots of little files totaling 40GB will take a lot more time to copy than a single 40GB file. It actually gets a LOT slower than 20MB/s, dropping down below 50KB/s sometimes. I think it's just because some of the files are only a few KB and so if each file is copied in a second, it reports a very low transfer rate.
Nevertheless, it is something I have to backup and the USB drives have been handling fine every night for over a year. So, there seems to be something up with the NAS not being able to handle it. I'm just ignorant of FreeNAS (or NAS in general) since this is my first use of it.

Can you attach the USB drive to another machine and try copying over?
Could the hard drive be corrupted?
Are the cables ok?

The external USB3.0 drives are the backup drives that the NAS is replacing. I have used them for quite a while. I don't believe there is any drive corruption and the same files (~40GB email backup) will copy to the secondary PC in 30 minutes vs over a day to the NAS. So, it doesn't appear the USB drives are corrupt or have bad cables.

In order to rule out the USB drives as the problem, I am copying the files directly from my PC to the NAS and then to the secondary PC to see what kind of difference there is. I'll post the results once that's done.

No. Any i5 will be way faster than any reasonable pool, making things faster overall.

I believe you. Just seems counter-intuitive. Compressing the files as they are copied would seem to be adding a processing step to the copy operation thereby slowing it down. Not sure how in the world it actually makes things faster. :-D

Thanks for the help, guys! I REALLY appreciate it. I'm really hoping I haven't spent $900 on all these drives for nothing. :-/

 

[skyline65]

What are these very small files? Is every backup unique or could you just sync changes? With my Mac I use Synk Pro which will only back up changed files from my Work HD. The initial sync takes a while but subsequent syncs are very quick.

 

[anodos]

Are you copying via samba? It might be a permissions issue (you can check /var/log/samba4/log.smbd for NT_STATUS_ACCESS_DENIED messages). Try copying using winscp and see if the problem goes away. You can speed up transfers via SCP/SFTP if you use a crap cipher like arcfour128.

On the other hand, operations with lots of small files can be a pain point for Samba. If you look at 'top' output you'll probably see that you have a single thread pegged at around 100%. There are some operations that samba performs synchronously and this can harm performance when dealing with tons of tiny files. For instance, see discussion here: http://marc.info/?l=samba&m=146175662822813&w=2

There are some things you can do to improve performance with lots of small files. Jeremy Allison outlines one strategy here: http://marc.info/?l=samba&m=139336252926228&w=2

If you want to try out JRA's advice, please do it in the following way:
1) via WebGUI create new dataset with "Case Sensitivity" set to "Insensitive"
2) via WebGUI add auxiliary parameter to samba share "case sensitive = true"

This will significantly improve samba performance with small files (but may not improve speeds for initial transfer). Don't do the above steps if Case Sensitivity is important for you.

 

[Ericloewe]

I believe you. Just seems counter-intuitive. Compressing the files as they are copied would seem to be adding a processing step to the copy operation thereby slowing it down. Not sure how in the world it actually makes things faster. :-D

It reduces the amount of data that needs to be written to disk. In particular, small enough files can be packed into the same block as the file metadata, saving a seek operation.

The CPU just has to be faster at compressing/decompressing than the disks' write/read speed.

 

[wblock]

How many files are in the target directory?

 

[Matt Tyree]

Okay, here are my results after four tests copying from my Windows 10 PC to the NAS and a Windows 7 PC.

You can see on the zip copy operation the connection to the Windows PC is actually weaker from the length of ethernet and the amount of switches in the mix. Only gets 50MB/s whereas the NAS gets 112MB/s.

What are these very small files? Is every backup unique or could you just sync changes? With my Mac I use Synk Pro which will only back up changed files from my Work HD. The initial sync takes a while but subsequent syncs are very quick.

I use SyncBackPro for backups. The main trouble files that I have run into are the multitude of small files from my email backup.
You are correct, I do have it set up to do a "Fast Backup" (SyncBack term) each night. It backs up to a folder named according to the day it was backed up. It only backs up whatever files changed that day. So each folder maintains essentially an almost identical backup other than the files changed on that day. So, yes, these nightly backup operations are much faster than copying the full 40GB. Unfortunately, that doesn't help me in trying to move those backups to the NAS.

Are you copying via samba? It might be a permissions issue (you can check /var/log/samba4/log.smbd for NT_STATUS_ACCESS_DENIED messages).

Ah, you're right. There are NT_ACCESS_DENIED messages in the log. I also see "matchname: host name/name mismatch: 192.168.1.100 != (NULL)" and "matchname failed on 192.168.1.100" quite a bit. 192.168.1.100 is my main PC I am trying to copy from.
Any suggestions on what I might have wrong in the permissions? I am connecting with the username and password I set up in the FreeNAS gui. I made a group and gave it ownership of the dataset and then made that user part of that group.
I followed this video: https://www.youtube.com/watch?v=RxggaE935PM

 

[SweetAndLow]

You can ignore the name mismatch error or turn off hostname lookups in smb.

This really looks like a small file problem. Try testing the advice from previous posts.

Sent from my Nexus 5X using Tapatalk

 

[Matt Tyree]

You can ignore the name mismatch error or turn off hostname lookups in smb.

What about the ../source3/smbd/service.c:560(make_connection_snum) create_connection_session_info failed: NT_ACCESS_DENIED" messages? Ignore those, too?

 

[Matt Tyree]

If you want to try out JRA's advice, please do it in the following way:
1) via WebGUI create new dataset with "Case Sensitivity" set to "Insensitive"
2) via WebGUI add auxiliary parameter to samba share "case sensitive = true"

This will significantly improve samba performance with small files (but may not improve speeds for initial transfer). Don't do the above steps if Case Sensitivity is important for you.

What exactly does case insensitivity do? If I copy a file with upper and lowercase letters, does it make it all lowercase? Or is it a comparison thing?

 

[skyline65]

Could you not zip the folder full of small files... then copy one big file over and then unzip once on the Freenas server? Would Freenas throw a wobbly?

 

[anodos]

What exactly does case insensitivity do? If I copy a file with upper and lowercase letters, does it make it all lowercase? Or is it a comparison thing?

Comparison thing. It will be case-insensitive, but case preserving (like a Windows server). For instance, if you have the folder "Doc", you can't create a new folder "doc", but it will be displayed how you create it.

 

[anodos]

What about the ../source3/smbd/service.c:560(make_connection_snum) create_connection_session_info failed: NT_ACCESS_DENIED" messages? Ignore those, too?

It depends on how many of these messages you're getting, and which client is causing them. Ideally, you shouldn't be getting "access denied messages". This can indicate that you're client is choking because it can't write where it wants to write. Since the backup job is completing (???), I assume that the client isn't choking on permissions.

 

[Matt Tyree]

Could you not zip the folder full of small files... then copy one big file over and then unzip once on the Freenas server? Would Freenas throw a wobbly?

Yep, good point. I definitely could do that to get past this particular hurdle. And that's a great way to handle this. However, it is indicative of a larger problem that I may hit one day during normal, auto backup operations. Say, I moved a folder full of lots of small files and the backup software wanted to copy the entire folder to the backup. I wouldn't know it was a problem until I noticed the backup operation was taking a full day or more.

It will be case-insensitive, but case preserving (like a Windows server).

Excellent. I'm all for making it behave as much like a typical Windows share as possible. I'm testing a copy operation with the case set to insensitive on a dataset and the share set to sensitive as you indicated. We'll see how much it improves things. Thanks for the tip!

It depends on how many of these messages you're getting, and which client is causing them. Ideally, you shouldn't be getting "access denied messages". This can indicate that you're client is choking because it can't write where it wants to write. Since the backup job is completing (???), I assume that the client isn't choking on permissions.

I agree, I definitely don't want anything to slow operations down. As far as I know I am connected and copying, changing, and deleting files without issue. I don't know where the "access denied" errors are coming from. My ignorance of Linux starts to become quite apparent in these situations. How do I go about determining where the errors are coming from?

Thanks again for all the help, guys! The FreeNAS community has been very good to me during my fledgling phase. :-D

PS: Is there a way to multi-quote on these forums?

 

[Matt Tyree]

Well, anodos, it appears the case-sensitivity trick did the job. Just copied over the same files in less than 40 minutes.
The only hiccup was at about 95% it popped up with a message asking if I wanted to copy the current file without its properties. I clicked yes, and the same message came up for the next 10 files or so. So, I checked the box to apply that decision to all files. Not sure why it got right to the end like that and started up with those messages. Not sure what impact that will have.

Thanks!

 

[anodos]

Well, anodos, it appears the case-sensitivity trick did the job. Just copied over the same files in less than 40 minutes.
The only hiccup was at about 95% it popped up with a message asking if I wanted to copy the current file without its properties. I clicked yes, and the same message came up for the next 10 files or so. So, I checked the box to apply that decision to all files. Not sure why it got right to the end like that and started up with those messages. Not sure what impact that will have.

Thanks!

Great! Glad to hear it worked. Case-sensitivity is one of those pain-points in Unix / Windows compatibility. Having to do repeated look-ups is expensive (especially when operations are done synchronously). It's not very noticeable when files are big, but it can become quite painful if you have lots of small ones.

In the future if you're seeking help with this share on the forums, make sure you indicate that you've made case sensitivity changes. I'm not aware of any problems being caused by these changes (it's a fairly typical samba workaround for performance problems with small files), but it'd be a useful datapoint if you do have problems.