I can only hit 40-45MB/s per host copying to FreeNAS

Status
Not open for further replies.

Ray Milyard

Patron
Joined
Aug 8, 2014
Messages
262
12 MB/s or 12 Mb/s?

If it's 12 MB/s it's pretty much exactly the throughput of a 100 Mb/s link (12.5 MB/s) so I assume some part of the chain in your network is degraded to 100M because of poor quality cables for example (you need CAT5E or better for a gigabit network).

It's 12Mb's. I was before last month or so of updates 120mb. Nothing else has changed.
 

fullspeed

Contributor
Joined
Mar 6, 2015
Messages
147
It's 12Mb's. I was before last month or so of updates 120mb. Nothing else has changed.

Probably a different issue with its own set of circumstances. I can get 400MB/s write through CIFS but only if I transfer from 10 different hosts concurrently.

I'm yet to find anyone that has had this problem.
 

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,472

fullspeed

Contributor
Joined
Mar 6, 2015
Messages
147
I've barely had time to play with this, but i'm narrowing it down in my spare time.. could use some help.

So I open up my windows file server and do a richcopy(this is a multithreaded file transferring tool) and it hits the 50MB/s cap, open up another richcopy and it won't break 50MB/s in parallel. I then drag and drop an ISO and it starts transferring at 50MB/s for a total of 100MB/s in parallel. When I open up a 2nd drag and drop cifs transfer they load balance and it stays at 100MB/s. So it's not actually limited per host on a network level, but perhaps on a stream level.. cliffnotes:

Fileserver1 Richcopy ~ 50MB/s total
Fileserver1 Richcopy2 ~ 50MB/s total (I cancel this)
Fileserver1 Explorer Copy ~ 100MB/s total
Fileserver1 Explorer2 Copy ~ 100MB/s total

TOP screenshots during this process, please let me know if anyone needs more specific info.

Richcopy transfer only:
upload_2015-3-20_11-40-54.png


Richcopy+Windows Explorer Copy
upload_2015-3-20_11-41-39.png


Richcopy+2 Explorer copies (bottlenecked)
upload_2015-3-20_11-42-4.png
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I helped someone else a year or two ago with a complaint of slow file copy. He was doing a program like richcopy that claimed to be faster, blah blah blah.

What did we find after 4 hours of intense diagnosis? Using those programs doesn't make it faster. I'm guessing that Explorer has some kind of special sauce that isn't open for 3rd party software that makes things work faster.

I, personally, have never bought the koolaid that those programs are faster. Some do verify and parallel copy, which can be useful in specific situations. But I never thought or expected that a program like Richcopy would make single file transfers faster. The limiting factor has never been that the Explorer code was too slow or wasn't multi-threaded.
 

fullspeed

Contributor
Joined
Mar 6, 2015
Messages
147
I helped someone else a year or two ago with a complaint of slow file copy. He was doing a program like richcopy that claimed to be faster, blah blah blah.

What did we find after 4 hours of intense diagnosis? Using those programs doesn't make it faster. I'm guessing that Explorer has some kind of special sauce that isn't open for 3rd party software that makes things work faster.

I, personally, have never bought the koolaid that those programs are faster. Some do verify and parallel copy, which can be useful in specific situations. But I never thought or expected that a program like Richcopy would make single file transfers faster. The limiting factor has never been that the Explorer code was too slow or wasn't multi-threaded.

Yeah I'm with you, I only use Richcopy is very specific situations.. such as when I had to copy 70 million small files from hundreds of thousands of small directories, in that case it actually did make a big difference but generally it does not.

In this particular situation I was just trying to narrow down where the bottleneck was coming from and I definitely have, I'm hoping someone can explain to me why windows explorer will never get over say, 45MB/s (even if I open multiple instances of them) , Richcopy will never get over 45MB/s (even if I open multiple instances of them) ..but if I run them both in parallel I get 90MB/s total.

Also if I use different hosts in parallel i'll get 300-400MB/s but never in more then 45MB/s in a single stream..

Edit, here is a screenshot showing the "Steps". You can see each time a server starts a transfer and immediately hitting some invisible cap.

upload_2015-3-23_10-57-25.png
 
Last edited:

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Are you still using LAGG? I don't recommend LAGG unless you have 10+ machines, and LAGG isn't supposed to improve single-machine performance (not a problem here obviously) but it can create other problems if not configured properly and using supported hardware.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Are the servers that are connecting in parallel, but only getting 45MB/s per stream, running anti-virus and if so which one/version?
 

fullspeed

Contributor
Joined
Mar 6, 2015
Messages
147
Are the servers that are connecting in parallel, but only getting 45MB/s per stream, running anti-virus and if so which one/version?

No antivirus, they are all on 10 gig links and transfer between each other at 300-350MB/s over CIFS.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Can you post a debug file for your system? System -> Advanced -> Save Debug
 

Apollo

Wizard
Joined
Jun 13, 2013
Messages
1,450
I've barely had time to play with this, but i'm narrowing it down in my spare time.. could use some help.

So I open up my windows file server and do a richcopy(this is a multithreaded file transferring tool) and it hits the 50MB/s cap, open up another richcopy and it won't break 50MB/s in parallel. I then drag and drop an ISO and it starts transferring at 50MB/s for a total of 100MB/s in parallel. When I open up a 2nd drag and drop cifs transfer they load balance and it stays at 100MB/s. So it's not actually limited per host on a network level, but perhaps on a stream level.. cliffnotes:

Fileserver1 Richcopy ~ 50MB/s total
Fileserver1 Richcopy2 ~ 50MB/s total (I cancel this)
Fileserver1 Explorer Copy ~ 100MB/s total
Fileserver1 Explorer2 Copy ~ 100MB/s total

TOP screenshots during this process, please let me know if anyone needs more specific info.

Richcopy transfer only:
View attachment 7107

Richcopy+Windows Explorer Copy
View attachment 7108

Richcopy+2 Explorer copies (bottlenecked)
View attachment 7109
Looking at your "top" results, smbd is maxed out. It seems to me your limitation is your poor core performance on your Freenas machine.
You will be able to scale up with multiple links as long as you have enough unused thread.
It seems to me your processor may not be changing clock and would stick to it's low frequency setting.
I would look at your BIOS setting to see if your core is allowing dynamic frequency adjustment or if it is rather lock to be at minimal performance.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Looking at your "top" results, smbd is maxed out. It seems to me your limitation is your poor core performance on your Freenas machine.
You will be able to scale up with multiple links as long as you have enough unused thread.
It seems to me your processor may not be changing clock and would stick to it's low frequency setting.
I would look at your BIOS setting to see if your core is allowing dynamic frequency adjustment or if it is rather lock to be at minimal performance.

Hehe. That's why I wanted to see the debug. The sysctls would give some insight as to what is going on.
 

fullspeed

Contributor
Joined
Mar 6, 2015
Messages
147
Sure, see attached.

I tried every BIOS setting I have on/off and nothing increased performance.. although I did notice a slight drop in performance with turbo + hyperthreading off.

Is samba really getting pegged? I've tried so many sysctl changes that you think would lower the utilization and give me a few megs at least but always same speed.
 

Attachments

  • sysctl_hw.txt
    5.7 KB · Views: 601
  • cpuid.txt
    4.7 KB · Views: 513
  • dmidecode.txt
    21.3 KB · Views: 586

titan_rw

Guru
Joined
Sep 1, 2012
Messages
586
Try 'top -SH'. That'll show individual threads of each process instead of bundling it up into the process. Any thread can never use more than 100% (of a core), so if any threads are at or near 100% you're single thread limited.
 

fullspeed

Contributor
Joined
Mar 6, 2015
Messages
147
Try 'top -SH'. That'll show individual threads of each process instead of bundling it up into the process. Any thread can never use more than 100% (of a core), so if any threads are at or near 100% you're single thread limited.

looks like four CIFS transfers right now, dba copying over sql backups.

upload_2015-3-24_13-36-8.png

upload_2015-3-24_13-37-4.png
 

fullspeed

Contributor
Joined
Mar 6, 2015
Messages
147
So just to update this thread, I managed to get 250-300MB/s via FTP and based on what I see in top it's an issue with samba not multithreading and pegging my cpu cores (as a few people have hinted at in this thread)

Now that I know this is the issue are there any specific sysctl/loader options to alleviate this? remember i've tried all the BIOS options, only looking at Freenas tuning at this point.

Other than that.. do I just have to get a more powerful rig? I might just plug the storage into a beefy R910 just to prove a point.

Thanks for all the helpful info btw
 

fullspeed

Contributor
Joined
Mar 6, 2015
Messages
147
Over two years later and this appears to have been fixed by upgrading to FreeNAS 11.1, I wish I knew what corrected this behavior.

This has caused me untold grief over the years, Time to crack a bottle of champagne!!

upload_2017-12-28_11-48-27.png
 
Last edited by a moderator:
Status
Not open for further replies.
Top