File transfers to freenas drop midway (10gbe)

[shaf]

I can upload around 6Gb of data at full speed before it bottles down to 1gb speeds.

 

[shaf]

Any ideas please as I am quite stuck now.

 

[Johnnie Black]

I can upload around 6Gb of data at full speed before it bottles down to 1gb speeds.

That would suggest to me that after filling the RAM cache, which I believe defaults to around 12% of total RAM, you're limited by the pool write speed, and if these are your pools it looks about right:

4x6TB WD Red (2 disk redundancy)
2x6TB WD Red (mirror)

 

[shaf]

That would suggest to me that after filling the RAM cache, which I believe defaults to around 12% of total RAM, you're limited by the pool write speed, and if these are your pools it looks about right:

So what do I need to do to get better performance?

 

[Johnnie Black]

A faster pool, here's an example I posted before writing to both my HDD and SSD pools, details on signature:

HDD Pool:

SSD Pool:

 

[shaf]

A faster pool, here's an example I posted before writing to both my HDD and SSD pools, details on signature:

Ok cool.

So as a quick test if were to put 2xSSDs in raid0 into my setup, I should see significant increase in performance even after it has used up all the RAM cache?

 

[Johnnie Black]

So as a quick test if were to put 2xSSDs in raid0 into my setup, I should see significant increase in performance even after it has used up all the RAM cache?

Yes, if they are fast SSDs, cheap TLC SSDs can sustain writes of about 100MB/s each, so not that much faster than your current pool.

 

[shaf]

Yes, if they are fast SSDs, cheap TLC SSDs can sustain writes of about 100MB/s each, so not that much faster than your current pool.

Something like these should be fine right?

Samsung 500 GB 860 EVO

 

[Johnnie Black]

Something like these should be fine right?

Yes, those are some of the fastest ones, you should get >500MB/s

 

[shaf]

Yes, those are some of the fastest ones, you should get >500MB/s

Thanks @Johnnie Black, your post explains the issue I was having.

Hopefully once the SSDs are installed I should see the 10Gb performance I am expecting.

 

[Spearfoot]

@shaf, are you using jumbo frames in your new configuration? I couldn't get 10G line speeds w/iperf until I enabled jumbo frames:

https://forums.freenas.org/index.ph...ceiving-network-data.53722/page-2#post-378891

Even now, after tuning and using jumbo frames so that I get >9GB iperf test results -- I never see anything close to 10G transfer rates with SMB, rsync, or replication. Sigh...

 

[shaf]

@Spearfoot, yes I am using jumbo frames in my new configurations.
I enabled this in the switches and the network cards. Also, enabled this in freenas by adding mtu 9000 in the interface options.

 

[Spearfoot]

@Spearfoot, yes I am using jumbo frames in my new configurations.
I enabled this in the switches and the network cards. Also, enabled this in freenas by adding mtu 9000 in the interface options.

Very good. Then it may be that, as suggested by others above, your bottleneck is the speed of your pool(s) and not your network.

 

[shaf]

So bought 3x500GB Samsung 860 EVO drives and put them into a stripe pool for testing. Happy to say that the 'issue' has been resolved. I'm getting aroung 400-500MB/s speeds.

As soon as it goes past the cache it doesn't flow as smoothly and is a little choppy but it doesn't drop thankfully.
Now I'm trying to get to configure for higher speeds if possible.

Thanks for your help everyone, especially to @Johnnie Black for pointing me into the right direction.