FreeNAS 11.1-U6 10Gig reads, writes start at 10Gig and then drop drastically

Status
Not open for further replies.

Frederico

Dabbler
Joined
Aug 28, 2013
Messages
12
Bought two Intel X520-DA1 cards with Yottamark sticker, verified on Yottamark's website, all good.

Connected using DAC between my Windows PC and my FreeNAS.

Configured 9014 MTU on both cards.

When copying a 4GB file from my NAS to my RAMDisk, I get full speed.
nas to ramdisk.jpg


When copying the same 4GB file from my RAMDisk to my NAS, I get full speed until about halfway, then it drastically drops to 130ish MB/s.
ramdisk to nas.jpg


NAS Specs:
Intel(R) Xeon(R) CPU E5-1603 v3 @ 2.80GHz
16GB ECC RAM
2x4TB drives in mirrored mode
zpool status.JPG


Any tips?
 
Joined
May 10, 2017
Messages
838
That's expect, first few GB are cached to RAM then you're limited by your pool performance, you'd need more disks or better yet some SSDs to get increased sustained write speeds.

Read I'm guessing is being done from the ARC, hence the speed.
 

scwst

Explorer
Joined
Sep 23, 2016
Messages
59
If I understand this correctly, we're roughly looking at these speeds:
  • Hard drive: 150 MB/sec
  • SATA II interface: 300 MB/sec
  • SATA III interface: 600 MB/sec
  • NVMe PCIe interface: 2500 MB/sec
  • 1 Gb ethernet: 120 MB/sec
  • 10 Gb ethernet: 1200 MB/sec
So the 10 Gb ethernet does allow you to have a higher speed, but then the hard drives become the limiting factor. You could upgrade them to SSDs, but if they're connected via SATA, that will only take you up to 550 MB, when the interface becomes the limit. The next step would be NVMe, but then your 10 Gb ethernet will not be fast enough. There is always something ...

(EDITED: Added SATA II interface speed for the other C2550D4I users out there)
 
Last edited:

Frederico

Dabbler
Joined
Aug 28, 2013
Messages
12
That's expect, first few GB are cached to RAM then you're limited by your pool performance, you'd need more disks or better yet some SSDs to get increased sustained write speeds.

Read I'm guessing is being done from the ARC, hence the speed.

Makes sense, but considering it's a 4GB file on a FreeNAS with 16GB, why doesn't it cache the whole thing to RAM? Is this a configurable setting? If I install more RAM, will it make a difference?
 
Joined
May 10, 2017
Messages
838
Makes sense, but considering it's a 4GB file on a FreeNAS with 16GB, why doesn't it cache the whole thing to RAM? Is this a configurable setting? If I install more RAM, will it make a difference?

Only a small part of total RAM is used to cache writes (10%?) it can likely be changed, and adding more RAM will also add to the cache since it's a percentage.
 
Joined
May 10, 2017
Messages
838
Sorry, can't help with that, but doubt that would be a good solution, possibly if you have a very specific use case where you only do small copies at a time, and then don't use the server until it finishes syncing to data to the pool, or else the more data there's to sync the slower the server would be while it's writing the data do the pool.

IMO best bet would be to use a VDEV with more devices, even with normal disks, a few of them in mirrors or RAIDZ can get you decent performance with 10GbE, my HDD pool now can only write @ around 400MB/s but it's because it's fairly full and likely fragmented, when it was new and IIRC it did around 800MB/s (possibly was 600MB/s, not sure) sustained writes.
 
Last edited:

c32767a

Patron
Joined
Dec 13, 2012
Messages
371
Does anybody know where I can change this percentage setting?

The OS manages that dynamically based on what other resource requirements it has (and some memory leaks, in certain versions of 11.1).

If the reason for the throttling is due to insufficient RAM for the buffer, then there are a number of tunables, but it would be far better for the stability and safety of your system to just add some additional RAM first.
 
Joined
Jul 3, 2015
Messages
926
Bought two Intel X520-DA1 cards with Yottamark sticker, verified on Yottamark's website, all good.

Connected using DAC between my Windows PC and my FreeNAS.

Configured 9014 MTU on both cards.

When copying a 4GB file from my NAS to my RAMDisk, I get full speed.
View attachment 25662

When copying the same 4GB file from my RAMDisk to my NAS, I get full speed until about halfway, then it drastically drops to 130ish MB/s. View attachment 25663

NAS Specs:
Intel(R) Xeon(R) CPU E5-1603 v3 @ 2.80GHz
16GB ECC RAM
2x4TB drives in mirrored mode
View attachment 25664

Any tips?
You can't expect to saturate a 10Gb link when your zpool is made up of 1 x mirror vdev of 4TB spinning drives.
 

Frederico

Dabbler
Joined
Aug 28, 2013
Messages
12
You can't expect to saturate a 10Gb link when your zpool is made up of 1 x mirror vdev of 4TB spinning drives.

I understand that, but considering my workload is bursty, I thought it would be cheaper to add more write cache instead of adding more discs.

I did some research and it looks like my options are:

1) Add more disks. Based on this article, I could get close to 500 MB/s write speed with
5x 4TB discs in raidz1.
Cost of adding 3 more 4TB WD Red drives: ~$400

2) Add a SLOG SSD. It looks like since I'm using SMB, all writes are async and therefore it wouldn't help to add a SLOG device because it only works for sync writes, is that right?
Cost of adding an SSD: Free (I already have one that's not being used)

3) Add more RAM. If the amount of RAM reserved to write caching is based on a percentage of total RAM, adding more RAM would increase by write cache size.
Cost of adding 16GB ECC RAM: ~$200 on eBay.

Did I miss anything?
 
Joined
Jul 3, 2015
Messages
926
I understand that, but considering my workload is bursty, I thought it would be cheaper to add more write cache instead of adding more discs.

I did some research and it looks like my options are:

1) Add more disks. Based on this article, I could get close to 500 MB/s write speed with
5x 4TB discs in raidz1.
Cost of adding 3 more 4TB WD Red drives: ~$400

2) Add a SLOG SSD. It looks like since I'm using SMB, all writes are async and therefore it wouldn't help to add a SLOG device because it only works for sync writes, is that right?
Cost of adding an SSD: Free (I already have one that's not being used)

3) Add more RAM. If the amount of RAM reserved to write caching is based on a percentage of total RAM, adding more RAM would increase by write cache size.
Cost of adding 16GB ECC RAM: ~$200 on eBay.

Did I miss anything?
1. If you want performance always go with mirrors and never RAIDZ. In theory your write performance doubles every vdev you add. Therefore based on the figures you have shown us (136MB/s write speed) add another vdev and get 272MB/s added another two vdevs on top of that and 544MB/s double that again 1088MB/s and your not far away from your 10Gb/s assuming you don't find another bottleneck somewhere else in the system.
2. Yes, SLOG only helps with sync writes.
3. More RAM is always good in the ZFS world in particular for read operations however write performance will often be defined by your zpool in the async world.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
I'd hardly classify 4GB of writes as "bursty" myself.

There are tunables to modify the amount of dirty (and unprotected, if you're writing async) data but ultimately you'll still be limited by how fast you can spew data to disk.
 
Status
Not open for further replies.
Top