FreeNAS 11.1-U6 10Gig reads, writes start at 10Gig and then drop drastically

[Frederico]

Bought two Intel X520-DA1 cards with Yottamark sticker, verified on Yottamark's website, all good.

Connected using DAC between my Windows PC and my FreeNAS.

Configured 9014 MTU on both cards.

When copying a 4GB file from my NAS to my RAMDisk, I get full speed.

When copying the same 4GB file from my RAMDisk to my NAS, I get full speed until about halfway, then it drastically drops to 130ish MB/s.

NAS Specs:
Intel(R) Xeon(R) CPU E5-1603 v3 @ 2.80GHz
16GB ECC RAM
2x4TB drives in mirrored mode

Any tips?

 

[Johnnie Black]

That's expect, first few GB are cached to RAM then you're limited by your pool performance, you'd need more disks or better yet some SSDs to get increased sustained write speeds.

Read I'm guessing is being done from the ARC, hence the speed.

 

[scwst]

If I understand this correctly, we're roughly looking at these speeds:

• Hard drive: 150 MB/sec
• SATA II interface: 300 MB/sec
  
• SATA III interface: 600 MB/sec
• NVMe PCIe interface: 2500 MB/sec
• 1 Gb ethernet: 120 MB/sec
• 10 Gb ethernet: 1200 MB/sec

So the 10 Gb ethernet does allow you to have a higher speed, but then the hard drives become the limiting factor. You could upgrade them to SSDs, but if they're connected via SATA, that will only take you up to 550 MB, when the interface becomes the limit. The next step would be NVMe, but then your 10 Gb ethernet will not be fast enough. There is always something ...

(EDITED: Added SATA II interface speed for the other C2550D4I users out there)

 

[Johnnie Black]

but then the hard drives become the limiting factor.

Exactly, the pool is the limiting factor, example transferring the same 4 files to both my pools:

8 x 4TB HDDs RAIDZ2

12 x 250GB SSDs RAIDZ2

Both transfers start at 1GB/s but only the SSD pool can sustain close to max speed.

 

[Frederico]

That's expect, first few GB are cached to RAM then you're limited by your pool performance, you'd need more disks or better yet some SSDs to get increased sustained write speeds.

Read I'm guessing is being done from the ARC, hence the speed.

Makes sense, but considering it's a 4GB file on a FreeNAS with 16GB, why doesn't it cache the whole thing to RAM? Is this a configurable setting? If I install more RAM, will it make a difference?

 

[Johnnie Black]

Makes sense, but considering it's a 4GB file on a FreeNAS with 16GB, why doesn't it cache the whole thing to RAM? Is this a configurable setting? If I install more RAM, will it make a difference?

Only a small part of total RAM is used to cache writes (10%?) it can likely be changed, and adding more RAM will also add to the cache since it's a percentage.

 

[Frederico]

Only a small part of total RAM is used to cache writes (10%?) it can likely be changed, and adding more RAM will also add to the cache since it's a percentage.

Does anybody know where I can change this percentage setting?

 

[Johnnie Black]

Sorry, can't help with that, but doubt that would be a good solution, possibly if you have a very specific use case where you only do small copies at a time, and then don't use the server until it finishes syncing to data to the pool, or else the more data there's to sync the slower the server would be while it's writing the data do the pool.

IMO best bet would be to use a VDEV with more devices, even with normal disks, a few of them in mirrors or RAIDZ can get you decent performance with 10GbE, my HDD pool now can only write @ around 400MB/s but it's because it's fairly full and likely fragmented, when it was new and IIRC it did around 800MB/s (possibly was 600MB/s, not sure) sustained writes.

 

[c32767a]

Does anybody know where I can change this percentage setting?

The OS manages that dynamically based on what other resource requirements it has (and some memory leaks, in certain versions of 11.1).

If the reason for the throttling is due to insufficient RAM for the buffer, then there are a number of tunables, but it would be far better for the stability and safety of your system to just add some additional RAM first.

 

[Johnny Fartpants]

Bought two Intel X520-DA1 cards with Yottamark sticker, verified on Yottamark's website, all good.

Connected using DAC between my Windows PC and my FreeNAS.

Configured 9014 MTU on both cards.

When copying a 4GB file from my NAS to my RAMDisk, I get full speed.
View attachment 25662

When copying the same 4GB file from my RAMDisk to my NAS, I get full speed until about halfway, then it drastically drops to 130ish MB/s. View attachment 25663

NAS Specs:
Intel(R) Xeon(R) CPU E5-1603 v3 @ 2.80GHz
16GB ECC RAM
2x4TB drives in mirrored mode
View attachment 25664

Any tips?

You can't expect to saturate a 10Gb link when your zpool is made up of 1 x mirror vdev of 4TB spinning drives.

 

[Frederico]

You can't expect to saturate a 10Gb link when your zpool is made up of 1 x mirror vdev of 4TB spinning drives.

I understand that, but considering my workload is bursty, I thought it would be cheaper to add more write cache instead of adding more discs.

I did some research and it looks like my options are:

1) Add more disks. Based on this article, I could get close to 500 MB/s write speed with
5x 4TB discs in raidz1.
Cost of adding 3 more 4TB WD Red drives: ~$400

2) Add a SLOG SSD. It looks like since I'm using SMB, all writes are async and therefore it wouldn't help to add a SLOG device because it only works for sync writes, is that right?
Cost of adding an SSD: Free (I already have one that's not being used)

3) Add more RAM. If the amount of RAM reserved to write caching is based on a percentage of total RAM, adding more RAM would increase by write cache size.
Cost of adding 16GB ECC RAM: ~$200 on eBay.

Did I miss anything?

 

[Johnny Fartpants]

I understand that, but considering my workload is bursty, I thought it would be cheaper to add more write cache instead of adding more discs.

I did some research and it looks like my options are:

1) Add more disks. Based on this article, I could get close to 500 MB/s write speed with
5x 4TB discs in raidz1.
Cost of adding 3 more 4TB WD Red drives: ~$400

2) Add a SLOG SSD. It looks like since I'm using SMB, all writes are async and therefore it wouldn't help to add a SLOG device because it only works for sync writes, is that right?
Cost of adding an SSD: Free (I already have one that's not being used)

3) Add more RAM. If the amount of RAM reserved to write caching is based on a percentage of total RAM, adding more RAM would increase by write cache size.
Cost of adding 16GB ECC RAM: ~$200 on eBay.

Did I miss anything?

1. If you want performance always go with mirrors and never RAIDZ. In theory your write performance doubles every vdev you add. Therefore based on the figures you have shown us (136MB/s write speed) add another vdev and get 272MB/s added another two vdevs on top of that and 544MB/s double that again 1088MB/s and your not far away from your 10Gb/s assuming you don't find another bottleneck somewhere else in the system.
2. Yes, SLOG only helps with sync writes.
3. More RAM is always good in the ZFS world in particular for read operations however write performance will often be defined by your zpool in the async world.

 

[HoneyBadger]

I'd hardly classify 4GB of writes as "bursty" myself.

There are tunables to modify the amount of dirty (and unprotected, if you're writing async) data but ultimately you'll still be limited by how fast you can spew data to disk.