ARC performance issues on 10gbe NIC

Status
Not open for further replies.

DaveY

Contributor
Joined
Dec 1, 2014
Messages
141
Testing 10gbe recently, I discovered that there may be a serious problem with ARC performance when paired with a 10gbe NIC. I'm running VMWare ESXi 5.1U2 over iSCSI against FreeNAS 9.3 Stable. I'm able to get full 10gbps write, read is still only half. ARC hit is as ideal as possible (99.17%) CrystalDiskMark numbers show a good picture. It's almost like ARC is slow in reading and pushing that data to the NIC? I originally started this thread here: https://forums.freenas.org/index.php?threads/10gb-tunables-on-9-10.42548/ my posts start at #40

Here's another thread where someone else reported a similar issue.

https://forums.freenas.org/index.php?threads/nfs-writes-blasting-fast-nfs-read-extremely-slow.19342/

I'm starting to think the bottleneck is now with some sort of tuning on FreeNAS end. Any FreeBSD/ZFS experts that can shed light on why ARC performance is suffering?

  • Windows 7 VM over 10Gbe
  • 19.43GiB (MRU: 15.78GiB, MFU: 6.48GiB) / 32.00GiB
  • Hit ratio -> 99.17% (higher is better)
  • Prefetch -> 62.82% (higher is better)
  • Hit MFU:MRU -> 90.07%:9.32% (higher ratio is better)
  • Hit MRU Ghost -> 0.00% (lower is better)
  • Hit MFU Ghost -> 0.00% (lower is better)

Screen Shot 2016-06-30 at 9.38.22 AM.png
 
Last edited:

DaveY

Contributor
Joined
Dec 1, 2014
Messages
141
FreeNAS graphs during CrystalDiskMark testing. Note the lack of disk I/O during reads which tells me it's only hitting ARC. Read is almost exactly half of writes.
 

Attachments

  • Screen Shot 2016-06-30 at 9.30.56 AM.png
    Screen Shot 2016-06-30 at 9.30.56 AM.png
    23.9 KB · Views: 401
  • Screen Shot 2016-06-30 at 9.31.05 AM.png
    Screen Shot 2016-06-30 at 9.31.05 AM.png
    21.8 KB · Views: 381
  • Screen Shot 2016-06-30 at 9.31.23 AM.png
    Screen Shot 2016-06-30 at 9.31.23 AM.png
    24.5 KB · Views: 368
  • Screen Shot 2016-06-30 at 9.31.31 AM.png
    Screen Shot 2016-06-30 at 9.31.31 AM.png
    34 KB · Views: 357
  • Screen Shot 2016-06-30 at 9.31.41 AM.png
    Screen Shot 2016-06-30 at 9.31.41 AM.png
    43.2 KB · Views: 354

zambanini

Patron
Joined
Sep 11, 2013
Messages
479
i just can tell you that the poor 4k speed limit is a freenas issue. you will find similar benchmarks here
 

DaveY

Contributor
Joined
Dec 1, 2014
Messages
141
@diehard, yes I believe the READ performance is low, considering that it's reading off ARC 99.17% of the time over a 10gbe NIC. Is it low compared to most setups? No. Most people would be happy to get 646MB/s of read speed. Perhaps that's your question?

This was more of a baseline test for our ESXi GigE to 10Gbe upgrade. I don't expect to get performance anywhere near this once I fill up the host with 50 VMs performing different operations. The bottleneck will move to my HDD eventually. BUT for this controlled environment where there are no other VMs competing for the 10gbe, I do expect it to push near twice what it's doing.

@zambanini, good to know someone is seeing the same thing. I've tuned so many variables and nothing made a difference. I do hope someone eventually addresses this.

Also, just to clarify, I'm really focusing on the sequential reads with 32 queue depth. This should really be an ideal situation for ARC. 4k reads always suffer for whatever reason.
 

zambanini

Patron
Joined
Sep 11, 2013
Messages
479
without a bug report ixsystem can not help. regarding the 4k Q1 results: that is the part that most vsphere setups will hurt. on a former software defined storage project we also run into the 13mbyte/s limit. On the same hardware we used another non zfs solution: about 90mbyte/sec on 4k Q1. terminal servers felt complete different. we need the rig soon, so we could not provide a testsetup for ixsystem
 

maglin

Patron
Joined
Jun 20, 2015
Messages
299
I'm thinking you are looking at great speeds. I'm going to assume that you have an SSD ZIL and SLOG since you also have enough RAM to cache writes coming in to get the bandwidth you are seeing. But the reads going out have to access something and those numbers make me think SSD. Just a thought.
 

diehard

Contributor
Joined
Mar 21, 2013
Messages
162
I don't know if anyone has gotten close to wire speeds on the X520/X540 NIC's.

You write speeds are "inflated" by compression, your read speeds are actual data across the wire.

I see similar results with the X540 and more powerful hardware in the same scenario.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
This is on the in-production NAS listed in my signature. NFSv3 backed datastore to ESXi6U2 host with sync disabled and no compression for test, Win 10 VM.


upload_2016-7-1_7-49-50.png

Edit: Sync writes could be better if I had a faster SLOG but they are fine with what I have sync enabled for.

With sync enabled:

upload_2016-7-1_7-54-49.png
 

maglin

Patron
Joined
Jun 20, 2015
Messages
299
So clearly I was wrong. I bet it has everything to do with tuning on the 10G stuff. To bad their isn't a default set of perimeters for each NIC that everyone could use. I know at my work center we used to get site to site (across the country) speeds of around 1.2Tbs when we where using older solaris clients that our COMM folks could tune. Our newer workstations OS which is still Solaris but a much newer version they no longer are able to set turntables and our site to site xfer has dropped to abysmal 200mbs speeds. It really sucks when I'm trying to pull a few TB of data as I basically just go out for a smoke now while I wait for my transfer. This is all on a 10G internal network with 3 OC12's connecting to each of our various sites.

I believe there is a 10G turntables thread around here or maybe its in the 10G primer thread. I just remember reading about it a month or so ago. Might want to go looking for that and see what you can gleam from it.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
So clearly I was wrong. I bet it has everything to do with tuning on the 10G stuff. To bad their isn't a default set of perimeters for each NIC that everyone could use. I know at my work center we used to get site to site (across the country) speeds of around 1.2Tbs when we where using older solaris clients that our COMM folks could tune. Our newer workstations OS which is still Solaris but a much newer version they no longer are able to set turntables and our site to site xfer has dropped to abysmal 200mbs speeds. It really sucks when I'm trying to pull a few TB of data as I basically just go out for a smoke now while I wait for my transfer. This is all on a 10G internal network with 3 OC12's connecting to each of our various sites.

I believe there is a 10G turntables thread around here or maybe its in the 10G primer thread. I just remember reading about it a month or so ago. Might want to go looking for that and see what you can gleam from it.
There is a thread about tuning your freeNAS box for 10Gbe. The trouble with default settings is that tuning is specific to your hardware and work load, so there are just too many variables to cover with defaults. Though I'd agree that out-of-the-box setting are way too low.

Here's a link to the 10gig tuning thread. https://forums.freenas.org/index.php?threads/10gb-tunables-on-9-10.42548/
 

zambanini

Patron
Joined
Sep 11, 2013
Messages
479
the iscsi values are impressive, thank you. time to create a new test env.
 

DaveY

Contributor
Joined
Dec 1, 2014
Messages
141
ahhh, the forum hasn't been notifying me of new posts.

@Mlovelace thanks for the performance numbers. Your numbers are looking closer to what I was hoping from my 10gbe. Still not sure why I can't get more speed out of ARC. Do you have any special tuning values on your FreeNAS to get those numbers? Also, what was your ARC hit ratio during that test? I know tuning all depends on workload, but would love to just duplicate what you have and see if I can get similar numbers.

Thanks
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
ahhh, the forum hasn't been notifying me of new posts.

@Mlovelace thanks for the performance numbers. Your numbers are looking closer to what I was hoping from my 10gbe. Still not sure why I can't get more speed out of ARC. Do you have any special tuning values on your FreeNAS to get those numbers? Also, what was your ARC hit ratio during that test? I know tuning all depends on workload, but would love to just duplicate what you have and see if I can get similar numbers.

Thanks
My tuning is related to network buffers, not arc/l2arc. As for the hit ratio, this NAS is in production, so there is no way for me to isolate what data was in the ARC at the time. The ARC would have been occupied with production data before the start of the test.
 
Status
Not open for further replies.
Top