Is this performance typical for NFS vs iSCSI?

[ZFS Noob]

I would like to open by claiming to be nothing other than a n00b, as my username explains. What I'm trying to do here is learn about ZFS, particularly how it's implemented in FreeNAS, and expecially those issues that affect it's suitability as a backend to my XenServer cluster. My VMs are all Linux, and those most sensitive to storage subsystem performance are running MySQL, with a typical 70/30 read/write ratio.

So far I've been impressed with iSCSI performance, and greatly unimpressed with NFS performance. I'm willing to accept that I've misconfigured things somehow, or that something else is wrong with my deployment, but I thought the best way to open the discussion was to document what I've done, and the performance I'm seeing.

The server I'm testing was repurposed, but here's the layout:

• Dell R710
• SAS/6i SATA HBA
• 4 Seagate Constellation ES.3 SAS drives
• 72G ECC RAM
• 1 Intel 320 SSD, currently installed as a log device.

I'd like people opinions: are these results about what I should be expecting, or should I tinker with sync NFS more? I've run iSCSI for the last 5 years or so and it's performed fine; I'd just prefer to switch to NFS for administrative reasons (primarily that replicating snapshots is easier, and I can get away with one virtual machine repository rather than having to segregate them for snapshotting.) If synchronous NFS is supposed to be this much slower than iSCSI, then so be it -- I'll rebuild the pool with my SSD as an L2ARC instead and be happy. If there's something else wrong, please point me in the right direction so we can try and figure out what it might be.

PERFORMANCE TESTING

I'm no expert here, but the folks at storage review seem to be. My database workload is 71% read: 29% write, which is pretty close to their 70/30 tests, so I've duplicated their test in FIO to generate the numbers here. My data file was only 1 gig in size, so I'm not arguing that these numbers are representative of what my server can do over sustained workloads, but since my databases are all < 10G in size I hope these numbers are close. I also hope the mirrored drives can keep up with my sustained workload, but there aren't that many writes in my environment, either.

My FreeNAS server is running with the default configuration, FreeNAS-9.2.0-RELEASE-x64 (Zpool created in the prior RELEASE version). The only thing non-standard is that I under-provisioned the SSD myself and added it to the zpool from the command line:

ZPOOL LAYOUT

Code:

zpool status  
  pool: nas1mirror
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Dec  9 22:45:50 2013
config
      
        NAME                                            STATE     READ WRITE CKSUM
        nas1mirror                                      ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/1b5a485d-6129-11e3-a041-0026b95bb8bd  ONLINE       0     0     0
            gptid/1bce1e51-6129-11e3-a041-0026b95bb8bd  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/1c420e8a-6129-11e3-a041-0026b95bb8bd  ONLINE       0     0     0
            gptid/1cb736f5-6129-11e3-a041-0026b95bb8bd  ONLINE       0     0     0
        logs  
          gpt/slog                                      ONLINE       0     0     0
 
errors: No known data errors

SLOG Layout (because I think I did this right, but a poorly configured SLOG would explain my problems)

Code:

gpart show ada0                                                                                                      
=>       34  234441581  ada0  GPT  (111G)  
         34       2014        - free -  (1M)     
       2048    4194304     1  freebsd-zfs  (2.0G)  
    4196352  230245263        - free -  (109G)  

THREE TEST LAYOUTS

I cleared one of my Xenserver hosts and ran the tests on that machine. No other processing was going on other than testing. While the FreeNAS server being tested has 4 GigE ports LAGGed together, unfortunately the XenServer host is using active/passive LAGG, as required by Xenserver/EqualLogic. That GigE connection is limiting the performance we can see here by providing a ceiling.

The only thing changed between these tests is as follows:

1. The first test has the test VM on an iSCSI share.
2. Test 2 used an identical VM that was on a NFS share, with sync=disabled.
3. Finally, sync was set to standard, the test file was deleted, and the test was rerun.

Results

What I expecteded was that iSCSI would be the fastest, async NFS would be a bit slower, and regular NFS would run at something like 50% of iSCSI.

I was wrong. Here's the quick summary:

Here's a screenshot I took from the performance graphs on the FreeNAS server while the synchronous NFS test was running, to give you another glance at the performance difference:

iSCSI: Blue. Async NFS: Green. Synch NFS: Red.
Here's the data I collected from FIO:

(Yeah, that's an image because I didn't know how to make a table in BBCode)
So, is this normal, or is something seriously wonky with my SSD, or is something else up?

 

[ZFS Noob]

Oh, the only other thing I can think of that might be relevant: the SAS card wouldn't see the SSD in the drive bays, so I mounted it in a Sonnet Tempo SSD PCI adapter. FreeNAS saw it fine, but that might be part of the issue as well.

 

[cyberjock]

Top be honest, there is no "expected" values. Its really about how much time you want to spend trying to get more performance out of your machine before you just say "screw it, I'm adding more hardware". Check out what I just posted to someone else's recent thread. As I said in the other thread, the topic has been discussed to death. It's not for the noobie trying to learn ZFS. I've played with ZFS for 2 years now and I feel like a noob with regards to tweaking ZFS for NFS. It's one of those things that just can't be handled in a forum. Someone really should write a book on the topic, because you pretty much need a degree in ZFS to get all this crap to make true sense.

Also, you are running it in XenServer, which virtually nobody does. There's been *maybe* 3 users that have used XenServer that I can think of in the last 2 years. Normally I'd say it won't matter that much, but experience here shows that's not completely true either.

Until a good book is written, the forum is doomed to have this topic reexamined every week or two when someone thinks there's something to add. There's tons to add, but what people are asking for is like me asking "how do I build a car.. from scratch". 99% of people will say "Just go buy a car you dumba**" and they'd be right. Either you get this stuff, or you have ALOT more research to do. I know more than most around here, and I still feel like I have ALOT more research to do.

 

[ZFS Noob]

Also, you are running it in XenServer, which virtually nobody does. There's been *maybe* 3 users that have used XenServer that I can think of in the last 2 years. Normally I'd say it won't matter that much, but experience here shows that's not completely true either.

That's true, as long as by "it" you mean "the test machine," not the FreeNAS machine, which is installed on the bare hardware.

Top be honest, there is no "expected" values. Its really about how much time you want to spend trying to get more performance out of your machine before you just say "screw it, I'm adding more hardware".

As long as someone says "yep - that looks about right" then I'll move on. Right now it just seems so counterintuitive that I've got to think there's something basic wrong.

Maybe not though. In which case there's always iSCSI, and with file-based extents (something else I should test) I can actually create and transfer snapshots...

 

[cyberjock]

That's true, as long as by "it" you mean "the test machine," not the FreeNAS machine, which is installed on the bare hardware.

As long as someone says "yep - that looks about right" then I'll move on. Right now it just seems so counterintuitive that I've got to think there's something basic wrong.

Maybe not though. In which case there's always iSCSI, and with file-based extents (something else I should test) I can actually create and transfer snapshots...

I did mean the VMs. But I can see how it might sound like I meant FreeNAS.

There is no "looks about right". Literally, there are parameters beyond your control that can single-handedly kill performance in magnificent ways. So the best I can say is "yes.. you got numbers.. that means the stuff works.. so that's "expected" because literally, that's about all you can assume. Synthetic tests have been proven over and over to just not work as well as throwing real VMs on a system and using them for real.

People have a really really hard time swallowing this because everyone believes that everything can be synthetically tested. The reality of it is there are things you can't synthetically test. Show me how you are going to benchmark a zpool that is 55% full and has fragmentation consistent with 2 years of *your* VMs on it. Oops, suddenly your testing parameters are totally unable to be established. Boom, failure to be testable. See how easy that was? And that was just a single aspect!

All I can say is your best bet is to setup the server how you find best for your situation, or pay someone to set it up in the best situation for your hardware and see how far you can go.

 

[ZFS Noob]

Just a follow-up: I removed the SSD from the pool and re-ran the tests. I got about 21% better performance across the board with the SLOG than without when running NFS with sync=standard (the only test where it matters), which is meaningful.

However, when iSCSI is running at ~ 55 times faster across most metrics, it's difficult for me to generate a whole lot of interest and energy into trying to improve NFS performance a whole lot in my environment. I'd rather dedicate that 120G SSD as an L2ARC and continue to run iSCSI for storage than see the administrative gains that NFS can offer.

It's possible that something like a ZEUSRAM would make a huge difference, but I don't know. I read a thread on another forum linked from one of jgreco's threads where SLOG devices were discussed and one poster commented that he got ~ 200 IOPS without an SLOG, and ~ 30,000 with the same drive I just tried.

Something ain't right, but I don't know the I'm the man to diagnose it.

 

[cyberjock]

I do offer consultant work. ;)

 

[ZFS Noob]

Thanks, CJ. The problem is that you're right: there's no telling where in the chain the problem is.