Single NFS write stream limited at 70MB/s?

[David E]

Hi all-
System Specs:
Xeon E3-1240v2, 32GB RAM, m1015 (9211 IT mode), 6xSeagate NAS 3TB drives
ESXi 5.5, FreeNAS 9.1.1 in a VM with 16GB of RAM, encrypted 2xraidz

I've been doing some testing over NFS with a new Haswell MBP. I've mounted it with wsize/rsize = 1MB, and on the ZFS side set sync to disabled. I've also enabled 9k jumbo frames everywhere.

From my MBP I can issue a dd to the mounted NFS: dd if=/dev/zero of=tmp.dat bs=2048k count=5k, and cap out at ~70MB/s. If I add a second stream (from the same machine) I can get a combined total of around 120MB/s (2x60). I also checked iperf in to the machine and it can achieve 1Gb/s no problem.

So the main question, why is a single stream only able to get 70MB/s? I would expect it to max out the bandwidth at 120MB/s. Is there something I need to tweak to enable this? Is anyone else able to get a single 120MB/s write stream over NFS on 1Gbe?

Thanks!

 

[cyberjock]

When you start playing with settings like the wsize and rsize, its a give and take. You are giving up something in relation to taking something else. If it wasn't the happy medium it wouldn't be the default. So color be surprised when you've started using custom settings and had problems.

sync=disabled is just stupid. I won't discuss it any further as its been discussed too many times before. It's in the forums if you want to know why. Just do a quick search and see what I mean.

FreeNAS in a VM is doable. It takes knowledge and wisdom to get it right. Both performance and reliability are at risk if you don't know what you are doing. Needless to say the second you virtualize FreeNAS you throw in a whole list of other potential problems. On that note, you are really on your own to figure out the problem because you virtualized. It's damn hard to troubleshoot a FreeNAS machine that isn't a real machine at all.

For people that aren't familiar with jumbo frames, they can be stupid to implement. Not sure how much experience you have with jumbo frames. But just like #1 if you customize your server excessively you run the risk of having other problems.

I have almost identical hardware to yours(E3-1230v2 so slightly less powerful) and I can saturate CIFS and NFS at any time(117MB/sec+ consistently). I don't have sync=disabled. I don't have jumbo frames set up. I don't have a wsize/rsize of 1MB. I do have an encrypted pool. I am running in a VM.

I also used the defaults.

 

[David E]

I assure you I wanted nothing more than to use the defaults, and tried them initially and had sub-par performance, hence my exploration of ways to fix said problems.

When you start playing with settings like the wsize and rsize, its a give and take. You are giving up something in relation to taking something else. If it wasn't the happy medium it wouldn't be the default. So color be surprised when you've started using custom settings and had problems.

sync=disabled is just stupid. I won't discuss it any further as its been discussed too many times before. It's in the forums if you want to know why. The search button can be your friend.

I've done quite a bit of reading, and intentionally disabled it so I could ensure I had some level of baseline performance that was sane. Once that was established I would turn it back on and then see what was what.

FreeNAS in a VM is doable. It takes knowledge and wisdom to get it right. Both performance and reliability are at risk if you don't know what you are doing. Needless to say the second you virtualize FreeNAS you throw in a whole list of other potential problems. On that note, you are really on your own to figure out the problem because you virtualized. It's damn hard to troubleshoot a FreeNAS machine that isn't a real machine at all.

4. For people that aren't familiar with jumbo frames, they can be stupid to implement. Not sure how much experience you have with jumbo frames. But just like #1 if you customize your server excessively you run the risk of having other problems.

I haven't used jumbo frames extensively in the past, but of course the theory is less packets per second -> less CPU overhead -> *ideally* better performance, or at least more CPU to run other tasks.

I have almost identical hardware to yours(E3-1230v2 so slightly less powerful) and I can saturate CIFS and NFS at any time(117MB/sec+ consistently). I don't have sync=disabled. I don't have jumbo frames set up. I don't have a wsize/rsize of 1MB. I do have an encrypted pool. I am running in a VM.

*This* is very useful feedback. What would be really helpful is an exact list of hardware components, and solid benchmarks indicating performance achieved on that configuration, so that someone can just buy the configuration that meets their minimum expected performance needs. I did read the hardware guide posted in your sig.

If you don't mind responding, I'd really like to know more about your setup. In particular:

-Do you have a SLOG, and if so what is it and how is it configured?
-Do you have a L2ARC, and if so how is it configured?
-What is your mechanical disk configuration?
-If you are exporting NFS back to ESXi, what kind of sequential write/read performance do you get to the mount from running dd in ESXi's ssh console?
-If you have a Linux (or similar) VM running with its image mounted via NFS, what sequential write/read performance do you get from running dd within the VM to its disk?
-If you have more than 1 NIC, have you tried seeing what your total write/read sequential performance is via multiple streams?
-Have you had to tweak anything outside of the defaults? If so, I'd love to hear what.

If you were in my area I'd take you out to lunch and pick your brain, thanks for any assistance :)

 

[cyberjock]

I assure you I wanted nothing more than to use the defaults, and tried them initially and had sub-par performance, hence my exploration of ways to fix said problems.

I've done quite a bit of reading, and intentionally disabled it so I could ensure I had some level of baseline performance that was sane. Once that was established I would turn it back on and then see what was what.

Makes sense to me.

I haven't used jumbo frames extensively in the past, but of course the theory is less packets per second -> less CPU overhead -> *ideally* better performance, or at least more CPU to run other tasks.

Yeah, with your CPU that won't matter enough for you to notice. It mattered when Pentium IIIs were state of the art. Not so much anymore.


-Do you have a SLOG, and if so what is it and how is it configured? No SLOG
-Do you have a L2ARC, and if so how is it configured? No L2ARC
-What is your mechanical disk configuration? 6x3TB in RAIDZ2 with encryption
-If you are exporting NFS back to ESXi, what kind of sequential write/read performance do you get to the mount from running dd in ESXi's ssh console? I'm not using NFS at all.
-If you have a Linux (or similar) VM running with its image mounted via NFS, what sequential write/read performance do you get from running dd within the VM to its disk? I am running a Linux VM on iSCSI, but no NFS on my network at all. dds to my boot disk are about 45MB/sec. But speed can fluctate wildly for a bunch of reasons.
-If you have more than 1 NIC, have you tried seeing what your total write/read sequential performance is via multiple streams? I haven't tested with more than 1 NIC since my zpool is my limiting factor anyway.
-Have you had to tweak anything outside of the defaults? If so, I'd love to hear what. No tweaks at all.

If you were in my area I'd take you out to lunch and pick your brain, thanks for any assistance :)

Hopefully you are somewhere warmer and dryer. It's cold and rainy here for the last week or so. I'll hop in my vehicle and meet you wherever you are if its warmer there!

I will tell you that with iSCSI having lots of RAM for ZFS caching and potentially adding an L2ARC can be very beneficial. Note that using an L2ARC does require you to have RAM for the index, so FreeNAS will want even more RAM. Unfortunately, if you are like me you are really hindered at expanding because you can't put more than 32GB in your system and you want to run other VMs too. I've seriously considered going to socket 2011 just for the additional RAM options. But since I'm on a fixed income and I haven't even had this motherboard and CPU for 4 months I'm really not particularly thrilled with going out and buying even more hardware. I really wasn't expecting to be using ESXi long-term which is why I bought the hardware I did. Now I can't give up ESXi...[/quote]

 

[David E]

-If you are exporting NFS back to ESXi, what kind of sequential write/read performance do you get to the mount from running dd in ESXi's ssh console? I'm not using NFS at all.
-If you have a Linux (or similar) VM running with its image mounted via NFS, what sequential write/read performance do you get from running dd within the VM to its disk? I am running a Linux VM on iSCSI, but no NFS on my network at all. dds to my boot disk are about 45MB/sec. But speed can fluctate wildly for a bunch of reasons.

Did you turn on sync=always for the zvols you are using for iSCSI? If not, are you just willing to accept the loss risk in the event of an outage? 45MB/s seems fast, IIRC it was about the speed I was seeing from within a VM when I had sync=disabled over NFS.

Hopefully you are somewhere warmer and dryer. It's cold and rainy here for the last week or so. I'll hop in my vehicle and meet you wherever you are if its warmer there!

California, head over, it is beautiful outside right now :)

I will tell you that with iSCSI having lots of RAM for ZFS caching and potentially adding an L2ARC can be very beneficial. Note that using an L2ARC does require you to have RAM for the index, so FreeNAS will want even more RAM. Unfortunately, if you are like me you are really hindered at expanding because you can't put more than 32GB in your system and you want to run other VMs too. I've seriously considered going to socket 2011 just for the additional RAM options. But since I'm on a fixed income and I haven't even had this motherboard and CPU for 4 months I'm really not particularly thrilled with going out and buying even more hardware. I really wasn't expecting to be using ESXi long-term which is why I bought the hardware I did. Now I can't give up ESXi...

Ya I am in the exact same boat, and also considering moving up to a two socket E5 system to get more cores and memory capacity.

Maybe you can help me directly reach the end goal, or at least evaluate if it is attainable. I'd ideally like to achieve at *least* 120MB/s of sequential read/write bandwidth from WITHIN one or more VMs. I'm struggling to see how this is attainable using synchronous NFS writes from ESXi. I did a quick dd test (dd if=/dev/zero of=tmp.dat bs=2048k count=5k) from the ESXi console with sync=standard, and on my 2xraidz it was showing ~9MB/s of writes. I then tried adding a Samsung 840 Pro 128GB SSD as a SLOG, it increased to around 24MB/s, but this is a far cry from my ideals (and was kind of shocking that it was only 3x faster.. is something fubar here?). I could trade the Samsung for an Intel S3700, but I don't know that it is going to be >6x faster, and this won't even include any of the overhead from within the VM's PV disk drivers either. Anyway, bottom line is I am fortunately in a position that I could throw more hardware at the problem, but I don't know what hardware will get me what returns.

 

[cyberjock]

Did you turn on sync=always for the zvols you are using for iSCSI? If not, are you just willing to accept the loss risk in the event of an outage? 45MB/s seems fast, IIRC it was about the speed I was seeing from within a VM when I had sync=disabled over NFS.

iSCSI doesn't do sync writes, so that feature doesn't matter except if you do sync=always since that forces all writes to sync writes. I'm not too worried about it in my configuration because:
1. In a worst case scenario, it will be 6 seconds worth of data.
2. I do snapshots of that setup, so if I had an issue I'd just roll back to the snapshot to ensure there was no chance of corruption. This is even betterthan sync=always because I know the file system on the VM will be consistent.
3. The performance penalty makes it suck.

California, head over, it is beautiful outside right now :)

I'm in my vehicle right now! LOL Keep the sun shining until I get there!

Ya I am in the exact same boat, and also considering moving up to a two socket E5 system to get more cores and memory capacity.

Maybe you can help me directly reach the end goal, or at least evaluate if it is attainable. I'd ideally like to achieve at *least* 120MB/s of sequential read/write bandwidth from WITHIN one or more VMs. I'm struggling to see how this is attainable using synchronous NFS writes from ESXi. I did a quick dd test (dd if=/dev/zero of=tmp.dat bs=2048k count=5k) from the ESXi console with sync=standard, and on my 2xraidz it was showing ~9MB/s of writes. I then tried adding a Samsung 840 Pro 128GB SSD as a SLOG, it increased to around 24MB/s, but this is a far cry from my ideals (and was kind of shocking that it was only 3x faster.. is something fubar here?). I could trade the Samsung for an Intel S3700, but I don't know that it is going to be >6x faster, and this won't even include any of the overhead from within the VM's PV disk drivers either. Anyway, bottom line is I am fortunately in a position that I could throw more hardware at the problem, but I don't know what hardware will get me what returns.

To get very high speeds you will need to setup a pool with multiple mirrored vdevs plus lots of RAM, a ZIL, and an SLOG. I'm not sure if this is a home server or work server. But I can tell you right now if this is for home use you will empty your pocketbook trying to get all of this. One person somewhere here did this at work and got good speeds for something just over $10k if I remember correctly. This isn't for the timid(or poor) and will probably require tweaking of ZFS for your exact hardware configuration.

As a cheaper alternative, you might get better luck with a bunch of UFS drives for the different VMs.

ZFS has a lot of overhead because it was designed to protect your data over pretty much everything else, including performance. You can get back some of that performance with enough hardware and tweaking.

I will tell you that it is absolutely attainable. But it could get very expensive and time consuming to get ZFS tweaked to work for your intended function.

 

[David E]

iSCSI doesn't do sync writes, so that feature doesn't matter except if you do sync=always since that forces all writes to sync writes. I'm not too worried about it in my configuration because:
1. In a worst case scenario, it will be 6 seconds worth of data.
2. I do snapshots of that setup, so if I had an issue I'd just roll back to the snapshot to ensure there was no chance of corruption. This is even betterthan sync=always because I know the file system on the VM will be consistent.
3. The performance penalty makes it suck.

Do you shut down all your VMs before doing the ZFS snapshot? I don't know of another way to be assured your VM's internal file system is in a consistent state, unless you were using XFS and forcing it to sync and wait, then trigger FreeNAS to take the snapshot, then resume the FS (seems painful)

To get very high speeds you will need to setup a pool with multiple mirrored vdevs plus lots of RAM, a ZIL, and an SLOG. I'm not sure if this is a home server or work server. But I can tell you right now if this is for home use you will empty your pocketbook trying to get all of this. One person somewhere here did this at work and got good speeds for something just over $10k if I remember correctly. This isn't for the timid(or poor) and will probably require tweaking of ZFS for your exact hardware configuration.

As a cheaper alternative, you might get better luck with a bunch of UFS drives for the different VMs.

ZFS has a lot of overhead because it was designed to protect your data over pretty much everything else, including performance. You can get back some of that performance with enough hardware and tweaking.

I will tell you that it is absolutely attainable. But it could get very expensive and time consuming to get ZFS tweaked to work for your intended function.

Definitely a work server. I'm pretty happy with the ~350/450MB/s sequential write/reads from the mechanical disks, but it sounds like the SLOG is really getting in the way. Does anyone know what kind of performance the boxes from ixsystems achieve? I wonder if throwing a fusion io drive at it would solve the problem (and hurt the pocket book), or if I should just buy a UPS, do daily snapshots, and disable sync (at least for the ESXi dataset).

 

[cyberjock]

Can you PM me your skype or phone number? Probably be faster than a bunch of IMs. ;P

 

[David E]

Can you PM me your skype or phone number? Probably be faster than a bunch of IMs. ;P

Done! :)

 

[jyavenard]

did you end up solving your issue ?

I'm experiencing similar "slow" NFS transfers between two freenas boxes; with very similar hardware to yours: E3-1220, 32GB RAM, 6xWD RAID; though encryption is disabled and I do not use VMware

I set a gigabit connection directly between the two boxes, no switch nothing.
copying a 11GB file across via NFS, gives me 94MB/s maximum
copying that same file across the same pool gives me 445MB/s (so that's local speed)
when I use rsync over NFS: I see speed fluctuating between 70 and 75MB/s only.

(fwiw, dd from /dev/zero gives me 459MB/s write on the box).

Interestingly, I get different result running the same test on the different box; one is almost consistently 20MB/s slower (they really are 100% identical). can't explain why

both nfs export and import are plain standard, no options provided whatsoever...

 

[David E]

I never fully solved it to the point that I would be maxing out each individual NFS stream, but I did find that in aggregate I could get multiple streams (across multiple NICs) at each close to line-rate, so I was pretty happy with that and just decided to move on. I did find that jumbo frames dropped my CPU use by a significant margin during these transfers (15-25%), although presumably this is mainly due to ESXi between the NICs and FreeNAS, but it also slightly decreased my transfer rates. If you solve it do let me know what the problem is :)

 

[one33_bpm]

I never fully solved it to the point that I would be maxing out each individual NFS stream, but I did find that in aggregate I could get multiple streams (across multiple NICs) at each close to line-rate, so I was pretty happy with that and just decided to move on. I did find that jumbo frames dropped my CPU use by a significant margin during these transfers (15-25%), although presumably this is mainly due to ESXi between the NICs and FreeNAS, but it also slightly decreased my transfer rates. If you solve it do let me know what the problem is :)

since you stated that you are using esxi5.5, you might be using a virtual nic. which virtual nic are you using? i've found that the e1000 usually caps out at 72MB/s whereas the VMXNET3 vnic is around 95MB/s