Configuring FreeNAS for Best NFS Performance

[Chris Tobey]

Hi everyone,

I'm trying to read as much as I can with every spare moment, so be gentle.

Here is my problem: I set up FreeNAS 9.1 on top of a vSphere server a few months ago and the work arounds to get it all going were a pain. We weren't seeing great performance but everything I can find says that ZFS + NFS + EXSi = bad. Now that we have the NAS we are highly dependent on it (as can be expected), but I gave up and went down over the weekend to get rid of the hypervisor. Now I am left with a straight up NAS but we aren't seeing the NFS performance we were hoping for. I also updated to FreeNAS 9.2.1 while I was at it. Our biggest usage for the NAS is to host Jenkins (Continuous Integration) data for our projects. This involves a lot of SVN checkouts which is where our main concern is. On an SVN checkout that takes 5 minutes on local disk, it's usually 15 minutes on the NAS, but can vary up to 2 hours (this makes me sad). Everything is live right now but I can go down for a short period if needed.

What I want: To optimize my configuration based on the hardware I am using to get the best possible speeds for NFS.

Hardware:
First, I know this is overkill. It was intended to host the VM server but that just made things complicated.
1 x SuperMicro 4RU Chassis - SC848A-R1K62B
4 x AMD Opteron(tm) Processor 6284 SE (for a total of 64 cores @ 2.7GHz)
256 GB DDR3 ECC RAM @ 1333MHz
1 x Adaptec RAID 72405 (24 SAS support)
6 x 4TB SAS HDD - Seagate Constellation ES.3 ST4000NM0023
1 x 120GB SSD - Samsung SM843T

I have the basics set up with NFS and CIFS enabled and working, the only problem is speed. ZFS was setup in RAIDZ-1 because I only had 5 drives at the time, as far as I am aware I can't migrate to a RAIDZ2 even though I have a sixth now, so it's sitting there as a spare.

Code:

FreeNAS 9.2.1
  pool: SG1
state: ONLINE
  scan: scrub repaired 0 in 10h56m with 0 errors on Tue Feb 11 01:43:07 2014
config:
 
        NAME                                            STATE    READ WRITE CKSUM
        SG1                                            ONLINE      0    0    0
          raidz1-0                                      ONLINE      0    0    0
            gptid/87cd9f83-560a-11e3-a185-000c29e0733d  ONLINE      0    0    0
            gptid/884c73c9-560a-11e3-a185-000c29e0733d  ONLINE      0    0    0
            gptid/88ca162e-560a-11e3-a185-000c29e0733d  ONLINE      0    0    0
            gptid/8947f033-560a-11e3-a185-000c29e0733d  ONLINE      0    0    0
            gptid/dadf2c63-5866-11e3-b40f-000c29e0733d  ONLINE      0    0    0
        spares
          gptid/9dab55e6-6674-11e3-b84a-000c29e0733d    AVAIL

Any ideas on settings I can try?
Any commands I can run to help understand my setup better?

 

[pbucher]

While I've never measured the difference between bare metal vs VM under ESXi performance on FreeNAS if you build it right you can get really great performance out of FreeNAS under a VM. I've got a VMed ZFS volume I can clock around a gigabyte per second writes to disk out of it.

That said I see 2 potential issues with your setup. 1st I'd get a slog device. If you are going to use ZFS properly and want want decent performance when writing you need a slog device. I'd recommend either a Seagate 600 Pro SSD or a Intel S3700. If you got the money go for a ZeusRAM or Fusion IO card. Next up I'd suggest replacing the Adaptec with a LSI card, Adaptec and FreeBSD are not the best combo, though they can work. Make sure you have the drives configured as JBOD so the raid controller isn't preventing ZFS getting raw access to the drives. On the software side I'd try kicking on autotune as a simple let's try something move and upgrade to 9.2.1(it's got some NFS specific improvements in it alone with some nice ZFS tuning done as part of the 9.2 process).

As for some quick and dirty benchmarking to validate the setup without having to deal with networking & NFS issues. Make a dataset with no compression and no atime set on the zfs options and then drop to the shell and try dd if=/dev/zero of=/ddfile bs=4k count=2000000 in the directory for the dataset. Based on your setup I'd expect to see something in the 300-400 megabytes per second range. Here's the output from my VMed FreeNAS install that has a very similar setup(6 Seagate ES 1TB drive in a RAIDZ2 array).

[root@sandbox] /mnt/test# dd if=/dev/zero of=ddfile bs=4k count=2000000
2000000+0 records in
2000000+0 records out
8192000000 bytes transferred in 19.005767 secs (431027070 bytes/sec)

1 last thing, in general spare disks don't come online in case of a failure automatically(you have to manually do it in the GUI or cli, last time I tested it). It's better take the spare disk and go to a higher raidz level, then it's online and you have no rebuild in case of a failure.

 

[Chris Tobey]

Sadly I cannot change out the Adaptec RAID card, but it is in HBA mode for raw access.

I am on 9.2.1 and I have enabled autotune to try and help. The original 5 disks were for a RAID5 configuration with the Adaptec RAID card and I tried to the get the sixth but couldn't. Once I finally have everything live they give me the sixth drive... Unless I can migrate from Z1 to Z2, not much else I can do with it.

I ran the dd command you suggested a few time and consistently saw ~255-270MB/s.

One thing to note is that I set 'sync=disabled' on the dataset in question and our SVN checkout times were reduced from the 15-120 minutes we were seeing to 5 minutes consistently. I feel that although this is a perfect solution, it will have it's downside as I believe the host can think the data is on disk when in fact it is not.

 

[pbucher]

One thing to note is that I set 'sync=disabled' on the dataset in question and our SVN checkout times were reduced from the 15-120 minutes we were seeing to 5 minutes consistently. I feel that although this is a perfect solution, it will have it's downside as I believe the host can think the data is on disk when in fact it is not.

There is a large downside, is that a hard shutdown(be it power, hardware failure, kernel panic) you could loose the entire pool due to corruption. You can maybe protect against it to some degree by taking periodic snapshots. The best solution is to put it back to standard and add a slog device, I've had really good luck using Seagate 600 Pro SSDs and they aren't that expensive, downsize is that are SATA and depending on how your Adaptec controller is setup you may be able to connect it or not.

Once you get performance backup and running I'd try saving off your config and give vmware a try again, based on your hardware I'd think you'd be able to run ESXi on this box with only a minimal performance hit.

 

[eraser]

What an interesting problem! Thanks for posting this.

I have a few thoughts to share. It has been a few years since I have worked with Subversion, so keep that in mind when reading my below thoughts.

=-=-=-=

If setting sync=disabled makes such a big difference, then you must be getting a lot of synchronous writes. Rather than leaving sync=disabled, the proper fix for this is to set it back to sync=standard (NOT sync=always) and then add a SSD drive to your ZFS volume as a SLOG (sometimes also called a ZIL). The SLOG will temporarily/immediately accept your synchronous writes to itself and therefore increase your performance. You don't need a very big SSD drive for this task, but it should be high quality.

=-=-=-=

How many files does a typical SVN checkout touch? How much data?

I would think that SVN checkouts are mostly file reads. I'll bet if you turn off 'atime' you'll notice some improvement.
According to this link, Subversion never uses 'atime' so it should be safe to turn off - https://www.orcaware.com/svn/wiki/Server_performance_tuning_for_Linux_and_Unix .

In fact, I'd try disabling 'atime' before investing in a SLOG.

You can disable 'atime' on your ZFS dataset configuration.

=-=-=-=

If you reboot your FreeNAS box it can take many many hours for your large ZFS Cache (ARC) to warm up. You can monitor the ARC size by running "arc_summary.py" from a shell. As a matter of fact, can you run that now and include the output in your next reply?

Can you confirm that your Subversion repo is set up with the FSFS backend (instead of BerkleyDB)? what version of Subversion are you running?

Lastly, instead of accessing your Subversion repository over NFS you could also consider using a dedicated Subversion server such as svnserve or through Apache. However, FreeNAS doesn't have either of these applications, so if you want to go this route you would most likely need to build a full FreeBSD server to handle both ZFS and subversion servers.

 

[pbucher]

Lastly, instead of accessing your Subversion repository over NFS you could also consider using a dedicated Subversion server such as svnserve or through Apache. However, FreeNAS doesn't have either of these applications, so if you want to go this route you would most likely need to build a full FreeBSD server to handle both ZFS and subversion servers.

Taking this a step further it should be possible to setup a jail and run subversion in that. That would negate NFS though you might still have sync writes and need a SLOG.

 

[bigphil]

I agree with eraser on this one. If sync=disabled produces the results you're expecting, it sounds like a lot of synchronous writes are happening. A SLOG might help in your situation. Like the others said...check the atime setting on the dataset. Also make sure your compression is set to something acceptable for best performance (lz4 or none). Also...its not like you have a lot of disks here. Only six 7200 rpm drives in a raidz pool. You're not going to be getting break neck speeds here if you have multiple clients accessing the data.

 

[pbucher]

Unless I can migrate from Z1 to Z2, not much else I can do with it.

No sadly you can't convert from Z1 to Z2. If getting physical access and a few large drives isn't a problem it's pretty easy to rebuild the arrays though. I just did this a few months ago, I used zfs send/recv and moved all the datasets off....nuked the zfs vol but told it to not delete the share settings...made a new array and moved the datasets back and rebooted and everything was good to go. I'd also backup the config just incase something does get lost, if you keep all the dataset & volume names the same you could just restore the backup if something gets lost.

On the adaptec thing, I feel your pain, I've got a server that has an adaptec controller in it and I can't do anything about it either. My next step is try to get them to put a decent SSD drive in so I can have better SLOG then I do now in that box.