Terrible Read and Write performance

[Donny Davis]

I will start by saying I am long time FreeNAS user. Performance has always been top notch, and reliable.

I have two HP dl380 G6 servers with 2x6 core procs, 24GB of ram, 16x 600G sas drives, two 9211 sas hba's, and two mellanox 10GBE cards.

zpool has been in two different configs, two 8 disk zpools in raidz3 and one raidz3 16 disk zpool

The issue I am having is with FreeBSD/NAS 10 where the write and read performance is vastly below that of hardware raid, lvm, and any other configuration.

I am seeing 8mb/s nfs transfer speeds between the two machines for large files(movies), and less for small files 3-4mb/s

With freenas 9.3 I can completely saturate a 1GB link transferring the same files with the same setup. (100-110mb/s)

I would gladly just use FreeNAS 9.3 is mellanox would toss us a bone and make a driver for freebsd 9x


Any ideas what I am doing wrong.

 

[dlavigne]

The issue I am having is with FreeBSD/NAS 10 where the write and read performance is vastly below that of hardware raid, lvm, and any other configuration.

Do you mean FreeNAS 10? This is still alpha software in the throes of a middleware re-write. AFAIK, debugging is on and a good portion of the middleware is still missing or in progress...

 

[Donny Davis]

Im not using the UI.

I'm really trying to figure out why there is such a massive performance gap between freenas 9.3 and freebsd/nas 10

 

[dlavigne]

The middleware rewrite has nothing to do with the GUI and everything to do with what happens under the hood. Best to wait at least another month or so before testing it.

 

[Donny Davis]

So you're saying that the zfs performance depends on the middleware, which does nothing other than provide a way to configure ZFS and the other freenas features such as service control.

Does the middleware in some way affect how ZFS runs in the kernel, and how NFS runs in the kernel?

Running a copy of a 4GB file from one directory to another yields 118MB/s speeds in freenas 10.... freebsd 10.1 provides similar results.

I have checked every tunable, and turned nerd knobs for the last two weeks to no avail.

I have also tried ZFS on Linux, which gives similar performance to freenas 9.3, which at this point is the road I am going to have to head down if I can't figure out why the performance is so terrible on free** 10

 

[Bidule0hm]

The question is why do you want to use FreeNAS 10 when it's not even in beta? Just use FreeNAS 9.3 and the problems will go away.

 

[SweetAndLow]

Running a copy of a 4GB file from one directory to another yields 118MB/s speeds in freenas 10.... freebsd 10.1 provides similar result

This is quite interesting, a local copy like this should give you weil over 300MB/s transfer speed. Especially with multiple vdevs. You need to just use freebsd 10 and ignore freenas for the time being. Figuring out your performance issues on a platform that isn't changing is better than one that is.

 

[Donny Davis]

The issue was two bad drives. I never received any errors, but when I put in a HP p800 with Linux as the OS, the p800 reported two drives as failed. I replaced them, and revisited this, as the issue went away.