Terrible FreeNAS performance compared to Windows

[cpmoran]

Apologies for the new thread from a newbie.

We recently retired a series of Hewlett Packard Proliant DL320s (Storage Servers, ex-2009/2010) which are a neat config. They have 12 (or 14) hotswap SAS drive bays, a Smart Array P400 controller (PCIe x8, 4-lane SAS), only 8Gb RAM max unfortunately and a Xeon 3020 2.4Ghz processor.

I thought I'd grab one and have a play with FreeNAS as a learning experience, plus I was quite keen to see if we could break through the 14Tb 'limit' of these servers as we had a bunch of 4Tb SATA drives lying around.

The Windows 2008 R2 Standard config, using the controller's RAID 6 with 1Tb 5400rpm 'WD Green' drives gives an ATTO bench in the high 90Mb/sec range. I don't remember specifically, but easily enough to saturate the onboard dual 1Gb ports. That would be fine for the purpose we have in mind.

We took the P400 out and swapped in an IT-mode generic LSI SAS2004 MPT2 'Spitfire' card, installed FreeNAS on bootable USB stick and we were in. Configuring up a ZFS-z or z2 we can't break through more than about 4Mb/sec which is just terrible compared to Windows and we don't understand why.

Individual disk write cache is on according to camcontrol.

We swapped the P400 back, put it into HBA mode and tried again. Still terrible.

We put the Smart Array P400 back into RAID mode, created a RAID 6 volume which then is configured as a single 'stripe' pool, and performance improves to ~10Mb/sec.

The CPU is doing nothing. I had thought it might be SAS contention, but that doesn't explain the reasonable Windows performance.

We're keen these machines don't go into the skip, because they're actually a nice little config that run cool and low power. It is only HP's artificial array limits that prevent larger-than-1Tb drives being installed using the native controller.

I've ordered an H220 from Art Of The Server to see if that makes any difference.

Otherwise, we're enjoying FreeNAS so far, just stumped with the performance difference.

 

[seanm]

Have you tried iperf? To see if it's network vs disks...

 

[cpmoran]

Have you tried iperf? To see if it's network vs disks...

Good point. We get 897Mbits/sec which is about where I'd expect.

 

[Basil Hendroff]

Have you got h/w RAID enabled? I was of the understanding that it's not a good idea to mix h/w and s/w RAID together.

 

[cpmoran]

Have you got h/w RAID enabled?

Per OP, no. The 12 drives are exposed directly.

We started with an Initiator mode HBA and then experimented with the original HP Smart Array controller in different modes seeing if we could find a hint as to where the bottleneck is.

 

[seanm]

I don't know much about FreeNAS performance tweaking/optimization, but since few others are replying so far...

8 GiB RAM is FreeNAS' minimum requirement. ZFS performs better with more RAM, as it will do lots of caching. I wonder if your speeds are suffering because you only have the min RAM.

 

[jgreco]

There is no "HBA mode" on the P400. Presenting VD's representing the drives is not the same thing. What's needed is a clean, well supported firmware to be running on the controller, along with a clean, well supported driver in FreeNAS. The P400 is none of these. A real HBA just talks to the drives and passes stuff back and forth verbatim. The P400 can not do that. I would love to wring the necks of various parties at HP, Dell, etc., who cannot wrap their heads around the idea that JBOD or other VD presentation strategies is NOT an "HBA mode". P.S. The P400 driver performance is known to be poor on FreeBSD.

Your H220 is a *great* move and should help substantially once you flash it to have LSI IT firmware 20.00.07.00. You will still be hobbled by the relatively low RAM.

 

[cpmoran]

There is no "HBA mode" on the P400.

Unless I'm dreaming, it accepted the CLI mode=hba command and just presented the disks no problems. Perhaps there are different versions? BTW, this is a P400 (not P400i "integrated") and is a regular PCI-e card. We never intended to stick with the P400 it was just to prove a theory that the hardware is capable of much higher speed.

We have used an LSI card we have handy with firmware 20, successfully flashed to IT mode and the performance is terrible compared to Windows.


UPDATE: We installed 11.3 RC2 and the speeds have gone up to 10Mb/sec using the LSI card which is still a long way behind Windows.

Also one of our guys thought they'd broken out one of these boxes and presented it as an iSCSI LUN and performance is fine, even just using the P400 in RAID 6 mode (yes, we know - it was an experiment). As we're going to find that version and load it too. We think 11.2 U2. So we'll be experimenting with non-SMB attachment methods today.

Lastly, we watched the RAM use carefully, it never peaked beyond about 2.5Gb in use and 11.3 RC2 now has a great reporting dashboard with this info directly onscreen showing the same.


UPDATE on the UPDATE: iSCSI mode gives us ~70Mb/sec and the CPU is pegged at 94%.

 

[jgreco]

Unless I'm dreaming, it accepted the CLI mode=hba command and just presented the disks no problems. Perhaps there are different versions? BTW, this is a P400 (not P400i "integrated") and is a regular PCI-e card. We never intended to stick with the P400 it was just to prove a theory that the hardware is capable of much higher speed.

You know those cars that the dealership offers to put racing stripes and a spoiler on? Having those things does not make it a race car even if the final invoice says "Racing package".

"mode=hba" proves that it isn't actually an HBA, because an actual true HBA doesn't have any other mode.

You really need the LSI HBA and driver. The P400's FreeBSD driver ("ciss") and the card firmware (any version) is known to be a bit of a flaky combination. That will not end well when something goes sideways and there's a disk error, or other heavy usage that causes the controller to freak. I took all our P400 and P400i's and stuck them in hypervisors, where they were not great but also not terrible. Also now deprecated as they're not supported in any ESXi past 5.5. So the P400 is basically best binned.

We have used an LSI card we have handy with firmware 20, successfully flashed to IT mode and the performance is terrible compared to Windows.

But that, that there's bad.

UPDATE: We installed 11.3 RC2 and the speeds have gone up to 10Mb/sec using the LSI card which is still a long way behind Windows.

Are we talking 10 megabits per second or ten megabytes per second?

https://www.ixsystems.com/community/threads/terminology-and-abbreviations-primer.28174/

Either way is terrible. It's *possible* that the 3020, being a single-core CPU, simply can't cope. ZFS is basically substituting CPU for dedicated RAID hardware. But it does have hyperthreading, and I would expect that that's gotta help avoid CPU contention. So despite a comment elsewhere saying you were maxxing out CPU, let's look at other stuff first.

1) Make sure you've got hyperthreading enabled.

2) Let's test the pool's local performance. Create a new dataset, turn off compression on the dataset, then log in and do "dd if=/dev/zero of=/mnt/mypool/testdataset bs=1048576 count=65536". Ideally this comes out with at least 50MBytes/sec. You can use control-T to monitor it if you're using a real console or PuTTY. I don't know if the GUI fakeconsole will handle ^T. Kinda betting you can actually crank some good speed out of the pool.

3) If that's the case, then we throw shade at the network controller. Because despite being an Intel-based product, HP has a long proud history of using non-Intel ethernet controllers that work for crap. The problem with a lot of these boogers is that they're uncommon controllers, and even if they are well-supported by HP on Windows, that doesn't mean anything for the free software world, where in many cases driver authors have to reverse-engineer the entire thing to make it work. So what kind of controller is your ethernet chipset showing up as?

4) And then we should use iperf to test it. And this might be a happy situation if it turns out to be the problem, because you can just drop a different controller in.

Also one of our guys thought they'd broken out one of these boxes and presented it as an iSCSI LUN and performance is fine, even just using the P400 in RAID 6 mode (yes, we know - it was an experiment). As we're going to find that version and load it too. We think 11.2 U2. So we'll be experimenting with non-SMB attachment methods today.

Lastly, we watched the RAM use carefully, it never peaked beyond about 2.5Gb in use and 11.3 RC2 now has a great reporting dashboard with this info directly onscreen showing the same.


UPDATE on the UPDATE: iSCSI mode gives us ~70Mb/sec and the CPU is pegged at 94%.

2.5 gigabits of memory in use? Doesn't make sense.

 

[cpmoran]

do "dd if=/dev/zero of=/mnt/mypool/testdataset bs=1048576 count=65536".

snip

iperf to test it.

root@freenas[~]# dd if=/dev/zero of=/mnt/data bs=1048576 count=65536
2048+0 records in
2047+0 records out
2146435072 bytes transferred in 2.105341 secs (1019519170 bytes/sec)

C:\Users\cpmoran\Desktop\iperf-3.1.3-win64>iperf3 -c 172.16.1.26
Connecting to host 172.16.1.26, port 5201
[ 4] local 172.16.1.5 port 56032 connected to 172.16.1.26 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 70.6 MBytes 584 Mbits/sec
[ 4] 1.01-2.01 sec 66.5 MBytes 559 Mbits/sec
[ 4] 2.01-3.01 sec 80.0 MBytes 672 Mbits/sec
[ 4] 3.01-4.01 sec 80.4 MBytes 675 Mbits/sec
[ 4] 4.01-5.01 sec 73.1 MBytes 614 Mbits/sec
[ 4] 5.01-6.01 sec 86.6 MBytes 728 Mbits/sec
[ 4] 6.01-7.00 sec 78.8 MBytes 662 Mbits/sec
[ 4] 7.00-8.00 sec 96.1 MBytes 808 Mbits/sec
[ 4] 8.00-9.00 sec 71.5 MBytes 601 Mbits/sec
[ 4] 9.00-10.02 sec 77.4 MBytes 640 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.02 sec 781 MBytes 654 Mbits/sec sender
[ 4] 0.00-10.02 sec 781 MBytes 654 Mbits/sec receiver

iperf Done.


These servers use a Broadcom ethernet chip, not sure which. We swapped in an Intel I350 T2 Server, no change.

It seems to be something going on at Samba level, because as I mentioned the iSCSI performance is ~70MBytes/sec.

 

[jgreco]

You need to do something different there on the dd line because it didn't throw any significant amount of data, and I should probably have been a little more specific such as /mnt/mypool/testdataset/file ... can you repeat that as /mnt/data/file, and make sure that compression on /mnt/data is disabled?

It is definitely possibly a "slow Samba" thing but if that's the case you REALLYREALLY need to make sure that hyperthreading is enabled, because you only have one CPU core and you need to make as much of that as possible.

Check in /var/run/dmesg.boot and make sure that there is a line such as "SMP: AP CPU #1 Launched!" in there. This is it spinning up the hyperthreaded CPU.

"sysctl hw.ncpu" should also read 2. If it reads 1, then you are probably not in for a good time and you really need to figure out how to get your system to enable hyperthreading.

 

[cpmoran]

It is definitely possibly a "slow Samba" thing but if that's the case you REALLYREALLY need to make sure that hyperthreading is enabled, because you only have one CPU core and you need to make as much of that as possible.

The Intel Xeon 3060 CPU in this machine is a dual-core, non-HT capable processor.

We'll have to leave this project for the moment. We've configured a pool, presented it as iSCSI and we're seeing ~120Mbytes/sec over the LACP bonded link. We'll let the host server do the dedupe and compression in Windows Server 2016 R2. It's all we need for the moment and the performance is great in this mode.

It certainly seems like a protocol software issue. We don't have any ability to test AFP but we'll re-rig and do some NFS tests later.

Thanks everyone for the suggestions.

 

[cpmoran]

FINAL UPDATE: we've given up.

NFS, AFP and iSCSI give the performance we would consider 'normal', into the 120Mbytes/sec range with dual 1Gbit links, but there is something seriously broken with the Samba layer on this out-of-the-box config.

Actually, it's not the hardware, because Windows and Unraid deliver Samba performance exactly as expected.

I can only put it down to a demon in FreeBSD somewhere.