CIFS shares are slow on HP Microserver

[realscorp]

My setup:
HP Microserver N40L (AMD Turion N40L, 8Gb ECC RAM)
4 x 2Tb WD hard drives raid edition (WD2003FYYS-02W0B0)

Freenas configuration:
Freenas 8.3 x64, installed on 4Gb USB Stick
All 4 drives was included in Raid-Z, so I've got 5.3Tb ZFS Volume
I've made 1 ZFS Dataset on this pool (1.5Tb reserved, 2.5Tb quota) and share it over CIFS.
Freenas was joined to AD (Win2008R2) so domain users can access share.

The problem is:
CIFS Perfomance is poor. 55-65MB/s, so it even cannot saturate Gigabit link.
I think the problem is with CIFS, not a ZFS cause I've tested ZFS perfomance with DD tool (copy 20Gb file) and found that write speed is about 270MB/s and read speed is about 370MB/s.
And I think CPU is not a bottleneck too. In process list smbd eats only about 45% on 1 client write: 1 client, 2 client

What I've already tried:
1. Enable Auto-tuning in Advanced options - no effect at all.

2. Adding some Tunables and Sysctl I've found is some threads on this forum. Also I've tried to add some tuning in Aux parameters in CIFS and tried all combinations of "Large RW", "Enable Sendfile (2)", "AIO" etc.

Now it all look like that (links to images):
Sysctl
Tunables
CIFS parameters
CIFS parameters 2
Share parameters

Result - no visible effect in performance

3. I've changed the on-board LAN adapter to discrete Intel Desktop MT 1000 - may be some speed incrementation by few MB/s but I'm not sure.


Please help - how to increase CIFS/SMB perfomance? What to check next?

 

[realscorp]

What else I have tried:
First of all I've deleted all Tunables and Sysctl I've added previously and performance fall back to 55MB/s. Then:

1. I've deleted Raid-Z volume and created ZFS Mirror volume instead. This has absolutely no effect in network CIFS performance, although CPU usage was slightly less.

2. Next I tried to delete ZFS volume and create UFS mirror raid.
I've got few MB/s increased speed

and slightly less CPU consumption

but performance was still poor - about 65MB/s or less.

3. I've enabled Jumbo Frames on manageable switch and make sure that Duplex settings on Freenas connected port has "Full" value.
No performance changes.

4. Reboot Microserver and look into BIOS setting. I've found that Write Cache on SATA drives was set to "Disable". I've enabled it and boot into Freenas, but performance hasn't changed.

5. Next I decided to exclude CIFS from the equation. AFAIK, the fastest type of Freenas shares is iSCSI. So i've build again a Raid-Z volume and shared a 3TB file extent via iSCSI. But write speed to an iSCSI-attached drive still was about 55MB/s!

On UFS volume over iSCSI i've the same results.



Now I don't know what to blame. Looks like now It's all about network performance. But I've got L2 Gigabit switch and tried 2 different LAN-adapters.
And why do the IRQ process take so many CPU?

Maybe someone can give some advices about what to do next?

 

[glipman]

Can you try using iperf to test the network performance?

 

[Milhouse]

45% CPU usage on a dual-core CPU would suggest it's close to maxing out one core... CIFS (ie SAMBA) is not multithreaded so it's quite likely that you are CPU bound.

 

[cyberjock]

As Milhouse said, you're likely maxed out for hardware since SAMBA is single threaded. Gotta have a beefier processor if you want Gb saturation.

 

[realscorp]

45% CPU usage on a dual-core CPU would suggest it's close to maxing out one core... CIFS (ie SAMBA) is not multithreaded so it's quite likely that you are CPU bound.

AFAIK, Samba starts new thread for every new client connected. When i tries to write on CIFS share with two clients I see in process list 2 smbd processes (take a look at a screenshot named '2 client' in my first post). But even with two writing clients I cannot reach more than 600mbit/s.

As Milhouse said, you're likely maxed out for hardware since SAMBA is single threaded. Gotta have a beefier processor if you want Gb saturation.

But I tried to use iSCSI (with CIFS or NFS disabled) and get only 55MB/s too. iSCSI is not as CPU consuming as SAMBA, AFAIK. So maybe it's not a SAMBA issue at all?


Can anybody comment is it normal that IRQ eats so much CPU? I am newbie with FreeBSD but when I see on my Win PC that 'system/interrupts' using 10+ percent of CPU I know that there is something wrong with hardware or drivers.

 

[realscorp]

Can you try using iperf to test the network performance?

I've made 2 tests, all with default settings. At first test Freenas connected to Win2008R2 host (Intel Gigabit adapter), at second Freenas was acting as a iperf server.
Here's the results:

 

[realscorp]

Can you try using iperf to test the network performance?

So from Windows host to Freenas I have ~700mbt/s and from Freenas to Windows only ~200mbt/s.

 

[cyberjock]

From back in the 90s, IRQs can take what was a superfast machine and slow it to a crawl. IRQs were never meant to be as popular as they became. They interrupt the CPU and that kills performance. Not sure if 10% is low or high, since measuring the lost performance from interrupt requests is hard to quantify.

Your issue with SAMBA is almost certainly because its single threaded. See http://www.samba.org/samba/docs/man/Samba-Developers-Guide/architecture.html. SAMBA does start a new thread for every new client, but the heavy lifting thread is still just 1 thread. The fact that you are dual core and at 45% is a giveaway that you are CPU limited.

If you want more performance you need to upgrade your CPU first. Then you will potentially be bottlenecked somewhere else(Are you using an Intel NIC?)(Do you have enough RAM?)(Can your zpool even provide the performance you want?). After the CPU upgrade you'll need to go back and analyze what your next bottleneck is. I can virtually guarantee you that you won't hit more than 60-70MB/sec or so if you don't have an Intel NIC.

You can find out if your NIC is a potential bottleneck right now with iperf. If you aren't hitting 900Mb/sec+ then your network topology will eventually bottlenecking you.

Edit: Just realized I didn't comment about your iscsi question.. iscsi has it's own potential performance killing issues. It just doesn't work well with ZFS because of copy on write. Testing with iscsi and NFS won't necessarily give you any valid information for identifying performance issues. I'm sure FTP will max out your network speed(whatever it is) but thats only because FTP is stupid simple for a protocol. You can only send and receive files with FTP. There is no locking of files, etc.

 

[realscorp]

From back in the 90s, IRQs can take what was a superfast machine and slow it to a crawl. IRQs were never meant to be as popular as they became. They interrupt the CPU and that kills performance. Not sure if 10% is low or high, since measuring the lost performance from interrupt requests is hard to quantify.

Yes, you're right, IRQ is used for interrupting code execution on CPU so interrupt handler like driver could handle it and resume code running.
It just looks strange to me when interrupts takes 50% of CPU time as shown on 'CPU Usage' graphs in my posts above. But may be it's normal for this hardware or this type of load or for FreeBSD.

Then you will potentially be bottlenecked somewhere else(Are you using an Intel NIC?)(Do you have enough RAM?)(Can your zpool even provide the performance you want?)

In posts above I've noted that I do use the Intel NIC now, and I have 8Gb of RAM and zpool speed tested by DD tool is about 370MB/s read and 270MB/s write.

You can find out if your NIC is a potential bottleneck right now with iperf. If you aren't hitting 900Mb/sec+ then your network topology will eventually bottlenecking you.

I've tested my network with iperf and posted the results but someone (moderator?) just deleted this post from the thread. I don't know why because it was just 3 little screenshots and some text description. I'll now try to post these screenshots back and will hope moderators wouldn't delete it this time.



So it looks like the network could be a bottleneck right now. I'm using an Intel 82574L NIC on FreeNAS host and Intel 82541 NIC on WinServer host and 3Com Gigabit switch between them. So I don't know how to optimize the network to check whether it bottlenecking the performance or not. I think I'll try to make a LAGG with discrete Intel NIC and on-board Broadcom NIC and try to write on Freenas with 2 or more clients and see how can it affect the SMB write speed.

 

[Milhouse]

Not deleted - your post had been marked for moderation by the forum (for some reason - possibly the image links). Approving now...

 

[Milhouse]

I'd now look at your switch... or even your cabling. I can get over 900Mbits between two N36L with Intel desktop NICs.

freenas1 (running FreeNAS 8.0.2-RELEASE) is connected to a Netgear GS608v3 8-port GigE switch, uplinked to a Netgear GS105E 5-port GigE switch, connecting to freenas2 (FreeNAS 8.3.0-RELEASE).

Code:

[root@freenas1] # iperf -c freenas2 -t 30 -m
------------------------------------------------------------
Client connecting to freenas2, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.3 port 52357 connected with 192.168.0.200 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  3.15 GBytes   903 Mbits/sec
[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)


[root@freenas2] ~# iperf -c freenas1 -t 30 -m
------------------------------------------------------------
Client connecting to freenas1, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.200 port 20692 connected with 192.168.0.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  3.26 GBytes   934 Mbits/sec
[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

 

[cyberjock]

LAGG won't improve the single link performance. LAGG is great for situations where you have tons of network links(aka.. lots of users). But ultimately 1 link per NIC. So you can't expect Samba to do anything more than what 1 NIC can do for 1 user. If you have 10 users then you may be a performance increase because you will have 10 links.

Those pictures are a little small. But it looks like the bottom picture says 195Mb/sec. If so you have a network problem(unless you had heavy traffic through that switch from other computers). iperf is designed to max out your network speeds while using almost no CPU power. If iperf tests are sucking, then you can't expect anything better than that. iperf really is the actual limit of what you can send/receive with the sun, moon, and stars all aligned exactly right.

 

[justicefriends]

hi,

i am a new writer to this forum and i am working on the same issue but with much less result - but i am in a testing phase prior AD migration of the system.

having 2 identical hp n40l micro server with 8Gb RAM and 4x2Tb disk set (all brand new) both on a cisco 3750

one of these runs freenas v8.3 x64 - with default config, cifs enabled with one user for test, the disks are in raidz2, deduplication and compression is off. this comp has an additional 8gb kingston stick to run freenas. share is on the raid.
other runs windows server 2008 r2 x64 - default config, one user for test, disks are in hw raid 10. (before win this was also freenas v8.2.3 x64)
this comp has a kingston ssd for the os but share is on the raid.
now working with the same desktop and doing up and downloads on these systems with a 4gig image file (same file all the case) results:
upload
- freenas : 20-25000kbytes/sec
- win : 90-110kbytes/sec

download
- freenas : 40-55000kbytes/sec
- win : 70-95kbytes/sec

my intention is similar to realscorp and i believe the question is the same: how to increase speed?
most of the moves that has been done by realscorpe i have made myself too (did not have tried ftp or iscsi)
- noticed though that microserver with freenas and slow network perf is a trending topic with many threads.

appreciate any thoughts.
cheers
jf

 

[cyberjock]

I don't have an N40L, but if you want more speed I'd first try an Intel NIC.

Also, I seem to remember the N40L having an AMD Turion CPU, which from personal experience years ago isn't a powerhouse CPU but is low power. If I'm right then the CPU may not be capable of faster speeds. Have you tried an iperf test and a dd test? If not I'd do both of those.

 

[justicefriends]

I don't have an N40L, but if you want more speed I'd first try an Intel NIC.

true story, they have hp nc107i PCIe gigabit server adapters (broadcom). both on win and freenas

Also, I seem to remember the N40L having an AMD Turion CPU, which from personal experience years ago isn't a powerhouse CPU but is low power. If I'm right then the CPU may not be capable of faster speeds. Have you tried an iperf test and a dd test? If not I'd do both of those.

also true, during these transfers the cpu hits 50%, same time upon starting the transfer the used memory goes up to 4gig. still in the same system
running win the same cpu / nic / mem results 5 times better performance.

no iperf and dd test yet. i was rather under the impression i am missing an obvious setting here or similar. (due the difference of compared results on same config)

tx
jf

 

[cyberjock]

Look at http://forums.freenas.org/showthread.php?8667-HP-n40l-4*3TB&highlight=n40l+performance

The 3rd post someone discusses that you shouldn't expect high performance from N40L, but is great for a small footprint. Sounds like you expected high performance from a device that can't provide.

 

[justicefriends]

hi, nope. i saw this i think.
but the main thing here is why is it ok with one OS and not ok with the other.

 

[cyberjock]

hi, nope. i saw this i think.
but the main thing here is why is it ok with one OS and not ok with the other.

I don't think you understand the magic of ZFS then.

 

[justicefriends]

i do not try to win a fight here - it is my understanding that most of the long time users have this understanding above my level as common stuff
which is the reason why i am here.

i understand that zfs2 is much slower then using the hardware raid10 mostly in writing. (as per zfs and iops)
my intention is to speed up the freenas system as much us possible variables, settings etc to see what's the top it can go on that hw
might be the best way to reach this is to leave raidz2 out at the door.