Yet another slow CIFS thread

[doomedtx]

I figured it's better to start another topic rather than tack on to another user's post, because they tend to get specific based on the individual's configuration. My system is in the signature. While I'm waiting on my SATA controller and 4 SATA drives, I decided to setup a ZFS volume on a 300GB PATA drive just to get comfortable with the way FreeNAS operates.

I enabled CIFS with Large RW, send with sendfile(2) and AIO enabled. I created a Windows share and made it 777 because I haven't yet learned how to make it writable to the windows user any other way. I'll figure that out later.

On Windows 7 Professional, I see the server as FREENAS.LOCAL and see the CIFS share "Test" under that.

With no config changes copy speeds from Windows to FreeNAS were abysmal, somewhere in the 2-7MB range. iperf indicated the performance was 523 Mbits/sec. The NIC is a PCI Realtek Gigabit and I verified 1000 is shown in the media line in ifconfig. A Windows ftp to the same share got me about 34MB/s on a large file copy.

Here's my current loader.conf:

Code:

#
# Boot loader file for FreeNAS.  This relies on a hacked beastie.4th.
#
autoboot_delay="2"
loader_logo="freenas"
#Fix booting from USB device bug
kern.cam.boot_delay=10000

# GEOM support
geom_mirror_load="YES"
geom_stripe_load="YES"
geom_raid3_load="YES"
#geom_raid5_load="YES"
geom_gate_load="YES"
ntfs_load="YES"
smbfs_load="YES"
xhci_load="YES"
vm.kmem_size_max="2560M"
vm.kmem_size="2560M"
vfs.zfs.arc_min="1536M"
vfs.zfs.arc_max="1536M"
vfs.zfs.prefetch_disable=0

hw.hptrr.attach_generic=0

With everything but the prefetch in there, the speeds were still less than 10MB/s. I added the prefetch line and am now up to about 35 MB/s. It's much better but still slower than I expected.

I also setup another volume on a 120GB PATA disk as UFS and created a Windows share for it. With no tuning the copy speed was 20-30 MB/s but the system panicked during the copy and had to be rebooted. I haven't tried it again since making the loader.conf changes mentioned above. It's not a high priority for me because the ultimate goal is to have 4x2TB drives in a ZFS configuration.

So, is there anything else I can tweak to get higher performance? FYI, I ran iperf again with different settings recommended in a post and came up with this:

Code:

[root@freenas] ~# iperf -c 192.168.2.7 -t 60 -i 10 -f M
------------------------------------------------------------
Client connecting to 192.168.2.7, TCP port 5001
TCP window size: 0.03 MByte (default)
------------------------------------------------------------
[  3] local 192.168.2.70 port 27492 connected with 192.168.2.7 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   647 MBytes  64.7 MBytes/sec
[  3] 10.0-20.0 sec   637 MBytes  63.7 MBytes/sec
[  3] 20.0-30.0 sec   635 MBytes  63.5 MBytes/sec
[  3] 30.0-40.0 sec   636 MBytes  63.5 MBytes/sec
[  3] 40.0-50.0 sec   633 MBytes  63.3 MBytes/sec
[  3]  0.0-60.0 sec  3820 MBytes  63.7 MBytes/sec

Oddly the Windows machine still reported 534 MBits/sec for this same test, but to be fair I've never heard of iperf before tonight so I'm not surprised I don't understand it.

 

[survive]

Hi doomedtx,

You should be seeing all 4GB if you are running the 64 bit build. What's the make and model of the board? What's the NIC?

I would expect more out of a core2 quad, but hold off worrying till you get the "real" drives. Why do you have a SATA controller on order, I would certainly consider using the motherboard ports.

-Will

 

[doomedtx]

The MB is an ASRock 4CoreDual-SATA2. I've been through a lot including unofficial firmwares but it is still stuck at 3313MB of RAM. It has 2 onboard SATA ports but I've ordered a controller so that I can have 4 or more drives in the array.

To clarify my earlier post, the controller is on order. I have 5x2TB drives in my DroboFS which will be used in the NAS as soon as I can get my 2.5TB of data stored somewhere else.

As an aside, the DroboFS is terribly slow (reading from it at 1-6MB/s over a Gigabit LAN right now) and tends to go offline every day, making it a suboptimal candidate for a storage server.

Edit: The NIC is a generic Realtek 8110.

Edit #2: The PATA drive admittedly doesn't seem to be that great:

Code:

[root@freenas] ~# dd if=/dev/zero of=/mnt/Test/ddfile bs=1024k count=16000
16000+0 records in
16000+0 records out
16777216000 bytes transferred in 283.293052 secs (59222123 bytes/sec)

 

[doomedtx]

Another datapoint: I turned on AFP and created an Apple Share. I was able to copy a 10.4 GB file from my iMac to the NAS in about 5 minutes. I didn't get the exact time, but at about 5 mins I was getting about 35 MB/s. Not terrible by any stretch, but it's about half of what I was expecting to see, Maybe faster SATA drives will make a difference. I should know in about a week, because it's going to take that long to copy the 2.5TB of data I'm using off those drives so I can install them in the NAS.

 

[Zorac]

do you have jumbo frames enabled? i found that made big difference for me.

 

[doomedtx]

I don't have it setup for jumbo frames because unfortunately the internal NICs in newer iMacs do not support them. I've read there are ways to setup the NIC to offer different MTU based on which computer is connecting, but I haven't implemented it yet. If I can get a steady 35 MB/s both directions I'll be way ahead of where I am now with Drobo FS. I had read so many posts about people getting speeds in the 70-100 range that I was hoping to get there, too.

 

[Zorac]

not directly related, but it may be of interest, between two windows machines, i went from 15MB/s to 70MB/s when i switch from a cheap dlink gigabit switch with no jumbo frames to a netgear switch with jumbo frames, nothing else was changed, just the switch. when i benched the drive locally (in the windows machine) it was yeilding 110MB/s. not sure if it was the jumbo frames, or just a better switch that cause the increase in speed.

 

[Zorac]

you made me curious, so i bench tested my test freenas box (it only i386 with a 200GB SATA drive), i had 35MB/s with ufs and only 17MB/s with zfs, tried tweaking zfs for a few hours trying the standard things in loader.conf, mtu size, and a number of other things, nothing made a difference to the transfer speeds to the zfs though. doesnt seem to be a process/memory issue though, so i'm not sure where the bottle neck is.

 

[Zorac]

i think im seeing the same problem as you, when i run iperf with freenas as the client, i get around 980Mbit a second, which is pretty much gigabit, when i run windows as the client i get 350Mbit/s, unless i stick in the -l 64KB option, then im getting fully gigabit. netstat isnt reporting any excessive errors, so it seems to be something with the tcpwindow size and the autotuning tha both freebsd and windows7 does.

 

[doomedtx]

I got the parts I needed and now my FreeNAS box has 5x2TB SATA drives in addition to the previously installed PATA drives. I created a ZFS volume which went much quicker than I'd expected. Before starting to copy all my data, I wanted to run some speed tests. I copied some large files from the internal PATA drive to the ZFS volume, and that went well. Then I tried copying a large (~3GB) file from there to Windows. The transfer started at about 25MB/s according to TeraCopy then slowed to 512KB/s. I've watched the copy for about 10 minutes, and while it will occasionally climb to 9MB/s, it's hanging out in the 512-650KB/s range.

Reading through the FAQs and forums, it seems like slow copy is generally related to memory. Here's what top looks like while the copy is going. It looks to me like the system is not running out of memory:

Code:

last pid: 22399;  load averages:  0.14,  0.05,  0.01    up 1+13:50:21  11:19:46
42 processes:  1 running, 41 sleeping
CPU:  0.0% user,  0.0% nice,  0.3% system,  0.0% interrupt, 99.7% idle
Mem: 141M Active, 45M Inact, 1789M Wired, 4300K Cache, 205M Buf, 1236M Free
Swap:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 2099 root          7  44    0 67092K 10120K ucond   3   0:40  0.00% collectd
 2470 www           1  44    0 19324K  4248K kqread  1   0:07  0.00% lighttpd
 1998 root          6  45    0   182M   114M uwait   2   0:06  0.00% python
 1736 root          1  44    0 11776K  2796K select  1   0:02  0.00% ntpd
 2795 root          1  44    0 33300K  5128K select  1   0:01  0.00% sshd
 2757 ted           1  44    0 33300K  5128K select  1   0:01  0.00% sshd
 6475 root          1  44    0 38060K  6816K select  1   0:01  0.00% nmbd
 2214 root          1  47    0  7832K  1516K nanslp  1   0:01  0.00% cron
22362 root          1  44    0 47928K 10888K select  1   0:01  0.00% smbd
 6446 avahi         1  44    0 16932K  2852K select  1   0:01  0.00% avahi-daem
 1296 root          1  44    0  6904K  1520K select  1   0:00  0.00% syslogd
 2755 root          1  44    0 33300K  5052K sbwait  2   0:00  0.00% sshd
 2559 root          1  76    0 64096K 24452K ttyin   1   0:00  0.00% python
 6479 root          1  44    0 46632K 10252K select  1   0:00  0.00% smbd
 2384 root          1  44    0  7836K  1644K select  1   0:00  0.00% rpcbind
 2760 root          1  44    0 13408K  3020K nanslp  1   0:00  0.00% smartd
 2797 root          1  44    0 10172K  2904K ttyin   0   0:00  0.00% csh

As a reminder, here is my loader.conf:

Code:

# Boot loader file for FreeNAS.  This relies on a hacked beastie.4th.
#
autoboot_delay="2"
loader_logo="freenas"
#Fix booting from USB device bug
kern.cam.boot_delay=10000

# GEOM support
geom_mirror_load="YES"
geom_stripe_load="YES"
geom_raid3_load="YES"
#geom_raid5_load="YES"
geom_gate_load="YES"
ntfs_load="YES"
smbfs_load="YES"
xhci_load="YES"
vm.kmem_size_max="2560M"
vm.kmem_size="2560M"
vfs.zfs.arc_min="1536M"
vfs.zfs.arc_max="1536M"
vfs.zfs.prefetch_disable=0

hw.hptrr.attach_generic=0

At this rate it would be better to put the drives back in the Drobo while I figure out what I'm doing wrong in FreeNAS. Any ideas?

 

[b1ghen]

Reading through the FAQs and forums, it seems like slow copy is generally related to memory. Here's what top looks like while the copy is going. It looks to me like the system is not running out of memory:

Code:

last pid: 22399;  load averages:  0.14,  0.05,  0.01    up 1+13:50:21  11:19:46
42 processes:  1 running, 41 sleeping
CPU:  0.0% user,  0.0% nice,  0.3% system,  0.0% interrupt, 99.7% idle
Mem: 141M Active, 45M Inact, 1789M Wired, 4300K Cache, 205M Buf, 1236M Free
Swap:

Isn't there supposed to be some swap assigned?
On my 6 drive RAIDZ2 I have 1 GB set aside for swap on each drive so I have 6GB of swap available when I run top:

Code:

last pid: 17871;  load averages:  1.07,  0.50,  0.29   up 12+23:20:16  22:37:12
39 processes:  2 running, 37 sleeping
CPU:  2.9% user,  0.0% nice, 66.0% system,  6.6% interrupt, 24.4% idle
Mem: 8848K Active, 69M Inact, 3431M Wired, 6868K Cache, 205M Buf, 398M Free
Swap: 6143M Total, 17M Used, 6127M Free

Also my smbd process is taking up a lot more than 0% cpu when a copy is running.