Poor performance with AFP/CIFS/NFS on new home setup

[screndar]

Hello,

I am relatively new to FreeNAS and SAN storage but I am an IT support professional with over 15 years of desktop/server support experience. I have been searching via google and these forums for several days and can't seem to figure out this highly irritating problem: network performance on my newly built home FreeNAS setup and my 27" iMac running Mac OS X 10.7.3 never gets higher than 7 MB/s when copying files to an AFP share. I have also tried CIFS and NFS with similar results.

I have built out a successful FreeNAS setup on some old HP servers at work and was able to use that server to present some LUNs to a test ESXi 4.1 cluster using iSCSI, so I have a good grasp on properly configuring FreeNAS.

Here is my home setup:
- FreeNAS-8.0.4-RELEASE-x64 (10351)
- Shuttle XPC Barebone SG41J4
- Intel Pentium E5700 @ 3.0 GHz
- 8 GB RAM, internal 160GB eSATA HD (where FreeNAS is installed)
- Intel Pro/1000 GT Desktop NIC (not using on board Marvel-Yukon NIC as it doesn't seem to work)
- Cisco RV220W Router
- Highpoint RocketRaid 622 Raid HBA
- SansDigital TR4M+B Enclosure populated with 4x1TB WD1003FBYX Enterprise HDDs
- SansDigital TR4M+BNC Enclosure populated with 4x1TB MK1002TSKB Enterprise HDDs
- Specific console errors seen when using AFP:

Code:

freenas cnid_dbd[5857]: read: Connection reset by peer
freenas cnid_dbd[5857]: error reading message header: Connection reset by peer

every 1-2 mins while trying to copy files to the share (do not see these errors when trying CIFS or NFS but performance is the same)
- IFCONFIG:

Code:

msk0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c011a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,VLAN_HWTSO,LINKSTATE>
	ether 80:ee:73:13:ba:93
	media: Ethernet autoselect
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 00:1b:21:c0:80:3c
	inet 192.168.1.15 netmask 0xffffff00 broadcast 192.168.1.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>

I am using ZFS and have tried several combinations of storage configs with no different results:
- RAIDZ
- RAIDZ2
- Striped
- only using one enclosure to eliminate mixing and matching different HDD manufacturers

I have updated the BIOS on the Mobo and Raid HBA card, couldn't find any updates for the Intel Pro/1000 NIC.

I am thinking this might be a low level network issue, but don't have extensive experience with networking under FreeBSD. I bought the intel Pro/1000 NIC since it was on the supported list and cheap and could do jumbo frames. Any advice would be appreciated.

 

[peterh]

I'd start with
- get and run ttcp to determind NETWORK throughput between server and client, do this in
both directions. ttcp is in "ports" but source is available ( ttcp.c) and builds happily on
both osx and freebsd.
If this indicates that network is ok, replace rocketraid with a plain (cheap)sata board

 

[ProtoSD]

You don't need to use ttcp, there's already iperf built in to FreeNAS, look at the last part of this FAQ question for a link on how to use it:
http://protosd.blogspot.com/2011/12/protosds-unofficial-freenas-8-faq.html#q18

Also, there are some tuneables you can change that could help. I'm trying to put together some more details on this topic because it's been a very commonly occurring one here lately. When I first put together my NAS I had similar issues and changed some settings (not GUI settings) and now things run more smoothly. It's been awhile and the list of stuff I've tweaked it too much to remember what settings do what.

Try searching for some of these to see what they're for, they may help. You can add them from the GUI System->Sysctls (one at a time):

EDIT: Here are some link with details:

(THIS ONE MAY BE THE BEST!) http://serverfault.com/questions/64356/freebsd-performance-tuning-sysctls-loader-conf-kernel

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-kernel-limits.html

http://proj.sunet.se/E2E/tcptune.html

Code:

kern.ipc.maxsockbuf=16777216
kern.ipc.nmbclusters=32768
kern.ipc.shmall=32768
kern.ipc.shmmax=67108864
kern.ipc.somaxconn=8192
kern.maxfiles=65536
kern.maxfilesperproc=32768
kern.maxvnodes=250000

net.inet.tcp.delayed_ack=0
net.inet.tcp.inflight.enable=0
net.inet.tcp.cc.default.algo=htcp
net.inet.tcp.path_mtu_disoovery=0
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspaoe=65536
net.inet.tcp.rfc1323=1
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=65536
net.inet.udp.maxdgram=57344
net.inet.udp.recvspace=65536
net.local.stream.recvspace=65536
net.local.stream.sendspace=65536

 

[screndar]

Thanks for the input and sorry for not getting back to this sooner, got very busy at work.

Here are the results from an iperf test:

Code:

iperf -c 192.168.1.15 -P 1 -i 1 -p 5001 -f k -t 10
------------------------------------------------------------
Client connecting to 192.168.1.15, TCP port 5001
TCP window size: 65.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.214 port 59016 connected with 192.168.1.15 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  90240 KBytes  739246 Kbits/sec
[  4]  1.0- 2.0 sec  90624 KBytes  742392 Kbits/sec
[  4]  2.0- 3.0 sec  90240 KBytes  739246 Kbits/sec
[  4]  3.0- 4.0 sec  90112 KBytes  738198 Kbits/sec
[  4]  4.0- 5.0 sec  90240 KBytes  739246 Kbits/sec
[  4]  5.0- 6.0 sec  90240 KBytes  739246 Kbits/sec
[  4]  6.0- 7.0 sec  90112 KBytes  738198 Kbits/sec
[  4]  7.0- 8.0 sec  90368 KBytes  740295 Kbits/sec
[  4]  8.0- 9.0 sec  90368 KBytes  740295 Kbits/sec
[  4]  9.0-10.0 sec  90368 KBytes  740295 Kbits/sec
[  4]  0.0-10.0 sec  903040 KBytes  739699 Kbits/sec
Done.

I have tried re-seating the RocketRAID card and adding all of the tunables and did not see any change in the AFP share performance.

I am starting to think that the problem may be the RocketRAID card - I am starting to see some weird behavior when rebooting the system where some disks are not showing up in the zpool properly due to a timeout. After rebooting again the disk comes back online fine. I did some more searching and I can't seem to find any good instructions on how to install a 3rd party driver - although the RocketRAID 622 is on the list of natively supported cards for FreeNAS 8.

I really want the flexibility of future expansion to larger HDDs so I was planning on using software raid and RAIDZ/RAIDZ2 instead of a hardware raid. My main motivation for this setup is to rip and store all of my DVDs (400+) and Blu-Rays so I can stream them to my AppleTVs either through my iMac via iTunes or hopefully in the future directly from FeeeNAS itself.

I am thinking I should just buy a regular eSATA card to use with the SansDigital enclosures. I am trying to use the RocketRAID right now because it came bundled with one of the enclosures. Does anyone have any recommendations on a good regular eSATA card that works well?

I will try playing around with different configs including a hardware raid to see if anything changes, but I think I will be buying a regular eSATA card as well.

 

[screndar]

Something is definitely going on with the RocketRAID card. If I create a hardware raid with the disks FreeNAS doesn't recognize it, in fact it just sees the individual drives and lets me create a software raid apparently on top of the (supposedly) already created hardware raid.

I should also mention I upgraded to 8.20 BETA 2. I am going to try re-installing 8.04 from scratch and possibly 7 as well to see if I do have a bad card.

I did test out the card briefly with Windows 7 on this system and it worked fine creating and accessing a hardware raid. Maybe the BIOS update borked the card.

Has anyone had any positive or negative experiences using a RocketRAID 622 card? Doing some quick googling I see that their customer support seems to be lacking if not non existent.

I would also appreciate any recommendations on a good eSATA PCI-e board that works well with FreeNAS 8. Thanks!