ok so I spent till 1am testing last night, so I hope this all makes sense.
First of I tested some tuning before I installed the new network card. below is the link to the configurations I used. I tried using the exact settings and various tweaks myself.
http://sourceforge.net/apps/phpbb/freenas/viewtopic.php?f=45&t=3767
http://harryd71.blogspot.com/2008/10/tuning-freenas-zfs.html
Results
I found that any tuning actually made my throughput worse and average drop of about 15-20%. it did however smooth out the peaks and troughs but ended up dropping the overall average through put.
Next I tested Roberto's posted loader.conf settings and was initially excited as the throughput was higher but only for a short time.
Installed Intel CT nic
reset all my settings and config files to default settings.
CIFS throughput was actually a bit better than FTP which was surprising. I could sustain 620-680 mb/s using the default settings. FTP was around 550-570mb/s. either one is very acceptable speeds for me.
I just had to continue testing so I added the configurations from the links above and again the results were a lower throughput. it dropped down to a max of 520 mb/s.
I again tested Roberto's config changes and was initially excited as it was sustaining 780+ mb/s but unfortunately after 3 minutes the throughput dropped to aprox 420-450mb/s and stayed at that rate.
Jumbo Frames testing
I tested various combinations of jumbo frame settings and found that 7k was about the best as far as stability, it didn't however increase the overall transfer speed but did reduce the choppy peak and troughs of the transfer. however seeing as I will be streaming over wireless and gig networks I decided the small enhancement did not warrant the effort to configure.
Wake on Lan Test
first off I believe the onboard nic can do WOL but seeing as I didn't test this card I cannot say for certain. I will test this tonight.
I did however test the Intel CT nic using free software I could called FUSION WOL. worked like a dream. the only thing you needed was either the IP address while the HP server was running and it would find the MAC address then the rest was as easy as a click of the button using the MAC address.
Summary
-Definite improvement with the Intel CT nic and I would recommend this upgrade. (remember it uses a PCIe bus so a lot of cards are just PCI - the intel CT PCIe card fits - I got mine from Harris technology in Sydney)
-Tuning settings I tested caused a slower throughput than using the default settings.
-Jumbo frames while a more consistent throughput did not make enough of an improvement to be worth the tweaking effort on this system.
-Wake on Lan works very easily with this network card.
if this does not answer everyone's questions or you would like more details please don't hesitate to ask. I am happy to help were I can.
I will test tonight the onboard nic for WOL compatibility