Samba performance limit about 50MByte/sec again and again

[Peter Jakab]

Hi All,

I spend lot of my time with performance test, but I can jump out from this problem. I have multiple systems invited in this test at home see below:
1.Old NAS called (not recommended compliant I know)
HW as in my signature below.
NIC: motherboard was msk0 but I purchased Intel PRO/1000 CT as I use today as em0

2. New one called DemoNAS (because just demonstrate 9.3 how will works with 24TB)
New NAS 9.3 latest version (mean FreeNAS-9.3-STABLE-201508250051)
Supermicro X9SRH-7F motherboard
Intel Xeon E5-1620 V2
one modul 16GB Low Volt RDIMM 1600MHz (M393B2G70QH0-YK0 a is recommended list)
Jails and initial datasets:
1*ST3120026AS (120GB old 7200.7) and 1*ST3250310AS (250GB old 7200.10) in stripe stolen from my very old desktops:)
For speed tests zpools raidz2:
4*Seagate Constellation ES 500GB ST500NM0011 in raidz2 just purchased for z2pool killing tests from ebay

3.Main PC
Win7x64 official latest patches
ASUS P8-Z68VPRO (mainboard with
I7-2600K
16GB RAM (as horrible big since 2011)
NIC: in past I use Intel 82579V Gigabit but I also purchased Intel PRO/1000 CT as I use today
Lot of disks (128SSD OCZ Vertex3 for OS and more Seagate for data: 1TB ST1000DM003, 2TB ST2000NM0011, old 2*320GB ST3320620AS etc.)

4.Dell PC
I purchased for low cost on corporate selling days. "Why not PC" for any testing.
Dell Optiplex 755MT
NIC: built in
HDD is rest of what is not used else where 1*ST3120026AS (120GB old 7200.7), 1 Seagate 80GB SATA and 64GB OCZ Vertex3
Dual OS boot Uhbuntu (SSD) and Win7x32(on 120GB)

5.Netgear switch GS108E

My problem I unable go over 40-50Mbyte/sec (MB/s) speeed limit.

What I made in past:
1. checked all of the related threads about "samba/SMB/CIFS" and "performance" but not help me.
2. I first realized my old NAS and Main PC have speed limitation about 40MByte/sec. I think this is known issue due msk0 NIC on NAS and also just 8GB RAM (non ECC I know).
3. So I plan and buy my new NAS for bigger and faster system. Bigger is true but I not able reach faster.
4. Because I have still have 40Mbyte/sec max throughput and iperf also show 350-450Mbit/s (Mbps) speed between MainPC-DemoNAS and OldNAS-DemoNAS I buy 2 qty of Intel PRO/1000 CT NIC cards.
5. So funny the shop send me Intel PRO/1000 GT (PCI not PCIe) so I can confirm same speed limits valid for PCI limited GT series. My measurement 300-350Mbit/sec. So I sent back and ask for PCIe cards.
----------------------------------------------------------------------------------------
Actually:
6. I don't know interrupt sharing could impact performance but in desktop/gaming (not server) motherboards not easy get PCIe slot without any sharing.
MainPC PCIe interupt sharing
have PCIe 2.0 x1_1 slot used. Shared with PCI1:empty, SATAController: here is the most of the HDD except 1TB which on different Marvell PCIe 6.0Gb/s controller). PCIeX1_2 not usefully due behind dual slot video card:) only alternative option PCIe X16_3 (in 1X mode) used by my old PATA controller this is also shared with PCIeX1_2, USB3 and 1394 controller)
OldNAS interupt sharing
the only PCIeX1_1 used due the other PCIEX16_2 disabled by primary passive cooling video card. Shared with USB3 and USB2.0_2 and SATA controller 2 (which I unable determine also have controller 1 and SATA363 in sharing table meawhile 6 SATA ports on motherboard).

7. Main PC to DemoNAS
Samba share on zpool2, max protocol SMB2, with socket options = IPTOS_LOWDELAY TCP_NODELAY.
Direction DemoNAS z2pool>Netgear>Win7x64 1TB disk (latest Intel NIC driver 12.7.28.0 2015.05.26)
stable 46-48MByte/sec (showed by TeraCopy) for 27.18Gbyte m2ts video file (ARC size 11.87GB). Windows Resource monitor show maximal about 50,568,000B/sec (nothing for this disk internal copy about 120MByte/sec). So this is 384Mbps (Megabit/sec) not good Looooong time. In second copy (until ARC) goes up to 56MB/s which is mean 448Mbps but after cache out drop down to 46MB/s again.

Stats:
"systat -vmstat" show the zpool 4 disk just have stable 64KB/t, 204tps, 12-13.66MByte/second 20-24% busy disk for da0-da3

Under verify Resource monitor show read peak from 1TB disk 196,804,000B/sec. Could you believe it 187Mbyte/sec peak?

top -P show smbd process WCPU is around 20.36%. And the CPU threads show 3.1% user on random thread for smbd and 21% for interupt on one stable other thread

No CPU and HDD bottleneck on DemoNAS

Win7x64 have CPU1 is 73-91% all other thread empty (99Mhz x 38 so runs on 3.8Ghz).

8. DemoNAS to Main PC
Reverse direction as before. I make same free space on Demo NAS z2test volume used 538.9GiB(29%) and 1.3TiB free. z2winshare used for this test is 259.1GiB(37%) used and 440.9GiB free. Inherit lz4 compression and I not configured for any encryption.
77-79MB/s

Terra copy 77-81MB/s

Vmstat (at begining of transfer higher numbers)
64-84KB/t
204-551tps
13.23-44.22MB/s
17-50%busy

top-P
WCPU 34.38% smbd
14.79-33.5% interrupt on CPU1
2.4-3.9% user on different CPU

So from 2 test above the read from NAS is 46-56MB/s and write 77-81MB/s

3. DemoNAS iperf server and MainPC jperf (could java limited?)

Code:

bin/iperf.exe -c 192.168.10.34 -P 1 -i 1 -p 5001 -f m -t 10
------------------------------------------------------------
Client connecting to 192.168.10.34, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[156] local 192.168.10.9 port 56714 connected with 192.168.10.34 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0- 1.0 sec  45.8 MBytes   384 Mbits/sec
[156]  1.0- 2.0 sec  45.8 MBytes   384 Mbits/sec
[156]  2.0- 3.0 sec  45.8 MBytes   384 Mbits/sec
[156]  3.0- 4.0 sec  46.3 MBytes   389 Mbits/sec
[156]  4.0- 5.0 sec  46.5 MBytes   390 Mbits/sec
[156]  5.0- 6.0 sec  45.9 MBytes   385 Mbits/sec
[156]  6.0- 7.0 sec  46.4 MBytes   389 Mbits/sec
[156]  7.0- 8.0 sec  45.4 MBytes   381 Mbits/sec
[156]  8.0- 9.0 sec  46.2 MBytes   387 Mbits/sec
[156]  9.0-10.0 sec  45.8 MBytes   384 Mbits/sec
[156]  0.0-10.0 sec   460 MBytes   385 Mbits/sec
Done.

with dual option

Code:

------------------------------------------------------------
Client connecting to 192.168.10.34, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[176] local 192.168.10.9 port 56736 connected with 192.168.10.34 port 5001
[192] local 192.168.10.9 port 5001 connected with 192.168.10.34 port 55377
[ ID] Interval       Transfer     Bandwidth
[176]  0.0- 1.0 sec  12.1 MBytes   101 Mbits/sec
[192]  0.0- 1.0 sec  62.7 MBytes   526 Mbits/sec
[176]  1.0- 2.0 sec  10.3 MBytes  86.7 Mbits/sec
[192]  1.0- 2.0 sec  65.1 MBytes   546 Mbits/sec
[176]  2.0- 3.0 sec  10.7 MBytes  90.0 Mbits/sec
[192]  2.0- 3.0 sec  59.7 MBytes   501 Mbits/sec
[176]  3.0- 4.0 sec  11.8 MBytes  98.8 Mbits/sec
[192]  3.0- 4.0 sec  64.7 MBytes   542 Mbits/sec
[192]  4.0- 5.0 sec  58.0 MBytes   487 Mbits/sec
[176]  4.0- 5.0 sec  11.3 MBytes  95.0 Mbits/sec
[176]  5.0- 6.0 sec  11.4 MBytes  95.9 Mbits/sec
[192]  5.0- 6.0 sec  64.1 MBytes   538 Mbits/sec
[176]  6.0- 7.0 sec  11.2 MBytes  93.7 Mbits/sec
[192]  6.0- 7.0 sec  63.9 MBytes   536 Mbits/sec
[176]  7.0- 8.0 sec  10.8 MBytes  90.6 Mbits/sec
[192]  7.0- 8.0 sec  64.0 MBytes   537 Mbits/sec
[192]  8.0- 9.0 sec  64.1 MBytes   537 Mbits/sec
[176]  8.0- 9.0 sec  11.1 MBytes  93.3 Mbits/sec
[176]  9.0-10.0 sec  11.4 MBytes  95.2 Mbits/sec
[192]  9.0-10.0 sec  63.3 MBytes   531 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[192]  0.0-10.0 sec   630 MBytes   528 Mbits/sec
[176]  0.0-10.0 sec   112 MBytes  93.9 Mbits/sec
Done.

They show my limitations.
You can put up questions about what is the bottleneck (NetGear, cabling etc.) until you not see my next post. About I tested:
- direct LAN cross cable (Systimax Cat5e) between the two system (netgear not impact performance) same samba speeds.
- boot from Uhbuntu USB stick on MainPC
iperf show 947Mbps one-direction and dual 400+700Mbps so I have same problem in win7.
But samba file transfer also limited below 50MByte/sec for him on same hardware.
- Disable ESET Smart Security 4 Firewall and Antivirus same speeds.
- Old FreeNAS directly connected DemoNAS via Systimax Cat5e cross cable iperf show for them also 9xxMbps. But samba same speeds
- Uhbuntu to DemoNAS
with socket options = IPTOS_LOWDELAY TCP_NODELAY
Uhbuntu samba copy from 31MB/s changed to 61MB/s. But this is not Intel PRO/1000 CT NIC card

Dear community:
I would like see over 100MByte/sec samba copy on my screen with my recommended HW:)

Sorry If I long. My problem also long story,
Jackson

 

[Peter Jakab]

So I focus on high CPU load on Win7x64 side
1.win7 tunning
http://www.sysprobs.com/windows-7-network-slow
netsh interface tcp set global autotuning=disabled
netsh interface tcp show global
46MB/sec goes up to 53-54MB/s at begining but on end still 44MB/s
CPU still limited to about 80% on core1
NOT HELP

2.samba option
https://bbs.archlinux.org/viewtopic.php?id=120142
https://wiki.samba.org/index.php/Linux_Performance
https://bbs.archlinux.org/viewtopic.php?id=115788
CIFS service add "strict allocate = yes"
same speeds not help

3.
http://superuser.com/questions/94715/windows-7-file-transfer-speed-over-gigabit-is-slow
Large Send Offload Disabled for IPv4 and IPv6
same speeds not help

4.
Same URL as in point 3 in middle "I noticed that file transfers between Windows network shares (Win2k3 -> Win2k8R2; Win2k3 -> Win7; some other
configurations) are unreasonably slow when TeraCopy (v2.1) is used"

Ohh could my fault so remove TeraCopy as default copy handler.
Problem: I lost automatic verify option (CRC check on original and copied file also). I use it since was bug in FreeNAS 8.x LAN driver (I spend several months to localise problem in to LAN driver not in RAM/CPU etc.
Workaround optional available in right click on file (not easy to use)

From NAS 65MB/sec and also benefit uses windows free memory for readback cache (TeraCopy not uses by default)
To NAS 66-67MB/sec (slowed down)

Result: One bottleneck solved, but why write goes down from 80MB/sec?

5.
NIC driver
Receive buffer up from 256 to 512
Interrupt moderation from Adaptive to High
windows copy 47MB/sec

Interrupt moderation from Adaptive to Extreme
52MB/sec with CPU1 load 50-60%

Interrupt moderation from Adaptive to Low
70-75MB/sec and CPU1 100%

Interrupt moderation from Adaptive to Minimal
74-78MB/sec and CPU1 100%

Interrupt moderation from Adaptive to off
80-85MB/sec and CPU1 100% and same load on peaks on other CPUs

TCP Checksum Offload (IPv4) from Enabled to Disabled
77-93,7MB/sec and PC response slowes down (task manager opens very slowly)
NOT HELP just cause problems Enable back again

Large Send Offload V2 (IPv4) Enabled to Disabled
78-80MB/sec
NOT HELP Enable back again

Results: Second CPU bottleneck caused lot of problems. Interrupt moderation helps but have higher load on CPU1 as cost.
Am I need to over clock my i7-2600k to get full gigabit speed? I don't think so. What is cause this high CPU interrupt load?
What is the other people settings meanwhile have over 100MB/s settings.

5.Tera copy manually started and enable "uses system write cache" in preferences
TeraCopy still 65MB/s meanwhile windows stable 80MB/s
Upload 80-89MB/sec with windows CPU1 100%

Results: TeraCopy have limit in 65MB/s but have auto verify feature. Am I need check RoboCopy also?

6. What is our disk top load limits in DemoNAS? Check in scrub operation
zpool scrub z2test
86-120KB/t
1081-1449tps
90-136MB/sec (average 115-124MB/sec disk limit)
86-98 %busy
Interrupts 3300-4500 on mps0 282

top -P show 92-92% idle nothing on this screen

2disk data+2disk parity mean
2*100MB/s=1600Mbps that is over 1Gigabit link speed

80MB/s transfer mean
vmstat show about 20MB/sec and 25%busy per disk, 5600 interrupts in igb0:que x (x=one of the 0-7 queues) and
smal 1100interupts on msp0 (if ARC cache over)

Results: Confirmed not DemoNAS disk limit. The limitation samewhere in Win7x64 side.
Also clearly visible two simple disk mirror or 3disk raidz1 could overload a gigabit interface with permanent load (as cyberjock many times menitoned).
I like

Code:

systat -vmstat

due show everything on one screen for server internal operation (only problem limited to first 7 disk)

PS. Jperf not good still show 350Mbps meanwhile the 80MB/s=640Mbps transfer possible. Bad measurement tool. Could java limited.

What else could wrong? Why I have with 100% 3,8Ghz CPU limited to 80MB/sec? How to go over 100MB/sec?

Bye,
Jackson

 

[cyberjock]

Teracopy is crap. You can stop using. I've had to tell 3 different people to stop using it because it slows them down.. Looks like you've already proven it will help your performance to stop using teracopy.

iperf is an industry standard tool. The fact that it is not giving you 880Mb/sec+ is what you should focus on, not the fact that the value is 350Mb/sec or so. So you have a network problem that you need to resolve, but there's no hint as to what it is. You'll have to do your own isolation to isolate the problem.

 

[solarisguy]

Did you try with a different (known to be good) switch?

Different cables?

Did you try a direct connection with a cross-over cable?

 

[Peter Jakab]

Hi All,

Thanks for the reponses. I made more additional test this night. Just I need sleep now and I will format it before upload. So I will put here tomorrow.

Bye,
Jackson

 

[Peter Jakab]

DemoNAS - MainPC direct cable connection
Fix ip for both side.Same setting as in Interupt Moderation Off above. Switch is excluded from transfer above

Samba copy from demoNAS to MainPC 59-68MByte/sec
From MainPC to demoNAS 84-91Mbyte/sec

NAS iperf server and MainPC jperf single

Show : Iperf output-1

bin/iperf.exe -c 192.168.10.34 -P 1 -i 1 -p 5001 -f m -t 10
------------------------------------------------------------
Client connecting to 192.168.10.34, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[156] local 192.168.10.9 port 50684 connected with 192.168.10.34 port 5001
[ ID] Interval Transfer Bandwidth
[156] 0.0- 1.0 sec 67.5 MBytes 566 Mbits/sec
[156] 1.0- 2.0 sec 69.1 MBytes 580 Mbits/sec
[156] 2.0- 3.0 sec 69.5 MBytes 583 Mbits/sec
[156] 3.0- 4.0 sec 70.3 MBytes 590 Mbits/sec
[156] 4.0- 5.0 sec 70.8 MBytes 594 Mbits/sec
[156] 5.0- 6.0 sec 69.8 MBytes 586 Mbits/sec
[156] 6.0- 7.0 sec 69.3 MBytes 581 Mbits/sec
[156] 7.0- 8.0 sec 69.3 MBytes 581 Mbits/sec
[156] 8.0- 9.0 sec 69.5 MBytes 583 Mbits/sec
[156] 9.0-10.0 sec 69.4 MBytes 582 Mbits/sec
[156] 0.0-10.0 sec 695 MBytes 582 Mbits/sec
Done.

CPU1 on MainPC is about 90%
demoNAS console shell show 583Mbit/sec

Show : Dual Iperf test-

bin/iperf.exe -c 192.168.10.34 -P 1 -i 1 -p 5001 -f m -t 10 -d -L 5001
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.10.34, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[176] local 192.168.10.9 port 50696 connected with 192.168.10.34 port 5001
[192] local 192.168.10.9 port 5001 connected with 192.168.10.34 port 62746
[ ID] Interval Transfer Bandwidth
[176] 0.0- 1.0 sec 16.1 MBytes 135 Mbits/sec
[192] 0.0- 1.0 sec 69.6 MBytes 584 Mbits/sec
[176] 1.0- 2.0 sec 16.3 MBytes 137 Mbits/sec
[192] 1.0- 2.0 sec 72.7 MBytes 610 Mbits/sec
[176] 2.0- 3.0 sec 16.3 MBytes 137 Mbits/sec
[192] 2.0- 3.0 sec 73.6 MBytes 618 Mbits/sec
[176] 3.0- 4.0 sec 16.1 MBytes 135 Mbits/sec
[192] 3.0- 4.0 sec 72.1 MBytes 605 Mbits/sec
[176] 4.0- 5.0 sec 16.0 MBytes 134 Mbits/sec
[192] 4.0- 5.0 sec 72.3 MBytes 606 Mbits/sec
[176] 5.0- 6.0 sec 16.1 MBytes 135 Mbits/sec
[192] 5.0- 6.0 sec 72.4 MBytes 607 Mbits/sec
[176] 6.0- 7.0 sec 16.1 MBytes 135 Mbits/sec
[192] 6.0- 7.0 sec 72.3 MBytes 606 Mbits/sec
[176] 7.0- 8.0 sec 16.1 MBytes 135 Mbits/sec
[192] 7.0- 8.0 sec 72.4 MBytes 607 Mbits/sec
[176] 8.0- 9.0 sec 16.1 MBytes 135 Mbits/sec
[192] 8.0- 9.0 sec 72.3 MBytes 607 Mbits/sec
[176] 9.0-10.0 sec 16.1 MBytes 135 Mbits/sec
[192] 9.0-10.0 sec 72.2 MBytes 606 Mbits/sec
[ ID] Interval Transfer Bandwidth
[176] 0.0-10.0 sec 161 MBytes 135 Mbits/sec
[192] 0.0-10.0 sec 722 MBytes 605 Mbits/sec
Done.

MainPC CPU1 100%
demoNAS console shell show
135Mbit/sec
606Mbit/sec

Reverse config
MainPC iperf server and demoNAS iperf client

Single direction
demoNAS "iperf -c 192.168.10.9" show 666Mbit/sec

Show : MainPC jperf

bin/iperf.exe -s -P 0 -i 1 -p 5001 -f m
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[176] local 192.168.10.9 port 5001 connected with 192.168.10.34 port 45242
[ ID] Interval Transfer Bandwidth
[176] 0.0- 1.0 sec 78.0 MBytes 654 Mbits/sec
[176] 1.0- 2.0 sec 78.7 MBytes 660 Mbits/sec
[176] 2.0- 3.0 sec 78.7 MBytes 660 Mbits/sec
[176] 3.0- 4.0 sec 79.5 MBytes 667 Mbits/sec
[176] 4.0- 5.0 sec 79.7 MBytes 669 Mbits/sec
[176] 5.0- 6.0 sec 79.8 MBytes 669 Mbits/sec
[176] 6.0- 7.0 sec 79.8 MBytes 670 Mbits/sec
[176] 7.0- 8.0 sec 79.9 MBytes 670 Mbits/sec
[176] 8.0- 9.0 sec 79.8 MBytes 670 Mbits/sec
[176] 9.0-10.0 sec 79.9 MBytes 670 Mbits/sec
[176] 0.0-10.0 sec 794 MBytes 665 Mbits/sec

Cable replaced to anothe: results same
Cable replaced from 15meter down to 1meter: results same

Because PCIEx1_1 shared IRQ with SATA try to move card.
Remove my PATA controller from PCIEx16_3 (in X1 mode actually. I dont know why called x16_3 meanwhile the configurable max speed is x4:) and put Intel PRO/1000 CT into this slot

I make fault: forget the new card start with default settings so I have to remove same test with bad settings.
Interupt moderation=OFF
Samba from MainPC to demoNAS 73-89MByte/sec
MainPC CPU1 100%

From MainPC to demoNAS
60-91MByte/sec slow stepping up
MainPC CPU1 100%

So the direct cable connection, excluding switch not help. But what I see that was interesting. The TCP traffic changes over the time. First time I see traffic start with 103MByte/sec after 1 sec down to 73MByte/sec and goes down to 36MByte/sec.
Next time the traffic slowly goes up as I noted above.

TCP protocol uses the available bandwidth based on packet loss results. They try overload bandwith and detect point when connection uses maximal available space. Lot of traffic shaping, rate limiting methods.
So this is could mean for me I have network limits/packet loss/stops which cause TCP goes down to lower bandwith. I need determine which limit causes this problem.

Jackson

 

[solarisguy]

CPU should not have 100% load. If it does, that is your limitation.

 

[anodos]

CPU should not have 100% load. If it does, that is your limitation.

I must say I've never seen iperf put an i7 to 90% and fail to saturate gigabit. I think something here is fundamentally broken. Perhaps the computers have been overly tuned.

Are these certified, new CAT5e / CAT6 cables? (I don't think I've ever seen a custom cable that worked perfectly). Is it in an industrial environment with lots of heavy machinery? :)

 

[Peter Jakab]

I must say I've never seen iperf put an i7 to 90% and fail to saturate gigabit. I think something here is fundamentally broken. Perhaps the computers have been overly tuned.

Are these certified, new CAT5e / CAT6 cables? (I don't think I've ever seen a custom cable that worked perfectly). Is it in an industrial environment with lots of heavy machinery? :)

Thanks for the response. Your both of them could right. Anybody have real measured CPU utilitization information about top speed file transfer how many core and utilitization visible?
Generally my MainPC not over clocked. ASUS TurboV EVO show 100Mhz BCLK, CPU 1.165V, DDR Volt 1.65, CPU ratio 38x (mean 3800Mhz), Memory used SPD clocking (4qty of Corsair CMX8GX3M2A1600C9, XMP-1600 mode 800Mhz 9-9-9-24).

The two 15m cross cables is labelled as:
E87647-DG 2835 5U AWM 60degreeC 30V VM-1----ENHANCED 350Mhz 26AWG 4PR STP EIA/TIA 568 CAT5e
This is Shielded Twisted pair but I never measured for UTP Cat5e standard.

Jackson, who will looking after this topic also

 

[solarisguy]

I do not know your BIOS..., you may want to verify whether your BIOS has any options that could be related to your Intel PRO/1000 CT network card or behaviour of PCI Express in general.

 

[Peter Jakab]

Hi All,

Thanks for the help. I close to final solution. I am working on that since 4-5 hours. I see more than 100Mbyte/sec Samba file transfer.
Of course I will write down what I made just I would like get stable state (I expect about 1-2 hour later).

Thanks for your help (which is realy help me):
Jackson

 

[Peter Jakab]

The final solution of the problem:
When I happy to see this error solved after 6-7 hours of checking (just today). Then I understand now why other peoples don't have same type of problem.

I was two way out of the trap: network side problem cause this issue or could inside my PC.

My main problem was this is the first over 100MByte/sec copy in my system. So I unable compare to others (just FreeNAS community member results).
Before you mis understand I have 3 other system which is also not provide over 100MByte/sec samba copy.

For network troubleshooting I need Wireshark or any instrument to check it. I am network engineer able to do that just this is very long way.
So I started with narrow down the CPU problem first. I tested lot of network configuration of Intel Gigabit CT settings. This page was my friend to check deeply what they mean and do: http://www.intel.com/support/network/sb/cs-025841.htm
But that is mean 10-20Mbps difference not more. OK then I go below to interface Clients, Services and Protocols.

The bug I found if I disable all unnecessary Clients, Protocols and Services in NIC adapter properties he speed goes over 100MBytes/sec.
Ahhh first time I see that (in my home environment :)
So I check one by one each protocol/service/clients which cause that.

My personal firewall ESET Smart Security caused this problem. I would like know why. I disabled the personal firewall module (inside GUI) but that is not help just if I remove them from interface. No problem that is totally official software for me and I also reseller to them so I have tech support. Why limit 535Mbps (half gigabit bandwidth)
I checked what is other peoples measurements on net when I see in throughput for Smart Security 8 (actual version). Just looking into my version on screen Smart Security 4. Whaaaaat?

For your understanding: I am network/security expert. But at home where I long term relationship with ESET. I also used McAfee (do you remember scan.exe vshield.exe in 8086) and official Symantec but both of them was more worst than ESET Antivirus and Smart Security todays (more stable, fast and secure mean good filtering of attacks). I renewed license this year January and I spend lot of time to upgrade my family about 15-20 PC, MAC and smart phones to new license (we are SOHO company from that point of view).
On my own MainPC I simple put the new license code and works. Why works (I don't have infections since 10th of years) because the virus database and same modules update also today and HW firewall also works. Just the big framework not upgraded yet. :D

So after consultation with Tech Support (OK guys we solved lot of problems in past but at now I am blaming my self as: how can over jumping 3-4 version question) I updated to Smart Security 8.
This is not easy today. If you simple remove personal firewall/malware protection/anti-virus independently are you behind HW firewall you could infected under short period of time also. So I separated my PC & kill Internet access on HW firewall and remove/install. Are you not able start again due license communication and new updates necessary. So install demo version enable up and configure, connect back to net register and update. Fine tunnel Firewall, HIPS and anti-virus perform first scans (I hope no new virus/malware/worms meanwhile).

Check traffic again tataaaa 107-117MByte/sec. So that is the cause why nobody else have this problem. Just I was the stupid who not updated the framework on my own MainPC. Meanwhile the full family all devices running on latest version:):)
Now I can check and compare the all other devices have full Gigabit bandwidth capability and correct them if not.

Guys thanks for the help/support I wasting lot of work hours (20-40h) to get the stable high speed as you already have. When I put up the post I was very frustrated. Without you I possible don't fix it out.

PS.: my old freenas still just 60-70Mbyte/sec but they also have memory/CPU limitations and Uhbuntu have missing Intel CT card.
Jackson