Traffic storm on ix interface?

[Durandal]

Hi,

A couple of weeks ago i activated my 10 GbE CX4 adapter in my FreeNAS box. I did some testing with good results. But today i checked the traffic graphs and they indicate that there's an awful ammount of information "storming" on that interface. Exactly from that date there is a constant stream of information flowing according to the graphs. I have not used them more except the initial testing, so i find it very strange. The graphs don't make sense if you check the ammount of traffic. Anyone that can give a hint of what it could be?

Attaching a "netstat" from the shell. I'm not that skilled in the FreeNAS CLI to investigate more. I've heard alot of tools like darkstat and such but i rather use the tools already installed.

Anyone that can give me a hint on where to start to troubleshoot this problem?

Hardware:

2x AOC-STG-I2 (10 GbE CX4 NICs, Intel 82598EB-based)
1x Woven Brocade LB4
1x FreeNAS server, 1x Windows 7 client

I have the 10 GbE network on a separate net (172.16.0.x) that has jumbo frames activated.

 

[dlavigne]

Post the output from ifconfig. Also, anything related in /var/log/messages?

 

[Durandal]

Here's the ifconfig:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=40098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWTSO>
ether 00:25:90:d7:59:6d
inet 192.168.0.210 netmask 0xffffff00 broadcast 192.168.0.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether 00:25:90:d7:59:6c
inet 172.16.1.1 netmask 0xffffff00 broadcast 172.16.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
ether 00:30:48:94:63:be
inet 172.16.0.210 netmask 0xffffff00 broadcast 172.16.0.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (10Gbase-CX4 <full-duplex>)
status: active
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
ether 00:30:48:94:63:bf
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:d4:a1:79:94:00
nd6 options=9<PERFORMNUD,IFDISABLED>
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair4a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 14 priority 128 path cost 2000
member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 12 priority 128 path cost 2000
member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 13 priority 128 path cost 2000
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 11 priority 128 path cost 2000
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:04:3b:00:0a:0a
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:b2:d6:00:0b:0a
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active

epair3a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:a2:58:00:0d:0a
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:e3:90:00:0c:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
epair4a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:88:4b:00:0e:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active

I cannot find anything related to this in the /var/log/messages, there's nothing indicating that stuff is happening on the 172.16.0.x-subnet at all.

 

[dlavigne]

Were you ever able to figure this out?

 

[Durandal]

No, actually not. I have not had the time to test some more i'm afraid. As soon as i activate the interface the "storming" begins.

 

[cyberjock]

Now I know why I'm getting random 10Gb/s worth of traffic from durandal.com! Geez.

On a serious note, my recommendation would be to enable the LAN port and see what port the traffic is going out on. 9.2.0 does have a vulnerability around the NTP amplification attack from a few months ago. If your FreeNAS box isn't behind a firewall or has the NTP port forwarded to the FreeNAS box you are probably being used for a DDoS attack. Also, if one of your desktops is compromised it may be triggering the NTP traffic. This all assumes your problem is related to the NTP vulnerability, which you haven't proven. I'm just speculating at what I consider to be the most likely cause.

 

[c32767a]

That's a lot of traffic. Is there any corresponding disk I/O?

What does "netstat -i 1 " produce? Just let it run about 10 seconds while the interface is enabled.

Something is fishy here.. The graph is reporting 30G. But you have a 10G interface. Not possible.


If the netstat shows a large volume of traffic, you can always log in to the cli from another interface (use SSH, not the CLI in the web page) and do a

tcpdump -i ix0 -n

If you're getting pelted with traffic, that will dump the packet headers to the CLI and the source should be obvious.
If you don't get anything back, then there's an instrumentation problem somewhere.