Odd networking dilema

Status
Not open for further replies.
O

ondjultomte

Contributor
Joined
Aug 10, 2015
Messages
117
I got FreeNAS under esxi 6 and I just added a Chelsio T420. I had to manually install the 5.5 drivers in esxi, that work out well and then I passed through the NIC to FreeNAS.

FreeNAS gui doesn't see this newly added NIC so I had to give it a IP via ifconfig


This is the iconfig
Code:
[root@freenas] ~# ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:0c:29:75:a9:ee
        inet 192.168.1.93 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:0c:29:75:a9:da
        inet 192.168.2.50 netmask 0xffffff00 broadcast 192.168.2.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:82:19:9a:9e:00
        nd6 options=1<PERFORMNUD>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000
        member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000
        member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:21:56:00:06:0a
        inet 192.168.1.94 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:12:6b:00:07:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:ab:cd:00:08:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
epair3a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:24:de:00:09:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
[root@freenas] ~#



This is the arp -a from FreeNAS
Code:
[root@freenas] ~# arp -a
? (192.168.1.94) at 02:21:56:00:06:0a on epair0a permanent [ethernet]
? (192.168.2.50) at 00:0c:29:75:a9:da on em1 permanent [ethernet]
RT-N65U (192.168.1.1) at 50:46:5d:91:11:f4 on em0 expires in 1196 seconds [ethernet]
? (192.168.1.3) at 02:6b:84:00:07:0b on em0 permanent [ethernet]
seedbox (192.168.1.2) at 02:28:50:00:09:0b on em0 permanent [ethernet]
? (192.168.1.4) at 02:89:1e:00:09:0b on em0 permanent [ethernet]
? (192.168.1.6) at 02:ba:45:00:0b:0b on em0 permanent [ethernet]
FREENAS (192.168.1.93) at 00:0c:29:75:a9:ee on em0 permanent [ethernet]
OpenELEC (192.168.1.30) at d0:63:b4:00:24:65 on em0 expires in 983 seconds [ethernet]
? (192.168.1.21) at 00:02:c9:58:3e:7a on em0 expires in 1176 seconds [ethernet]


THis is ping to my wks from FreeNAS
Code:
root@freenas] ~# ping -c3 -S 192.168.1.94 192.168.1.21
PING 192.168.1.21 (192.168.1.21) from 192.168.1.94: 56 data bytes
64 bytes from 192.168.1.21: icmp_seq=0 ttl=64 time=0.293 ms
64 bytes from 192.168.1.21: icmp_seq=1 ttl=64 time=0.338 ms
64 bytes from 192.168.1.21: icmp_seq=2 ttl=64 time=0.311 ms

--- 192.168.1.21 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.293/0.314/0.338/0.018 ms



Ony my wks I deleted the arp entry 192.168.1.94 , arp -d

Code:
Interface: 192.168.1.21 --- 0x1c
  Internet Address      Physical Address      Type
  192.168.1.1           50-46-5d-91-11-f4     dynamic
  192.168.1.2           02-28-50-00-09-0b     dynamic
  192.168.1.3           02-6b-84-00-07-0b     dynamic
  192.168.1.93          00-0c-29-75-a9-ee     dynamic
  192.168.1.123         0c-c4-7a-6d-a9-bb     dynamic
  192.168.1.176         00-0c-29-4e-e6-8c     dynamic
  192.168.1.217         0c-c4-7a-6a-e7-52     dynamic
  224.0.0.2             01-00-5e-00-00-02     static
  224.0.0.251           01-00-5e-00-00-fb     static
  224.0.0.252           01-00-5e-00-00-fc     static
  224.0.0.253           01-00-5e-00-00-fd     static
  239.255.255.250       01-00-5e-7f-ff-fa     static
  255.255.255.255       ff-ff-ff-ff-ff-ff     static


I then ping from 192.168.1.94 on freeNAs again and then check the arp entry on the wks

Code:
Interface: 192.168.1.21 --- 0x1c
  Internet Address      Physical Address      Type
  192.168.1.1           50-46-5d-91-11-f4     dynamic
  192.168.1.2           02-28-50-00-09-0b     dynamic
  192.168.1.3           02-6b-84-00-07-0b     dynamic
  192.168.1.4           02-89-1e-00-09-0b     dynamic
  192.168.1.30          d0-63-b4-00-24-65     dynamic
  192.168.1.93          00-0c-29-75-a9-ee     dynamic
  192.168.1.94          00-0c-29-75-a9-ee     dynamic
  192.168.1.123         0c-c4-7a-6d-a9-bb     dynamic
  192.168.1.176         00-0c-29-4e-e6-8c     dynamic
  192.168.1.217         0c-c4-7a-6a-e7-52     dynamic
  192.168.1.255         ff-ff-ff-ff-ff-ff     static
  224.0.0.2             01-00-5e-00-00-02     static
  224.0.0.251           01-00-5e-00-00-fb     static
  224.0.0.252           01-00-5e-00-00-fc     static
  224.0.0.253           01-00-5e-00-00-fd     static
  239.255.255.250       01-00-5e-7f-ff-fa     static
  255.255.255.255       ff-ff-ff-ff-ff-ff     static


That is not what I expected.
192.168.1.93 00-0c-29-75-a9-ee dynamic
192.168.1.94 00-0c-29-75-a9-ee dynamic

And that explains why I only get about 1 GBit in network throughtput.

C:\Program Files (x86)\jperf\bin>iperf -c 192.168.1.94
------------------------------------------------------------
Client connecting to 192.168.1.94, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[240] local 192.168.1.21 port 64463 connected with 192.168.1.94 port 5001
[ ID] Interval Transfer Bandwidth
[240] 0.0-10.0 sec 1.02 GBytes 878 Mbits/sec


The Chelsio NIC can be pinged from the host it self.
There is no IP conflict, why is IP .94 mapped to the wrong MAC?

Can I update the FreeNAS gui to see the Chelsio NIC ?
 
Last edited:
O

ondjultomte

Contributor
Joined
Aug 10, 2015
Messages
117
Hm this is why

[root@freenas] ~# arp -i epair0a -a
? (192.168.1.94) at 02:21:56:00:06:0a on epair0a permanent [ethernet]

[root@freenas] ~# arp -i em0 -a
RT-N65U (192.168.1.1) at 50:46:5d:91:11:f4 on em0 expires in 1196 seconds [ethernet]
? (192.168.1.3) at 02:6b:84:00:07:0b on em0 permanent [ethernet]
seedbox (192.168.1.2) at 02:28:50:00:09:0b on em0 permanent [ethernet]
? (192.168.1.4) at 02:89:1e:00:09:0b on em0 permanent [ethernet]
? (192.168.1.6) at 02:ba:45:00:0b:0b on em0 permanent [ethernet]
FREENAS (192.168.1.93) at 00:0c:29:75:a9:ee on em0 permanent [ethernet]
OpenELEC (192.168.1.30) at d0:63:b4:00:24:65 on em0 expires in 3 seconds [ethernet]
? (192.168.1.21) at 00:02:c9:58:3e:7a on em0 expires in 1156 seconds [ethernet]
 
O

ondjultomte

Contributor
Joined
Aug 10, 2015
Messages
117
That is odd, one would except that a arp request for 192.168.1.94 would result in that the Chelsio 02:21:56:00:06:0a would answer, not the intel GB NIC ...

freenas%20nic.PNG
 
depasseg

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
ondjultomte said:
I had to manually install the 5.5 drivers in esxi, but that work out well and then I passedthrough the NIC to FreeNAS.
Click to expand...
What's really going on? If you passed through, you should need drivers in ESX.
 
O

ondjultomte

Contributor
Joined
Aug 10, 2015
Messages
117
There is not problem with the driver, I added that manually in esxi and as you can see from the ifconfig the chelio card is there.
The problem is that all traffic is passed through the intel nic (192.168.1.93) at 00:0c:29:75:a9:ee em0, not (192.168.1.94) at 02:d8:b5:00:06:0a on epair0a

The wrong nic/mac answers the arp request for 192.168.1.94 on the network.



Odd thing tho, a MAC lookup on 02:d8:b5:00:06:0a doesn't show it belongs to Chelsio.
 
Last edited:
zambanini

zambanini

Patron
Joined
Sep 11, 2013
Messages
479
post your vmx config file here, so we can see the details
 
depasseg

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
If you have a 10GbE Chelsio T420, it should show up as cxgbe0 (or 1).

Code:
cxgbe0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:07:43:12:6c:e0
        inet 172.16.1.81 netmask 0xffffff00 broadcast 172.16.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet 10Gbase-Twinax <full-duplex>
        status: active
cxgbe1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500


My guess is that it isn't being passed through correctly. And the FreeNAS isn't detecting it properly to load the proper drivers.
 
O

ondjultomte

Contributor
Joined
Aug 10, 2015
Messages
117
Code:
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "11"
nvram = "FreeNAS.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
numvcpus = "3"
cpuid.coresPerSocket = "3"
memSize = "26624"
sched.cpu.units = "mhz"
powerType.powerOff = "soft"
powerType.suspend = "hard"
powerType.reset = "soft"
scsi0.sharedBus = "physical"
scsi0.virtualDev = "lsilogic"
scsi0.present = "TRUE"
ide1:0.deviceType = "atapi-cdrom"
ide1:0.fileName = "emptyBackingString"
ide1:0.present = "TRUE"
floppy0.startConnected = "FALSE"
floppy0.clientDevice = "TRUE"
floppy0.fileName = "vmware-null-remote-floppy"
pciPassthru0.id = "00000:002:00.0"
pciPassthru0.deviceId = "0x0086"
pciPassthru0.vendorId = "0x1000"
pciPassthru0.systemId = "5635846f-3953-7988-8787-0cc47a6ae752"
pciPassthru0.present = "TRUE"
displayName = "FreeNAS"
guestOS = "freebsd-64"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
uuid.bios = "56 4d df 69 68 c8 ec 61-54 61 86 f0 5f 75 a9 da"
uuid.location = "56 4d df 69 68 c8 ec 61-54 61 86 f0 5f 75 a9 da"
vc.uuid = "52 01 07 58 96 03 cb 1f-10 23 8e dc 2e 9a 84 0b"
chipset.onlineStandby = "FALSE"
wwn.enabled = "TRUE"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "26624"
sched.mem.minSize = "26624"
sched.mem.shares = "high"
virtualHW.productCompatibility = "hosted"
sched.swap.derivedName = "/vmfs/volumes/56358acf-e7fe5065-0210-0cc47a6ae752/FreeNAS/FreeNAS-cc02cd7e.vswp"
replay.supported = "FALSE"
replay.filename = ""
migrate.hostlog = "./FreeNAS-cc02cd7e.hlog"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "FreeNAS_1.vmdk"
scsi0:0.present = "TRUE"
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "16"
pciPassthru0.pciSlotNumber = "160"
vmci0.pciSlotNumber = "33"
vmci0.id = "1601546714"
monitor.phys_bits_used = "42"
vmotion.checkpointFBSize = "4194304"
vmotion.checkpointSVGAPrimarySize = "4194304"
cleanShutdown = "FALSE"
softPowerOff = "FALSE"
tools.syncTime = "FALSE"
extendedConfigFile = "FreeNAS.vmxf"
toolsInstallManager.lastInstallError = "0"
toolsInstallManager.updateCounter = "5"
ethernet2.virtualDev = "e1000"
ethernet2.networkName = "VM Network"
ethernet2.addressType = "generated"
ethernet2.present = "TRUE"
ethernet2.pciSlotNumber = "32"
ethernet2.generatedAddress = "00:0c:29:75:a9:ee"
ethernet2.generatedAddressOffset = "20"
ide1:0.allowGuestConnectionControl = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.networkName = "Storage networkx"
ethernet0.addressType = "generated"
ethernet0.present = "TRUE"
ethernet0.pciSlotNumber = "34"
ethernet0.generatedAddress = "00:0c:29:75:a9:da"
ethernet0.generatedAddressOffset = "0"
tools.remindInstall = "FALSE"
scsi0:1.deviceType = "scsi-hardDisk"
scsi0:1.fileName = "FreeNAS_3.vmdk"
scsi0:1.present = "TRUE"
scsi0:1.redo = ""
sched.scsi0:1.throughputCap = "off"
sched.scsi0:1.shares = "normal"
sched.cpu.affinity = "all"
ide1:0.startConnected = "FALSE"
ide1:0.clientDevice = "TRUE"
pciPassthru1.id = "00000:007:00.1"
pciPassthru1.deviceId = "0x4001"
pciPassthru1.vendorId = "0x1425"
pciPassthru1.systemId = "5635846f-3953-7988-8787-0cc47a6ae752"
pciPassthru1.present = "TRUE"
pciPassthru1.pciSlotNumber = "192"
pciPassthru3.id = "00000:007:00.2"
pciPassthru3.deviceId = "0x4001"
pciPassthru3.vendorId = "0x1425"
pciPassthru3.systemId = "5635846f-3953-7988-8787-0cc47a6ae752"
pciPassthru3.present = "TRUE"
pciPassthru4.id = "00000:007:00.3"
pciPassthru4.deviceId = "0x4001"
pciPassthru4.vendorId = "0x1425"
pciPassthru4.systemId = "5635846f-3953-7988-8787-0cc47a6ae752"
pciPassthru4.present = "TRUE"
pciPassthru5.id = "00000:007:00.4"
pciPassthru5.deviceId = "0x4401"
pciPassthru5.vendorId = "0x1425"
pciPassthru5.systemId = "5635846f-3953-7988-8787-0cc47a6ae752"
pciPassthru5.present = "TRUE"
pciPassthru6.id = "00000:007:00.5"
pciPassthru6.deviceId = "0x4501"
pciPassthru6.vendorId = "0x1425"
pciPassthru6.systemId = "5635846f-3953-7988-8787-0cc47a6ae752"
pciPassthru6.present = "TRUE"
pciPassthru7.id = "00000:007:00.6"
pciPassthru7.deviceId = "0x4601"
pciPassthru7.vendorId = "0x1425"
pciPassthru7.systemId = "5635846f-3953-7988-8787-0cc47a6ae752"
pciPassthru7.present = "TRUE"
pciPassthru3.pciSlotNumber = "256"
pciPassthru4.pciSlotNumber = "1184"
pciPassthru5.pciSlotNumber = "1216"
pciPassthru6.pciSlotNumber = "1248"
pciPassthru7.pciSlotNumber = "1280"


freenas%20conf.PNG
 
depasseg

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
I have to ask - Why are you passing the NIC through to freenas? Why not let ESX handle it?
 
depasseg

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
ondjultomte said:
No reason at all except that I wanted to test it.
Click to expand...
This issue seems like it is more of an ESX issue. Not really an "odd networking dilema".
 
Status
Not open for further replies.

Similar threads

R
  • Locked
em0 missing on bridge0 after upgrade to FreeNAS 11.1
Replies
7
Views
5K
fracai
fracai
phynias
upgrade to 11.2-U3 and now jails have no network access
Replies
2
Views
1K
phynias
phynias
M
  • Locked
iocage VMs and multiple bridges problems
Replies
9
Views
2K
short-stack
short-stack
N
  • Locked
No network access in jail on virtual box
Replies
8
Views
2K
NasKar
N
STREBLO
  • Locked
Properly creating iocage bridges for VNET
Replies
4
Views
4K
STREBLO
STREBLO
Top