Can't Ping w/o DHCP, Network ARP or Route Issue

[Steve@xxx.xxx.xxx.xxx]

FreeNAS-8.3.1-RELEASE-p2-x64
HP DL380 G5 Dual Xeon Quad Core 5450 3GHz 8GB ECC RAM, E200 RAID Controller, 3-146GB 10k SAS in RAID1 with spare, 2 NC373i Multifunction Gigabit Network Adapters with TCP/IP Offload Engine

On initial setup w/DHCP I get interfaces bce0 on 192.168.1.92 and bce1 on 192.168.1.93 and can ping both and get into GUI from a laptop on same subnet 192.168.1.0/24. The goal is a small iSCSI SAN as part of an Oracle VM virtualization project. I have been able to successfully set that up and see LUNS from the Oracle side with MPIO, but now I want to use different subnets to separate the iSCSI traffic from other network traffic with VLANS.

I'm using a Cisco 3550 L3 Switch with IP routing enabled. Switch Virtual Interfaces (SVIs) IPs are setup as 192.168.1.78/24, 10.0.10.254/24, 10.0.20.254/24, 10.0.30.254/24, 10.0.40.254/24, 10.0.50.254/24 and 10.0.60.254/24. All switchports are setup as trunks and I get the same results with or without VLANs configured in FreeNAS.

I've added IP address 10.0.50.1 to bce0 and 10.0.60.1 to bce1 which I want to use for the MPIO, so now the network summary looks like this in FreeNAS
Name IPv4 Address
bce0 10.0.50.1/24
bce0 192.168.1.92/24
bce1 192.168.2.93/24 DHCP had this at 192.168.1.93 but GUI forced change when adding IF IP 10.0.60.1
bce1 10.0.60.1/24
Nameserver
192.168.1.254
Default route
192.168.1.78
I can ping from the CLI on FreeNAS and the switch, to all hosts on all networks except for the two I need, and from the laptop to all hosts on all networks except the two FreeNAS MPIO IPs that I need to reach at 10.0.50.1 and 10.0.60.1. If I change bce0 from 10.0.50.1 to 10.0.40.1, then I can ping 10.0.50.254 from FreeNAS but not the 10.0.40.254 with console message of
ping: send to: Host is down

Looking at Packet traces with Wireshark on the laptop, when trying to ping 10.0.50.1 or 10.0.60.1, I can see the 3550 broadcasting ARP requests "who has 10.0.50.1?, Please tell 10.0.50.254" but there is no ARP reply from the FreeNAS interface at 10.0.50.1 and that seems to be the problem.

The netstat -rn output looks like this
Internet: Destination Gateway Flags Netif
10.0.50.0/24 link#1 U 0 4 bce0
10.0.50.1 link#1 UHS 0 0 lo0
10.0.60.0/24 link#2 U 0 0 bce1
10.0.60.1 link#2 UHS 0 0 lo0
38.229.71.1
192.168.1.254 UGHD3 0 16 bce0 3504
50.116.27.42
192.168.1.254 UGHD3 0 16 bce0 3504
127.0.0.1 link#9 UH 0 14159 lo0
192.168.1.0/24 link#1 U 0 3215 bce0
192.168.1.92 link#1 UHS 0 0 lo0
192.168.2.0/24 link#2 U 0 0 bce1
192.168.2.93 link#2 UHS 0 0 lo0
198.60.22.240 192.168.1.254 UGHD3 0 16 bce0 3504

Internet6: Destination Gateway Flags Netif Expire
::/96 ::1 UGRS lo0
::1 link#9 UH lo0
::ffff:0.0.0.0/96
::1 UGRS lo0
fe80::%lo0/64 link#9 U lo0
fe80::1%lo0 link#9 UHS lo0
ff01::%lo0/32 fe80::1%lo0 U lo0
ff02::%lo0/32 fe80::1%lo0 U lo0

Thanks in advance for your help.

 

[Steve@xxx.xxx.xxx.xxx]

I don't know for sure if the packet is getting to the bceX interfaces through the switch or not. I need a way to find out what packets are hitting the interfaces to see if they are being dropped or just not getting there, does anyone know if tcpdump will show that?

In the meantime I'm going to post my switch config on a Cisco forum and see if someone there can see something wrong.

 

[cyberjock]

Normally, if you are like many other people, the issue is that you have not properly setup the FreeNAS server's network configuration when you choose to not use DHCP.

There is a whole host of reason why it may not work without DHCP either. For example, some routers won't let you use static IPs to connect to the WAN without setting up the static IP on the router and leaving the server in DHCP mode.

 

[Steve@xxx.xxx.xxx.xxx]

It turns out it was the switch config, once I changed the switchport from trunk mode to access mode on the two ports going to FreeNAS, I was able to get access. Now the next step will be to see if the linux servers with bond ports and subinterfaces will pass traffic to multiple vlans across a trunk port....
Update: I have Linux and Windows machines connected to trunk ports working fine, I expect my error was not updating the Gateway to the new subnet. I will test later and confirm.

 

[Steve@xxx.xxx.xxx.xxx]

It turns out it was the switch config, once I changed the switchport from trunk mode to access mode on the two ports going to FreeNAS, I was able to get access. Now the next step will be to see if the linux servers with bond ports and subinterfaces will pass traffic to multiple vlans across a trunk port....
Update: I have Linux and Windows machines connected to trunk ports working fine, I expect my error was not updating the Gateway to the new subnet. I will test later and confirm.

 

[Steve@xxx.xxx.xxx.xxx]

So it seems like a bug to me, Linux and Windows do not have this problem, just FreeNAS. Why can't freenas communicate over a trunk port?

 

[cyberjock]

In many cases Windows does things it shouldn't do because the average user is to incompetent to make things work with the proper settings. In those examples Windows worked when it shouldn't have while FreeBSD didn't work with the same broken settings. The issue is because Windows doesn't work properly. This was discussed in great depth back in Feb/March time frame.

I count vouch for Linux though.

What would be interesting would be to duplicate your network settings on FreeBSD 9.1 just to see if it is just as broken. If not, then FreeNAS is broken somehow and a bug report should be filed.

 

[Setius]

Steve, can you post your switch config please? I'm curious to see what you are doing.

 

[Steve@xxx.xxx.xxx.xxx]

Sure, as it turns out I expect that running FreeNAS on access ports is probably better than trunks as the FreeNAS iSCSI server really only needs to communicate in one vlan anyway, where other servers are using bonded ports with multiple vlans. However if I decide to add other services later, I may want to have multiple VLANs, but I haven't seen anything to indicate that FreeNAS supports bonds with multiple subnets in separate vlans. This isn't the final config, I still need to add more security and QoS

Relevant FreeNAS access port config on gi0/1 and gi0/2 is:

interface range gi0/1-2
switchport access vlan 50 ! if this is changed to switchport trunk encapsulation dot1q along with changing next line per comment, no traffic goes through
switchport mode access ! if this is changed to switchport mode trunk along with above change no traffic goes through
!switchport nonegotiate ! added this line only with trunk mode
channel-group 1 mode passive ! this was added later when I switched from MPIO to LACP, haven't tried LACP over trunk

!!!!!! Current Complete Config Below !!!!!!!

OVM_Switch1#show run
Building configuration...

Current configuration : 4967 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname OVM_Switch1
!
enable secret 5 $1$R5sW$sNsYQngSOlDL52Um5xA6O1
!
no aaa new-model
ip subnet-zero
ip routing
no ip domain-lookup
!
vtp mode transparent
!
!
crypto pki trustpoint TP-self-signed-3761618688
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3761618688
revocation-check none
rsakeypair TP-self-signed-3761618688
!
!
crypto pki certificate chain TP-self-signed-3761618688
certificate self-signed 01
30820244 308201AD A0030201 02020101 300D0609 2A864886 F70D0101 04050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 33373631 36313836 3838301E 170D3933 30333031 30303035
30315A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 37363136
31383638 3830819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
8100D535 9D688BE7 8A6A9AD8 8BEE13BF C0B58083 33CA91FF 10112AF6 31AD0BA3
F8A0609F A8067866 172620F2 8C07C6D9 4D757D25 C8C3D618 5E9F98C2 603AF35A
7526F7CC 7E5FCBAA 991B232D EB0E4C41 02B78BF1 74E61264 442D2432 C1B9F964
A73B8342 5F49F328 A6D8C783 1B2CBC68 F97F0ABD 5981684D CEF65E5C 25A4D125
D5D70203 010001A3 6C306A30 0F060355 1D130101 FF040530 030101FF 30170603
551D1104 10300E82 0C4F564D 5F537769 74636831 2E301F06 03551D23 04183016
8014C70C 91D252DC CFD6DAA3 0652EDD0 7FE20E44 0B9A301D 0603551D 0E041604
14C70C91 D252DCCF D6DAA306 52EDD07F E20E440B 9A300D06 092A8648 86F70D01
01040500 03818100 7DF38C6C 1ECB02E6 998174E3 A2182FCC 3D291879 A1E8E6E4
A1884018 4696B7A2 FF83D7C2 42A6725B FE139EB7 6BB82D8D D3043722 488A89B8
6679C4E1 CF5DAD5B CB8B25F4 8D7AE324 0804B467 677D4A78 D519EF8F 79182C05
33BB56D9 DA177AEE 8D6FFECC 95A71909 7D701463 9D451E52 FA17D2DD FC0EFBFE
30F7B839 CFEBDCF0
quit
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
vlan 10
name Management
!
vlan 20
name LiveMigrate
!
vlan 30
name ClusterHeartbeat
!
vlan 40
name VirtualMachine
!
vlan 50
name iSCSI_MPIO_Ch1 ! name needs to be changed no longer using MPIO
!
!
!
!
!
!
!
interface Port-channel1
switchport mode dynamic desirable
!
interface Port-channel2
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
!
interface Port-channel3
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
!
interface GigabitEthernet0/1
switchport access vlan 50
switchport mode access
channel-group 1 mode passive
!
interface GigabitEthernet0/2
switchport access vlan 50
switchport mode access
channel-group 1 mode passive
!
interface GigabitEthernet0/3
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
channel-group 2 mode passive
!
interface GigabitEthernet0/4
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
channel-group 2 mode passive
!
interface GigabitEthernet0/5
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
channel-group 3 mode passive
!
interface GigabitEthernet0/6
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
channel-group 3 mode passive
!
interface GigabitEthernet0/7
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
!
interface GigabitEthernet0/8
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
!
interface GigabitEthernet0/9
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
!
interface GigabitEthernet0/10
switchport trunk encapsulation dot1q
switchport mode trunk
switchport nonegotiate
!
interface GigabitEthernet0/11
switchport mode dynamic desirable
shutdown
!
interface GigabitEthernet0/12
switchport mode dynamic desirable
shutdown
!
interface Vlan1
ip address 192.168.1.78 255.255.255.0
!
interface Vlan10
ip address 10.0.10.254 255.255.255.0
!
interface Vlan20
ip address 10.0.20.254 255.255.255.0
!
interface Vlan30
ip address 10.0.30.254 255.255.255.0
!
interface Vlan40
ip address 10.0.40.254 255.255.255.0
!
interface Vlan50
ip address 10.0.50.254 255.255.255.0
!
!
ip classless
ip route 0.0.0.0 0.0.0.0 192.168.1.254
ip route 10.0.10.0 255.255.255.0 Vlan10
ip route 10.0.20.0 255.255.255.0 Vlan20
ip route 10.0.30.0 255.255.255.0 Vlan30
ip route 10.0.40.0 255.255.255.0 Vlan40
ip route 10.0.50.0 255.255.255.0 Vlan50
!ip http server
!ip http secure-server
!
!
!
control-plane
!
!
line con 0
exec-timeout 0 0
password 7 00071A150754
logging synchronous
login
line vty 0 4
exec-timeout 0 0
password 7 14141B180F0B
logging synchronous
login
line vty 5 15
exec-timeout 0 0
password 7 14141B180F0B
logging synchronous
login
!
end

OVM_Switch1#wr mem
Building configuration...
[OK]
OVM_Switch1#

 

[Setius]

Switch config looks ok, besides these routes. No need for these since they are already in your route table as directly connected interfaces.

ip route 10.0.10.0 255.255.255.0 Vlan10
ip route 10.0.20.0 255.255.255.0 Vlan20
ip route 10.0.30.0 255.255.255.0 Vlan30
ip route 10.0.40.0 255.255.255.0 Vlan40
ip route 10.0.50.0 255.255.255.0 Vlan50

To be honest, If you intend to do iSCSI MPIO, there is no need for a LAG setup. A properly setup MPIO solution has HA built into via the MPIO. I would use two dedicated NIC for iSCSI and one or two for mgt/client access. For your VM host, I would recommend two dedicated NIC for iSCSI also in the same manor.

example
NIC1 iSCSI "ip 10.0.50.x no gateway"-> 3550 g0/1 "sw access vlan 50" "sw mode access"
NIC2 iSCSI "ip 10.0.60.x no gateway"-> 3550 g0/1 "sw access vlan 60" "sw mode access"
NIC3 Mgt "ip 192.168.1.x gateway 192.168.1.78"-> 3550 g0/x "sw access vlan 1" "sw mode access"


If you insist on passing vlans over LAG, a few quick qoogle shows several bugs in drivers in the 8.x code for freebsd. You might have better luck with FreeNAS 9.1 like Cyber mentioned.

just my .02

 

[Steve@xxx.xxx.xxx.xxx]

Thanks for the feedback, not sure where I got the idea to add those extra route commands - my first time setting up L3 switch.

Originally the plan was to use MPIO and that was all setup on FreeNAS; however, due to having only 2 NICs to work with and limitations in Oracle VM where MPIO on bonds are incompatible with the Virtual Machine role on tagged VLAN interfaces, MPIO was not an option which forced the change to LACP. More NICs and 10GB switch with more ports would be nice, but with budget issues for now, stuck working with 2 NICs per server and 1GB. Still not sure why FreeNAS wouldn't communicate across trunk ports when I had MPIO setup and hopefully I won't need trunk ports.

I wasn't planning to use 9.1 yet as for what I'm doing didn't see anything new I had to have nor did I see any mention of bug fixes I needed. I was also concerned that 9.1 was so new, maybe I should just wait. I wonder if the driver bugs were fixed with 9.1? Well since my flash drive/OS died and needs to be reinstalled anyway, I guess I could burn a 9.1 CD and install that. I wonder will I be able to use my system backup from 8.3 or will I need to just reconfigure under 9.1?

Thanks again

 

[Steve@xxx.xxx.xxx.xxx]

So I installed 9.1, tried to configure lagg for about an hour and failed. probably because I had manually setup NIC interfaces. (Should have read docs better!) I tried setup through shell with ifconfig but gui doesn't recognize ifconfig settings, and I don't believe they survive reboot without editing rc.conf which maybe corrupted the OS last time so I didn't want to do that. The docs for 9.1 say make sure no interfaces are manually configured or lagg will fail, so I presume I would have to start with DHCP. Part of the problem is I don't have a router yet, just a L3 switch and although I could setup switch to allow interfaces to get DHCP from the gateway, I didn't realize I needed to start with a DHCP setup. So after waiting a long time for DHCP and NTP to fail, the 9.1 system came up. Without a router, my gateway can't do NAT for the subnet I'm using for my VLAN. So I just restored my old 8.3 configs and after reboot, everything seemed to be fine except Web address IP was blank but I was still able to get into the GUI so I entered the IP I was using, saved it, exported a backup and now trying to reboot again, it is taking a long time. The only service I had setup in 8.3 was iSCSI and the portal had an address left over from my MPIO setup so I had to delete the portal which deleted some of the other iSCSI setup but that will only take a minute to recreate.

If it comes back up then I will reconfigure switch to allow DHCP from the gateway, restore factory settings and and try following the 9.1 docs to see what happens. If it doesn't come back up I guess the OS is hosed again and will just reinstall 9.1 again with DHCP setup on switch and see what happens. Strange how it came back up after the restore but won't reboot now. Hope that router comes tomorrow. Still hasn't booted, so thinking back, originally I had MPIO all setup with interfaces manually configured which may be why lagg setup failed, So I guess I will find out if the moral of the story is you can't just switch from MPIO to LACP, you have to start over from a DHCP setup. Ctrl-Alt-Delete won't reset server, same as last time which is when I pulled the plug and/or hit the power button which maybe hosed the OS more than editing the rc.conf.

 

[Steve@xxx.xxx.xxx.xxx]

At the end of paragraph 5.3 the Docs say "NOTE: the FreeNAS® system must be rebooted after configuring the lagg device and TCP access will be lost during reboot. Do not configure the interfaces used in the lagg device before creating the lagg device." Then in 5.3.2 the docs say "Select the desired aggregation protocol, highlight the interface(s) to associate with the lagg device, and click the OK button.
Once the lagg device has been created, it will be listed in the tree under an entry which indicates the type of protocol. As seen in Figure 5.3b, it will also appear in View Link Aggregations."

May I suggest that 5.3.2 be updated right after it says "and click the OK button' to mention something to the effect that...

"Connection to the web interface will be lost at this point and the system must be rebooted from the terminal screen. You may also have to change your switch settings to communicate through the new lagg interface. After reboot you may also have to enter a default gateway from the terminal screen in order to get back into the GUI.

Once back into the GUI, the lagg device will be listed in the tree under an entry which indicates the type of protocol. As seen in Figure 5.3b, it will also appear in View Link Aggregations."

Like MPIO, LACP still only works through a switchport in access mode, if the switchport is changed to a trunk communication fails. Perhaps this should also be noted until a work around or fix is available!

Also it would be nice to be able to change the aggregation protocol type after lagg creation rather than having to delete the lagg and start over.

 

[cyberjock]

I'll add it Steve. It seems straight forward, to me.. but I don't see any reason to not include it. :P

 

[Steve@xxx.xxx.xxx.xxx]

Another problem I guess I'll have to create a new thread for this tomorrow, too tired now. After creating a zfs volume the system rebooted by itself. When it came back up the terminal didn't offer an IP to access the GUI. The ifconfig looks OK and the default gateway is still there, and I can ping the gateway but there is no access to the web interface. Could it be that when I went into system settings and put in the web IP address, that I didn't specify a port because I thought it would just default to 80? Why was the Web address under system settings blank when the lagg had an IP and I was in the web interface. I think I might just go back to 8.3 now that I know how to setup the lagg unless someone can tell me how to get back in...

Thanks for your help. Good Night.

PS I wonder what option 7 on the terminal "Reset WebGUI login credentials does", I haven't set any credentials but maybe it will fix my access problem? Nope still locked out.

 

[cyberjock]

You don't have to set a port if you do http or https. Those are "defaulted" to the appropriate ports unless you are actually using a different port, in which case you add the appropriate port.

 

[cyberjock]

And the Reset WebGUI resets the admin account for the webui and the password if you forgot it.

 

[Steve@xxx.xxx.xxx.xxx]

Hello Cyber Jock,

Hold that edit, there was one more step before I lost connection. Now this is this morning on 8.3 because 9.1 was no longer accessible from the GUI after setting up a ZFS vol so I went back to 8.3, but now I remember 9.1 was the same and I forgot to mention a step.

After pressing OK, I was able to edit the lagg interface group. After adding the new IP to the lagg, that is when connection to the GUI is lost and the system needs to be rebooted. In my case my DHCP addresses were 192.168.1.92 and 192.168.1.93, on bce0 and bce1, and after creating the lagg with bce0 and bce1, then setting the IP to 10.0.50.100, connection is lost so I rebooted, changed switch config to support lagg and went into the terminal to set the default gateway to match the lagg subnet, then I was back in through the GUI at 10.0.50.100 after reboot. I'm not sure what would happen if I had tried to set the lagg to 192.168.1.92 0r 93?

Also noting that before reboot ifconfig shows bce0 and bce1 still at 192.168.1.92, 93, lagg at 10.0.50.100 with bce0,1 in laggport and default route at 192.168.1.254.
After reboot, ifconfig shows no IP address on bce0, bce1, lagg at 10.0.50.100. I was able to access the GUI at 10.0.50.100 in a different browser but Firefox gave me an error, tried clearing session cookies but that didn't help..

Request Method: GET
Request URL: http://10.0.50.100/
Software Version: FreeNAS-8.3.1-RELEASE-p2-x64 (r12686+b770da6_dirty)
Exception Type: SuspiciousOperation
Exception Value:
Invalid characters in session key
Exception Location: /usr/local/lib/python2.7/site-packages/django/contrib/sessions/backends/file.py in _key_to_file, line 43
Server time: Mon, 19 Aug 2013 08:48:38 -0700

 

[cyberjock]

Here's the page I edited, let me know what and where you think it should be edited and I'll make the appropriate changes. I've never messed with lagg connections in FreeNAS and I'd rather not do 1/2 a job. :)

 

[cyberjock]

Oops. Link: http://wiki.freenas.org/index.php/Link_Aggregations