Ping replies takes the wrong way

[Revir]

I have a network setup with two LANs, SYS and INT, connected via a pfSense firewall.
Firewall permits SYS to access INT but blocks INT from accessing SYS.
My FreeNAS has two network interfaces, the webGUI is on SYS, from now on called ifSYS,
and the "share" interface is on INT, from now on called ifINT.
When both ifSYS and ifINT are connected and I ping ifINT from SYS I get replies but
when I disconnect ifSYS I don't get ping replies any more.
Still firewall and computers on INT get ping replies from ifINT when ifSYS is disconnected.
FreeNAS default gateway is set to firewall INT interface but I've tried without default gateway and
also default gateway set to firewall SYS interface but that doesn't make any difference.
If I do a pfSense packet capture on ifINT when pinging it from firewall I can see both requests and replies,
but when I ping ifINT from SYS I only see the requests, no replies.
That the firewall don't see the replies indicates that FreeNAS is bypassing the firewall, replying directly
from ifSYS and not from the pinged interface, ifINT.
Is that really how it should be? Shouldn't each interface be independent?
Does this mean that the FreeNAS actually has routing functionality activated?
Is there any way to modify the configuration so that packets go the right way?

I'm running FreeNAS 11.3 on a "maxed out" PowerEdge 2900 gen.III and my switches are Catalyst 2960-G
...and here's the ifconfig output:

Show : ifconfig output:

bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: member of lagg0
options=c01ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 00:1b:21:aa:c7:c8
hwaddr 00:1e:4f:30:47:16
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: member of lagg0
options=401ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:1b:21:aa:c7:c8
hwaddr 00:1b:21:aa:c7:c8
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: member of lagg0
options=401ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:1b:21:aa:c7:c8
hwaddr 00:1b:21:aa:c7:c9
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: member of lagg0
options=401ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:1b:21:aa:c7:c8
hwaddr 00:1b:21:aa:c7:cc
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: member of lagg0
options=401ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:1b:21:aa:c7:c8
hwaddr 00:1b:21:aa:c7:cd
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: sys
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 00:1e:4f:30:47:14
hwaddr 00:1e:4f:30:47:14
inet 192.168.100.27 netmask 0xffffff00 broadcast 192.168.100.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: int
options=401ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:1b:21:aa:c7:c8
inet 192.168.101.56 netmask 0xffffff00 broadcast 192.168.101.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp lagghash l2,l3,l4
laggport: bce0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

 

[sretalla]

That the firewall don't see the replies indicates that FreeNAS is bypassing the firewall, replying directly
from ifSYS and not from the pinged interface, ifINT.
Is that really how it should be? Shouldn't each interface be independent?
Does this mean that the FreeNAS actually has routing functionality activated?

FreeBSD (and hence FreeNAS) certainly uses routing table knowledge to select the right interface for a packet based on the target address.

The fact that FreeNAS has an address on the subnet of ifINT means any packet destined for an address on that subnet will go straight out ifINT.

when I disconnect ifSYS I don't get ping replies any more.

Because you aren't sending the packets out...? (source interface disconnected)

Perhaps this whole thing is down to subnet addressing... are SYS and INT actually separate, routable IP subnets? (or are they just different addresses on the same subnet?)

Is there any way to modify the configuration so that packets go the right way?

Have a look at the example section here:

route

www.freebsd.org

(also some tools like route show ... that may help you troubleshoot)

Making any changes permanent here will need converting to an init script or tunable depending what you want to have happen.

 

[Revir]

Thanks for your reply sretalla
FreeBSD is of course an operating system with loads of configuration settings but I expected that if you bundle it with a product the settings would be appropriate for the product. I consider ping being a very useful tool to make sure you have contact with a network interface and when ping doesn't reply there's something wrong, I started to reconfigure my firewall to find and correct the error. If this routing mechanism had stopped this unorthodox method when the interface was down and instead used the "normal" route I wouldn't have had this mess.
My packets where sent out correctly through the firewall and into the INT LAN, that's what pfSense packet capture reported. Neither with nor without FreeNAS SYS interface up the packets reached FreeNAS INT interface but pfSense packet capture never reported any replies. With FreeNAS SYS interface up my client got replies, when SYS interface was down I got no replies. The nets are different subnets, 192.168.100/24 and 192.168.101/24.
I will study the route command, thanks for the link. Suddenly I have loads of FreeBSD things around me (FreeNAS, pfSense and soon Raspberry Pi and at least one application server) so I better get started to learn how they work

 

[Revir]

Thanks again sretalla
I ran the command: route delete -net 192.168.100.0/24 and that made it!!!
Next question is how often I have to run it, is after every reboot enough or do FreeBSD check the routes every hour or so?
I guess I go to the FreeBSD forum for the rest of my questions, thanks a lot

 

[Kcaj]

Looking at your ifconfig output, it looks like you using link aggregation, it that intentional?

 

[Revir]

Yes, I use five 1GB NICs to create a LACP lagg. My Cisco switch handles everything correct so that's not the problem. The thing is FreeBSD tries to be smart and avoiding my firewall in at least ICMP handling and don't like that. Every basic network packet should pass the firewall. If you build something on your own, like webserver on DMZ accessing SQL-server on an internal net you should of course pass the request through the server and not via firewall but ordinary net services like ping should not take that shortcut. At least that's my opinion...

 

[Revir]

Sorry "should have course" should of course be "should of course"

 

[Revir]

Sorry again "should have course" should of course be "should of course". Training on ugly vi editor.....

 

[Revir]

WTF, I can't correct it. Should have course should be Should OF course in every message above

 

[Revir]

Incredible... "Should ofcourse" it should be

 

[Kcaj]

I ran the command: route delete -net 192.168.100.0/24 and that made it!!!

If you put your configuration back to how it was before to made this modification (maybe reboot) could you then output netstat -r and also edit your original post with the IPs of networks and interfaces then we might be able to help...

 

[Revir]

Route table shows 192.168.100.0 as a possible route, I have to destroy that line at least after every reboot to make sure ICMP packets choose the "right" (but not shortest) way . I changed one config file, don't remember the name, and now try to find it again. Tried to get some help from "ls -help" in /etc/rc.d dirctory. The answer was soo typical "usage: ls [-ABCFGHILPRSTUWZabcdfghiklmnopqrstuwxy1,] [-D format] [file ...]. Really user friendly...
How do I list files in chronological order

 

[Kcaj]

To me that sounds like its not quite right, can you still output netstat -r give the IPs? I would rather add a route then destroy it....

 

[Revir]

This is what netstat -r gives:
Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 192.168.101.1 UGS lagg0
localhost link#7 UH lo0
192.168.100.0/24 link#6 U bce1
192.168.100.27 link#6 UHS lo0
192.168.101.0/24 link#8 U lagg0
192.168.101.56 link#8 UHS lo0

Obviously it claims it has a route from 192.168.101 to 192.168.100 and that is wrong, that traffic should go through the firewall

 

[Kcaj]

Obviously it claims it has a route from 192.168.101 to 192.168.100 and that is wrong, that traffic should go through the firewall

Arr.....Im not sure

- Packets trying to get to the .100 network will go out the interface bce1 unless its for host .100.27 then the packet will be sent to the loopback interface
- Packets trying to get to the .101 network will go out the interface lagg0 unless its for host .101.56 then the packet will be sent to the loopback interface
- All other packets will be sent via lagg0 to .101.1

Are you trying to ping between .100.27 and .101.56?

Have a read of this: Gateways and Routes

 

[Kcaj]

Figure it out?

 

[Revir]

Not really, rather I have to accept it as it is. I guess that's why they invented FIB. If I understand it right FIB should make it possible to have one routing table per interface, am I right?