FreeNAS 9.10.2-U6 - Added 10GB Intel X520-DA2 and can't get networking to work

[Itamar Croitoru]

I'd like to start by stating that this is a fairly old build but, it is still running, it's completely internal, and it doesn't need an upgrade. Since it's in production, I didn't want to upgrade it and take any chances. If I need to at some point or get around to replacing it entirely, then I'll install whatever is current.

Server: HP ProLiant DL 380e Gen8
Card: Intel X520-DA2 (Dual Port 10G card)
Build: FreeNAS-9.10.2-U6 (561f0d7a1)
Switch: MikroTik CRS317-1G-16S+

Small history:

I recently had a hardware failure with this server where it failed to boot up and lost the OS disk. The old hardware was super old and the onboard SATA controller was dying.
I bought this replacement hardware and brought over the boot SSD, the 12 x 2 TB WD drives for storage, and the LSI card in IT mode for an HBA.
As it so happens (was planned), the new server has more PCIe slots than the old one so I slapped the Intel X520-DA2 in there to start building out my replacement 10G iSCSI network to replace my current 1G iSCSI network.

What worked:

After booting the FreeNAS up on the new hardware I ignored the 10G card at first and went through the process of reviewing everything I could think of to make sure the FreeNAS was working right and it all seemed well.
Eventually, I turned to the new 10G parts and plugged in the DAC cables between the X520 and the switch and started to configure the switch.
I then turned my attention to FreeNAS, where I started running into issues.
The card seems to appear correctly. I've compared the drivers from lspci and pciconf to what people posted in forms and it seems to be correct. So I set both ports a static IP, each in their own range.

What isn't working:

I am unable to ping or traceroute past the NIC IP itself. For the first port, I gave the bridge on the switch 10.10.7.1 and the NIC on FreeNAS 10.10.7.2. I can ping 10.10.7.2 and it works as expected but 10.10.7.1 fails as if it's never heard of it. The same for the other NIC.

Any ideas?
Anything someone wants me to include?

 

[ChrisRJ]

Has this card worked before with the same DAC cables connected to the same switch? Trying to isolate the root cause ...

 

[Itamar Croitoru]

Has this card worked before with the same DAC cables connected to the same switch? Trying to isolate the root cause ...

The switch, the DAC, and the card are all brand new. I have no other 10G networking to be able to swap with for testing.

 

[louisk]

When you say each interface has an ip in its own range, do you mean subnet or prefix or something else?
When you say bridge on the switch, is this bridge an isolated broadcast zone or is it shared with additional broadcast zones (if yes, which ones is it shared with)?

 

[Itamar Croitoru]

When you say each interface has an ip in its own range, do you mean subnet or prefix or something else?
When you say bridge on the switch, is this bridge an isolated broadcast zone or is it shared with additional broadcast zones (if yes, which ones is it shared with)?

I created 2 new VLANs on the switch, 7 and 8. I assigned the IPs of 10.10.7.1/24 and 10.10.8.1/24 to those VLANs on the switch. I then went on ix0 and assigned it 10.10.7.2/24 and ix1 with 10.10.8.2.

I assumed that after doing this and confirming from ifconfig that they got assigned that way, I'd be able to ping to 10.10.7.1 or 10.10.8.1 as I am able to do on my normal 1G ethernet links.

I believe that MikroTik switches use a bridge in the same way as a bridge interface in Linux/Unix, in that each is its own broadcast zone. I did so because in the MikroTik confirmation, it allows you to assign a VLAN to a single other device, though multiple devices can be assigned to a bridge. So, I assigned the odd NICs to bridge7 and the even NICs to bridge8 and then assigned VLAN7 to bridge7 and VLAN8 to bridge8. I'm very new to MikroTik, so making a lot of assumptions there.

Am I making sense?

 

[ChrisRJ]

I would suggest to start with the simplest possible setup (no VLANs etc.). This will allow to verify that the gear works at all and is not DOA (dead on arrival).

 

[louisk]

I created 2 new VLANs on the switch, 7 and 8. I assigned the IPs of 10.10.7.1/24 and 10.10.8.1/24 to those VLANs on the switch. I then went on ix0 and assigned it 10.10.7.2/24 and ix1 with 10.10.8.2.

I assumed that after doing this and confirming from ifconfig that they got assigned that way, I'd be able to ping to 10.10.7.1 or 10.10.8.1 as I am able to do on my normal 1G ethernet links.

I believe that MikroTik switches use a bridge in the same way as a bridge interface in Linux/Unix, in that each is its own broadcast zone. I did so because in the MikroTik confirmation, it allows you to assign a VLAN to a single other device, though multiple devices can be assigned to a bridge. So, I assigned the odd NICs to bridge7 and the even NICs to bridge8 and then assigned VLAN7 to bridge7 and VLAN8 to bridge8. I'm very new to MikroTik, so making a lot of assumptions there.

Am I making sense?

Yup. Makes sense. Sounds reasonable. I’m not familiar with mikrotik so I can’t comment on the accuracy of that, but generally that seems right.
1. Is the interface reporting up/up on both the nas and switch?
2. Can you run tcpdump on one of the ix interfaces and see any traffic from the switch? Can you ping from the switch to the nas? Might need to specify source ip or interface.

 

[Itamar Croitoru]

So after the comments here, I thought about how to test the components I changed, given my limits and I came up with a test that gave me interesting results.

I took one of the new 10G cards and plugged it into my desktop, and ran one of the 10Gtek cables to that card. I gave it a different static IP (10.10.7.3) and set the gateway to the switch's static IP (10.10.7.1) and disabled IPv6.

I then connected the other end in two different places. One was into the switch, and the other was into the other server, to act as a cross-over cable.

When in the switch, the same issue. Unable to ping the switch's (VLAN's) IP.
When in the other server, I can ping the other IP just fine.

So at this point, I'm going to assume that my 10Gtek 10Gb 5m DAC is working with my Intel X520-DA2 cards, as two computers with a single cable between them can ping across.

My assumption now that is MikroTik made their switches more complicated then they need to be and that I should go study how to configure them beyond network basics.

Thanks for the help

 

[louisk]

So after the comments here, I thought about how to test the components I changed, given my limits and I came up with a test that gave me interesting results.

I took one of the new 10G cards and plugged it into my desktop, and ran one of the 10Gtek cables to that card. I gave it a different static IP (10.10.7.3) and set the gateway to the switch's static IP (10.10.7.1) and disabled IPv6.

I then connected the other end in two different places. One was into the switch, and the other was into the other server, to act as a cross-over cable.

When in the switch, the same issue. Unable to ping the switch's (VLAN's) IP.
When in the other server, I can ping the other IP just fine.

So at this point, I'm going to assume that my 10Gtek 10Gb 5m DAC is working with my Intel X520-DA2 cards, as two computers with a single cable between them can ping across.

My assumption now that is MikroTik made their switches more complicated then they need to be and that I should go study how to configure them beyond network basics.

Thanks for the help

I suspect that there is something simple that was missed.
For example, you can configure a vlan, setup the IP, and forget to tag/untag the interface so nothing goes anywhere.
Good luck!