NIC Link State Down as soon as I set a static IP

[BRIO]

Title says pretty much everything.

I actually have 3 different NIC on the server. 2 different 10Gb/s port on the IO of the Motherboard and I added a StarTech dongle to connect the server to my ISP. Since I am currently switching gear, I decided to reset configuration to defaults to start everything from scratch.

The problem I encounter is that AS SOON as I set a static IP for the NIC that is connected to the ISP, both 10Gb/s NIC goes DOWN and I can't set static IP for both of them. Below are the step I do in the order I do it right after I reset the configuration to defaults.

1. As you can see it seems like the 10Gb/s NIC are set to 0.0.0.0 by default - Don't know if I should keep that kind of address for the static IP?

2. Here is what it looks like once I log in after the reset. All NIC are UP

3. Again, everything is fine

4. Then I set up the IPv4 Default Gateway - all the NIC are still UP after that setup

5. Here I set the static IP address for the NIC connected to the ISP

6. And here is the problem, both 10Gb/s NIC are now DOWN after I setup the static address for the NIC connected to the ISP

I thought is it the StarTech Dongle that creates some weird stuff to the NAS? Anyway, curious to have your toughts on the issue and hopefully I can setup both of my 10Gb/s NIC properly.

Hardware:
StarTech dongle connected to the server: https://www.amazon.ca/StarTech-com-...-1-spons&sp_csd=d2lkZ2V0TmFtZT1zcF9hdGY&psc=1
Motherboard: ASUS Pro WS WRX80E-SAGE SE WIFI II AMD WRX80

Thanks in advance!

 

[BRIO]

Alright, here is a little update: I managed to progress a little but I still can't make the 10Gb/s NIC works properly when they are connected to my computer.

The only way I'm able to make the ix0 and ix1 (both 10Gb/s NIC) work is when I reset the server configuration to default setting with both NIC connected to my ISP (if the NIC are not connected to the ISP, I get a 0.0.0.0/8 IP address). If I want to keep the 10Gb/s NIC UP, I have to keep the default IP that TrueNas gives me after the reset. Otherwise, if I create a new Static IP, the NIC goes DOWN. So the only NIC I can put a static IP that works is the ue0 since it is connected to my ISP. Once ue0 is set, ix0 ans ix1 are UP but I can't access them through a web browser with their IP address no more.

to sum it all:
ue0 - works properly when I create a dedicated static IP address (the NIC is connected to the ISP router)
ix0 - Link State UP but can't make the IP address work in the web browser (and I can't change the static IP address)
ix1 - Link State UP but can't make the IP address work in the web browser (and I can't change the static IP address)

 

[jgreco]

Welcome to the forums.

Sorry to hear you're having trouble. Please take a few moments to review the Forum Rules, conveniently linked at the top of every page in red, and pay particular attention to the section on how to formulate a useful problem report, especially including a detailed description of your hardware, what exact model and manufacturer of ethernet card this is, what exact model and exact manufacturer of SFP+ your SFP is, what sort of switch you're using and how it is configured, and all the other relevant stuff that would be helpful to debug your setup.

 

[BRIO]

Welcome to the forums.

Sorry to hear you're having trouble. Please take a few moments to review the Forum Rules, conveniently linked at the top of every page in red, and pay particular attention to the section on how to formulate a useful problem report, especially including a detailed description of your hardware, what exact model and manufacturer of ethernet card this is, what exact model and exact manufacturer of SFP+ your SFP is, what sort of switch you're using and how it is configured, and all the other relevant stuff that would be helpful to debug your setup.

Thank you for the clarification here and sorry about the detailed description, here it is:

SERVER
Motherboard: ASUS Pro WS WRX80E-SAGE SE WIFI II AMD WRX80
CPU: AMD Ryzen Threadripper PRO 5955WX
RAM (128GB): Team T-Create Classic 10 LAYERS 64GB (2 x 32GB) 288-Pin DDR4 SDRAM DDR4 3200 (PC4 25600) Desktop Memory Model
USB-C to Ethernet to connect the server to the ISP: StarTech USB-C to Ethernet
"M.2 server": 16.0TB OWC Accelsior 8M2

in between the server and PC - Cat-6 cables: Amazon basic Cat-6

PC
Motherboard: MSI Prestige X570 Creation
CPU: AMD Ryzen 9 5900X
RAM (128GB): Team T-Create Classic 10 LAYERS 64GB (2 x 32GB) 288-Pin DDR4 SDRAM DDR4 3200 (PC4 25600) Desktop Memory Model
10GB PCIe Card: Intel X540-T2 Dual 10GB port

DISK SHELF
Irrelevant for the current problem

Here is roughly how it is all connected together:

So to make it short, I used to have my computer directly connected to the disk shelf with the same pieces as showed above in the built. The only pieces I added is the server in between (ASUS Pro WS WRX80E-SAGE SE WIFI II). I reset my TrueNas configuration in install the new server but now I can't go past setting the IP for the 2 10Gb/s NICs that link the server to the computer. I can connect both together as I can see some small upload/download trafic on the TrueNas dashboard, the connection can even show as "UP" but for some reason I can't setup a static IP to both of these NICs as stated above.

I followed that procedure, I don't know if I'm missing someting?

Everything I tried:
1. Set up a static IP through the GUI (like in the video above) - 10Gb/s NICs then show as "DOWN" as soon as I set a static IP for the ISP NIC

2. Tried to set up a static IP through the TrueNas interface right after I reset the configuration - this is the only way I can keep the 10Gb/s port to "link up with the computer" but I can't set a static IP for them when I enter the GUI - as long as I keep the 10Gb/s NIC connected to the ISP, TrueNas gives them some IP address when I reset the settings to default, THEN I can connect my NAS to my computer and it shows as 10Gbase-T and I have little upload/download showing up but I can't change the IP of the 10Gb/s NICs, the conexion doesn't work.

3. I tried to play in the BIOS of the motherboard but it doesn't seem to change anything

Hope all these information can help you answer your question - only thing I'm missing is the "SFP" as you mentioned as I use cat-6 cable. Thanks in advance!

 

[jgreco]

It doesn't appear as though you have a switch in your architecture, and it appears from the console messages that you have configured multiple IPv4 ethernet interfaces on the same layer 2 network, which is not cool.

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

You probably need to fix your network first so that other problems become more apparent.

 

[BRIO]

It doesn't appear as though you have a switch in your architecture, and it appears from the console messages that you have configured multiple IPv4 ethernet interfaces on the same layer 2 network, which is not cool.

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

You probably need to fix your network first so that other problems become more apparent.

hmm, glad to know it shouldn't be complicated! First of, what do you mean by "you have a switch in your architecture"? Because as far as I know, I don't have any switch in my setup but I want to be sure I'm not forgeting something here

 

[samarium]

If you are plugging multiple cables into Fizz, then it is a switch too.
It is probably is only a 1000Mb/s or 1Gb/s switch unless it is a very special ISP modem.
This is the 192.168.0.x/24 network, or the 192.168.0.x/255.255.255.0 network.
Usually the ISP modem would be running a DHCP server and handing out dynamic addresses to PCs/server connecting to that network.

The 10G connections are, I think directly connected between the server and the PC?
Often these go thru a dedicated 10G switch.
Usually new hardware can handle direct connected cables, but some older hardware cannot and requires a switch in the middle.
They should be on a different network, like 192.168.1.x/24, and the PC 10G connections should also be on this network.
Depending on the configuration the second 10G channel might need to be on another network like 192.168.2.x/24.
If you try and use 192.168.0.x/24, then it generally won't work.
When you get to the 10G networks, try just one cable at a time to verify on both ends that you have a connection, then try the next cable.