Network interface is not enabled and then disabled during boot

[xumix]

Subj. But If I use "Configure interfaces" from the boot menu, and manually configure the "igb1" it starts to work as expected.
I use a Terra Miniserver based on Asus p10s-i + ikvm module.
What could be wrong? I tried adding default route but no luck. What other logs can I provide? I've found no dhcp client logs.

Here is dmesg, you can see the link goes down UP after ZFS mount and it goes DOWN right after GEOM_ELI.

root@freenas[~]# dmesg

Show

Copyright (c) 1992-2019 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.3-RELEASE-p7 #0 r325575+ca0f1a6ba25(HEAD): Tue Apr 21 20:46:20 UTC 2020
root@tnbuild01.tn.ixsystems.com:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 8.0.0 (tags/RELEASE_800/final 356365) (based on LLVM 8.0.0)
VT(efifb): resolution 800x600
CPU: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz (3000.15-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Structured Extended Features3=0x9c002400<MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,SSBD>
XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
TSC: P-state invariant, performance statistics
real memory = 8589934592 (8192 MB)
avail memory = 8150740992 (7773 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-119 on motherboard
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1500074282 Hz quality 1000
random: entropy device external interface
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd1 at kbdmux0
mlx5en: Mellanox Ethernet driver 3.5.1 (April 2019)
nexus0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
padlock0: No ACE support.
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
Event timer "HPET1" frequency 24000000 Hz quality 440
Event timer "HPET2" frequency 24000000 Hz quality 440
Event timer "HPET3" frequency 24000000 Hz quality 440
Event timer "HPET4" frequency 24000000 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
xhci0: <Intel Sunrise Point USB 3.0 controller> mem 0xde300000-0xde30ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0: waiting for BIOS to give up control
xhci_interrupt: host controller halted
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <Intel Sunrise Point AHCI SATA controller> port 0xf050-0xf057,0xf040-0xf043,0xf020-0xf03f mem 0xde310000-0xde311fff,0xde31e000-0xde31e0ff,0xde31d000-0xde31d7ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
vgapci0: <VGA-compatible display> port 0xe000-0xe07f mem 0xdc000000-0xddffffff,0xde000000-0xde01ffff irq 17 at device 0.0 on pci2
vgapci0: Boot video device
pcib3: <ACPI PCI-PCI bridge> at device 28.5 on pci0
pci3: <ACPI PCI bus> on pcib3
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xd000-0xd01f mem 0xde200000-0xde27ffff,0xde280000-0xde283fff irq 20 at device 0.0 on pci3
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 38:d5:47:00:76:bd
igb0: PHY reset is blocked due to SOL/IDER session.
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
pcib4: <ACPI PCI-PCI bridge> at device 28.6 on pci0
pci4: <ACPI PCI bus> on pcib4
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xc000-0xc01f mem 0xde100000-0xde17ffff,0xde180000-0xde183fff irq 21 at device 0.0 on pci4
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: 38:d5:47:00:76:be
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <memory> at device 31.2 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
acpi_syscontainer0: <System Container> on acpi0
wbwd0: <Nuvoton NCT6791 (0xc8/0x03) Watchdog Timer> at port 0x2e-0x2f on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
freenas_sysctl: adding account.
freenas_sysctl: adding directoryservice.
freenas_sysctl: adding middlewared.
freenas_sysctl: adding network.
freenas_sysctl: adding services.
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ipmi0: IPMI device rev. 1, firmware rev. 2.05, version 2.0
ipmi0: Number of channels 0
ipmi0: Attached watchdog
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub0: 22 ports with 22 removable, self powered
ugen0.2: <Kingston HyperX Alloy FPS Mechanical Gaming Keyboard> at usbus0
ukbd0 on uhub0
ukbd0: <Kingston HyperX Alloy FPS Mechanical Gaming Keyboard, class 0/0, rev 2.00/1.17, addr 1> on usbus0
kbd2 at ukbd0
ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ada0 at ahcich2 bus 0 scbus2 target 0 lun 0
ada0: <WDC WD20EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device
ada0: Serial Number WD-WCC4M4JVSZP3
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors)
ada0: quirks=0x1<4K>
ada1 at ahcich3 bus 0 scbus3 target 0 lun 0
ses0: ada0 in 'Slot 02', SATA Slot: scbus2 target 0
ada1: <WDC WD20EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device
ada1: Serial Number WD-WCC4MKUYLVE7
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors)
ada1: quirks=0x1<4K>
ada2 at ahcich5 bus 0 scbus5 target 0 lun 0
ses0: ada1 in 'Slot 03', SATA Slot: scbus3 target 0
ada2: <TS128GMTS430S S0423A> ACS-2 ATA SATA 3.x device
ada2: Serial Number F300710359
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada2: Command Queueing enabled
ada2: 122104MB (250069680 512 byte sectors)
ses0: ada2,pass2 in 'Slot 05', SATA Slot: scbus5 target 0
random: unblocking device.
Trying to mount root from zfs:freenas-boot/ROOT/default []...
lo0: link state changed to UP
igb1: link state changed to UP
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/8/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
GEOM_MIRROR: Device mirror/swap0 launched (2/2).
GEOM_ELI: Device mirror/swap0.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: hardware
igb1: link state changed to DOWN
ums0 on uhub0
ums0: <Kingston HyperX Alloy FPS Mechanical Gaming Keyboard, class 0/0, rev 2.00/1.17, addr 1> on usbus0
ums0: 5 buttons and [XYZT] coordinates ID=1
uhid0 on uhub0
uhid0: <Kingston HyperX Alloy FPS Mechanical Gaming Keyboard, class 0/0, rev 2.00/1.17, addr 1> on usbus0
CPU: Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz (3000.15-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Structured Extend

 

[xumix]

The title should be read as "Network interface is enabled and then disabled during boot"
Also Version: FreeNAS-11.3-U2.1

 

[xumix]

The problem is: the system has 2 interfaces, and freenas is not saving the proper config after restart.
As soon as "Configuring interfaces" message appears `igb1` goes down.
If I do `dhclient igb1` in the shell after restart the interface goes UP again and I'm able to access web ui.
BTW why can't be both interfaces configured as DHCP?

 

[jgreco]

BTW why can't be both interfaces configured as DHCP?

Because it doesn't make any sense. Two DHCP sessions means two sets of information. Which DHCP session controls the default route? Which one controls nameservers? If both of them offer additional routes, how does that work? Windows has been struggling with this for decades and it is still a mess over there.

DHCP is generally not something you should be using for infrastructure systems in the first place. If your DHCP server goes down, your infrastructure systems become inaccessible. If your DHCP renegotiates a new IP, your NAS storage appears to hang. Even if you just renew an existing lease, there are situations where you will lose connectivity momentarily.

DHCP should be used to bootstrap your NAS onto the network and then you should wire it in where it is expected to live with a static IP configuration. A NAS is *not* a "dynamic host" and therefore probably shouldn't be configured by something called "Dynamic Host Configuration Protocol." I mean, sure, you might be able to prove me wrong by DOING it, pretty much anyone could "make" it work, but this is one of those things where just because you CAN do something, doesn't mean it is a good idea.

 

[xumix]

Because it doesn't make any sense. Two DHCP sessions means two sets of information. Which DHCP session controls the default route? Which one controls nameservers? If both of them offer additional routes, how does that work? Windows has been struggling with this for decades and it is still a mess over there.

Thanks, that makes sense.

If your DHCP server goes down, your infrastructure systems become inaccessible.

Yup, same as with any other infra. Who would use nas w/o network?

DHCP should be used to bootstrap your NAS onto the network and then you should wire it in where it is expected to live with a static IP configuration.

So, I removed dhcp and made igb1 static IP, now it works but this is clearly not user friendly and not working as intended.

I want DHCP since it provides routes, dns, etc. I just make my lease for NAS permanent.
Why would one NOT use a protocol designed for exactly the purpose of network configuration?

 

[jgreco]

Yup, same as with any other infra. Who would use nas w/o network?

Networks continue to exist even if the DHCP server has died. Anything configured for a static IP continues to work without a care in the world.

So, I removed dhcp and made igb1 static IP, now it works but this is clearly not user friendly and not working as intended.

"User friendly" is in the eyes of the beholder. I find it more user friendly when things are wired down and I can rely on them being where I assigned them.

"Not working as intended" is something you'd need to explain.

I want DHCP since it provides routes, dns, etc. I just make my lease for NAS permanent.
Why would one NOT use a protocol designed for exactly the purpose of network configuration?

Because it's designed for dynamic hosts, not for general network configuration. That is the "Dynamic Host", the "DH" of "DHCP". It's right there in the name.

Having spent decades building infrastructure and having had to fix overly clever network designs where someone built everything around DHCP, I do not perceive DHCP as a great tool. It's fine for the gear that pops up on the network, especially like wireless devices.

See, I run up against stuff like the fellow who's configured his FreeNAS and his ESXi both with DHCP. One day the DHCP server dies. Hours later, all the VM's wedge, because the NAS refreshed its DHCP lease, got nothing, and so the hypervisor's storage "went away". His hosting operation grinds to a halt and getting in to fix it is not fun.

Or the network where someone added a router to it and it had DHCP enabled by default, resulting in a network with two DHCP servers that ended up fragmenting the network into two different IP ranges (one the intended, one 192.168.0.0/24).

 

[xumix]

"Not working as intended" is something you'd need to explain.

3. Booting — FreeNAS®11.3-U2 User Guide Table of Contents

www.ixsystems.com

This is how it is intended to work. There is nothing in the manual saying I must use static IP.

During boot, FreeNAS® automatically attempts to connect to a DHCP server from all live network interfaces. After FreeNAS® successfully receives an IP address, the address is displayed so it can be used to access the web interface. The example in Figure 3.1 shows a FreeNAS® system that is accessible at

http://10.0.0.102

.

And this does not work, because Freenas is not properly configuring to work with DHCP.

That is the "Dynamic Host", the "DH" of "DHCP". It's right there in the name.

Yeah, how about read it as dynamic configuration, not dynamic host.

See, I run up against stuff like the fellow who's configured his FreeNAS and his ESXi both with DHCP. One day the DHCP server dies. Hours later, all the VM's wedge, because the NAS refreshed its DHCP lease, got nothing, and so the hypervisor's storage "went away". His hosting operation grinds to a halt and getting in to fix it is not fun.

Or the network where someone added a router to it and it had DHCP enabled by default, resulting in a network with two DHCP servers that ended up fragmenting the network into two different IP ranges (one the intended, one 192.168.0.0/24).

Sure, we may even talk about a meteorite falling and breaking everything but how is this applied to default NAS plug&play config?
Just because DHCP is not properly working currently I will not be able to configure my box w/o a monitor or a console (like it is even mentioned here https://www.ixsystems.com/documentation/freenas/11.3-U2/booting.html#obtaining-an-ip-address)

 

[jgreco]

https://www.ixsystems.com/documentation/freenas/11.3-U2/booting.html#obtaining-an-IP-address

This is how it is intended to work. There is nothing in the manual saying I must use static IP.

There are a lot of dumb things that you can do in life, many of which are even recommended by people.

And this does not work, because Freenas is not properly configuring to work with DHCP.

Okay. That's nice.

Yeah, how about read it as dynamic configuration, not dynamic host.

Why don't you read it as Unicorn Monday Free Gold. If you're going for a misread, go big, get rich.

You know, some of us do this stuff professionally. You don't have to like my information, but I've been doing this a really long time, I've seen lots of things, and I have made a career out of making the Internet work. I don't really care if you want to configure your network entirely with DHCP. It's something I know doesn't work real well in practice. I already said:

I mean, sure, you might be able to prove me wrong by DOING it, pretty much anyone could "make" it work, but this is one of those things where just because you CAN do something, doesn't mean it is a good idea.

so if you aren't interested in the opinion of someone who has been doing this for a living for decades, that's fine, but please don't start cherry-picking words out of acronyms to try to make yourself "right". Trying to force DHCP into this role is architecturally wrong. People do it. Some people get it to work kinda-right. It isn't actually possible to make it work entirely correctly due to stuff I've already mentioned here.

I have no interest in debating the reality that you would like to have. I am busy working in the reality that exists. If you have a fact- and reality-based argument to make, I will be happy to entertain that.

 

[xumix]

There are a lot of dumb things that you can do in life, many of which are even recommended by people.

So Freenas manual is written by dumb people? Nice.

You know, some of us do this stuff professionally. You don't have to like my information, but I've been doing this a really long time, I've seen lots of things, and I have made a career out of making the Internet work.

Are you a freenas dev? If so, there should be Advanced mode for the people like you, leave "works by default" settings for normals.

I have no interest in debating the reality that you would like to have. I am busy working in the reality that exists. If you have a fact- and reality-based argument to make, I will be happy to entertain that.

I've read many threads here I find the attitude as pure cancer. No surprise people run to Linux, UnRaid and OMV from this elitist BS.

 

[Patrick M. Hausen]

https://www.ixsystems.com/documentation/freenas/11.3-U2/booting.html#obtaining-an-IP-address

This is how it is intended to work. There is nothing in the manual saying I must use static IP.

During boot, FreeNAS® automatically attempts to connect to a DHCP server from all live network interfaces. After FreeNAS® successfully receives an IP address, the address is displayed so it can be used to access the web interface.

After FreeNAS successfully reveives an IP address ... "an" address as in one address on one interface. Not two. And this perfectly works as documented. I connect newly installed FreeNAS systems to my LAN, type "freenas.local" intro the address bar of my browser - presto!

But as @jgreco explained in great detail, running a DHCP client on two interfaces does not make sense. This is not how IP is supposed to work. And it is "Dynamic Host Configuration Protocol". By definition a host is a system with one interface.

Patrick

 

[xumix]

After FreeNAS successfully reveives an IP address ... "an" address as in one address on one interface. Not two. And this perfectly works as documented. I connect newly installed FreeNAS systems to my LAN, type "freenas.local" intro the address bar of my browser - presto!

Right. My setup receives 0 IPs therefore this thread exists. Only 1 interface is connected to network at the time.

 

[jgreco]

I'm closing this thread as the OP is getting combative and isn't receptive to helpful information.

 

[Patrick M. Hausen]

That was not apparent from your initial posting. At least not to me. Was this a clean install? Can you perform a packet trace (tcmpdump) for port bootpc on your DHCP server?