FreeNAS Mini - Locked out without IP connectivity

[Glen356]

I have a problem with the FreeNAS Mini I recently purchased from iXsystems.

My problem: I am not able to establish a TCP/IP connection; even over the IPMI management port. The FreeNAS Mini is a headless server, so my previous recovery strategies do not really apply.

My question: Having lost IP connectivity, what recovery strategies are there for a headless server like my FreeNAS Mini?

Background information: Initially I was able to use the HTML GUI via Firefox without issue. What started this problem was a change made to an unused ethernet interface (igb1); I was using igb0 at the time. After saving an IP address change, an error was displayed, and from there...well, things just got worse. So this does not look like a hardware problem. More likely a user problem. iXsystems do not offer support unless the issue is hardware related; an automated response from iXsystems directed me to this forum. So far I have not found anything in the FreeNAS documentation or on the internet. Which does not mean there isn't anything to find, I just have not found anything yet. If anyone has any tips, please share :)

Strategies going forward: Without the use of any of the ethernet ports, I can still establish alternative access by attaching a keyboard to a USB port or attaching a PC to the serial port. But having done so, I will still be staring out into a bottomless pit as to what should come next: There is no standard output to a screen nor any speaker to sound a beep. I am hoping the serial port, by default, is used as a standard output. If so, by tomorrow I will have a null modem cable to monitor the serial port during the boot process. I may end up searching for how to change the boot device priority should a clean install be required. Sigh.

If anyone can help, I thank you in advance and appreciate your time.

 

[Chris Moore]

I am not able to establish a TCP/IP connection; even over the IPMI management port. The FreeNAS Mini is a headless server, so my previous recovery strategies do not really apply.

It has a VGA port. If your network settings are not working for some reason, you may need to get a monitor with a VGA connector. The IPMI address, for the management port, and the IP address of the network for data are supposed to be two different addresses.

 

[Chris Moore]

Without the use of any of the ethernet ports, I can still establish alternative access by attaching a keyboard to a USB port or attaching a PC to the serial port. But having done so, I will still be staring out into a bottomless pit as to what should come next:

Have you checked the manual? It talks about configuring the network interface under first boot:
https://www.ixsystems.com/documentation/freenas/11.2/booting.html#booting

 

[Chris Moore]

I am hoping the serial port, by default, is used as a standard output. If so, by tomorrow I will have a null modem cable to monitor the serial port during the boot process.

This is off by default, but you can enable it from the GUI, once you get back into the GUI by configuring the network interface.
Here is the manual reference:
https://www.ixsystems.com/documentation/freenas/11.2/system.html#advanced
They don't give much detail on it, but the toggle to turn the serial port on is on the "Advanced" screen.

 

[Glen356]

Thank you Chris! Using the VGA interface did the trick. Following the boot-up the Console setup was there waiting for me.

My original goal was to have a static IP address for the NAS, so I had changed the IP network and disabled DHCP. That was my mistake. For that change to work the NAS has to be directly connected to my router, and it is not: it is connected to an unmanaged switch. Remote networks can only be learned from dynamic routing protocols or by creating static routes, neither of which are options for me. I could just plug the NAS into the router, but that just seems like a potential booby trap.

No, I prefer to keep things simple. I disabled DHCP on both of the NAS NIC's, and assigned them host addresses in the available upper range, which DHCP can use, but never has.

Regarding IPMI: There is some funky behavior I have not yet wrapped my head around, but I will dig deeper and expect to sort it out.

Thank you for your help.

 

[Ericloewe]

Regarding IPMI: There is some funky behavior I have not yet wrapped my head around, but I will dig deeper and expect to sort it out.

Think of it as a separate system, with its dedicated NIC. Yes, the first NIC of the host can be shared with IPMI, but that's a detail.

 

[Glen356]

Think of it as a separate system, with its dedicated NIC. Yes, the first NIC of the host can be shared with IPMI, but that's a detail.

Thanks Eric. I have recovered connectivity over the dedicated IPMI network interface card. Perhaps others may be interested in how the IPMI was failing and what action resolved it. I will describe how I set up IPMI using the new FreeNAS GUI and then what resolved the problem.

IPMI Configuration: Using a Firefox browser, I signed into the FreeNAS GUI and went to Network, and then IPMI. From there, I disabled DHCP and added the IP address: 192.168.14.25 with a 255.255.255.252 subnet mask. The default gateway is 0.0.0.0.

Symptoms of the failure: Unable to load IPMI page using a Firefox or ping its assigned IP. Using the new FreeNAS GUI and going to Network>IPMI, where, if I changed the channel from 8 to 1 (located near the top of the page) the connectivity would only briefly recover. Note: in my NAS, Channel 1 is associated with NCSI (eth0) and Channel 8 with IPMI (eth1). The Mini's motherboard UEFI access showed the BMC network configuration with two ethernet interfaces (NCSI & IPMI) and each had the same IP assignments and different MAC addresses: Both ports had their 'config address source' as Unspecified. My experience has been that whenever two different MAC addresses co-exist with the same IP address the TCP/IP protocol will be unreliable.

What was changed: Using the UEFI access, I changed the NCIS and IPMI configurations:
o Config Address Source fields for both ports were changed from 'Unspecified' to 'Static'
o NCIS port was left with 0.0.0.0 for the IP address, subnet mask, and default gateway (changing to Static causes this)
o IPMI port and its IP config was changed to match what had been specified in FreeNAS GUI for IPMI

I would like to take this opportunity to thank both Eric and Chris for their interest and guidance. I am a new member to this forum and have lots to learn. If I am not doing things correctly then feel free to let me know. No doubt I will make mistakes going forward , I can only promise to do my best not to make the same mistake twice.

 

[Glen356]

What follows is a brief summary of what I did to recover connectivity to...well, everything. If you have a similar failure this may help you, but I offer this warning: Your hardware may be different in an important way and there may have been significant version changes from the time of this writing. So proceed at your own risk. :)

How I recovered IP connectivity to my local LAN:

1. Attached a monitor to the NAS VGA interface and keyboard to a NAS USB port.
2. After powering up my FreeNAS Mini the Console setup was ready for use on the VGA monitor.
3. Using the Console setup, I configured the NAS LAN with a static IP; when prompted for "Configure I/F for DHCP", I entered no. Note: My router has DHCP enabled and I do not want my NAS address to change. So I assigned an IP addresses ending with .250. Because my subnet mask is 255.255.255.0, DHCP has 254 host addresses to choose from, which means it is unlikely the NAS host address will change.
4. In the Console setup I then selected "Reboot" to recover.

How I regained access to the IPMI dedicated port:

1. I configured a spare NIC on my PC with a static 192.168.12.26/30 host address; the PC throughout the IPMI recovery was never connected to the local LAN. Using a straight through ethernet cable I directly connected the NAS IPMI port to the NIC. Note: Many older NICs are not able to auto-detect and correct, so using a crossover cable will assure each transmit is sending to the other's receive.
2. Accessed the NAS BIOS, (actually UEFI) and changed the BMC (Baseboard Management Controller). The motherboard I am using has the navigation path Server Mgmt then BMC network configuration. What I found were two interfaces on the IPMI port: NCSI (eth0) and IPMI (eth1). I found both interfaces had [Unspecified] in their "Config address source" fields and both ports were configured with the same192.168.12.25/30 host address and each port had its own MAC addresses. I changed both port's "Config address source" fields to [Static] which removed the IP config on both ports. I made no further changes to NCSI (eth0). I made the following changes to IPMI (eth1):
   • IP: 192.168.14.25
   • Mask: 255.255.255.252
   • Router IP: 0.0.0.0
   • Save and Exit
3. Power-cycled both the PC and NAS just to be sure. And sure enough the dedicated IPMI port was working for me again.

What I learned about IPMI:

• Intel's documentation of IPMI version 2 is 644 pages long.
• From the IPMI UI: going to Configure, and then Network, gets you to the IPMI port config, not the regular ethernet port.
• ipmitool is available from the FreeNAS shell and I think it is more powerful then the IPMI UI (with a learning curve).
• Two ipmitool commands I used were "ipmitool bmc info" & ipmitool lan print n; where n is the channel number.
• ipmitool confirmed channel 1 is NCSI (eth0) and channel 8 is IPMI (eth1); as expected.
• NCSI was not useful to me, I got it to work but never very well or for very long; so I basically disabled it.
• IPMI, along with BMC, provides an amazing variety of maintenance choices that I will likely never use.
• Powering up my NAS from the IPMI port is pretty cool (but I will probably only do it once).
• Everybody says keep the IPMI port off your local LAN and I think everybody is right! It just does not belong there.