Random thoughts on headless configurations and decapitated threads

Status
Not open for further replies.

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
On the topic of IPMI, ILO, or other funtastic remote access technologies ... which someone had just been asking about. I believe the question was something like "why do something so complicated" (as using IPMI when managing a server locally). Which I think is a great question, as I've been on both sides of it.


I've been running remote access technologies like serial console (~30 years) and KVM-over-IP (~15 years) for remote access and control of computing equipment. Serial console was very limiting, but made for lots of excitement when you'd need to dial in to a remote POP and use C-Kermit to upload a patched binary for a problem. Then along came KVM-over-IP, a magical technology only possible on high-ish speed Internet. Years ago, most KVM-over-IP solutions just weren't very well-rounded or practical, in part because it was only KVMoIP. You had to find other ways to deal with resetting hung gear, like rack PDU's. You had to find other ways to deal with remotely reloading or upgrading an OS on a server.

And a lot of people who think of KVM-over-IP, still think of it in that way. But today, it's better. It's MUCH better than that. With the advent of IPMI, you have a miniature system-on-a-chip at your disposal to manage the computer. It can provide you with keyboard and video over the network, yes... but it also gives you access to the status of the system, and control over the system, and the ability to plug USB peripherals into the system remotely. It is a dream!

Today, it is very common for me to sit on my fat arse in my office and to not lay hands on equipment except to plug it in.

We actually have a great shop bench with six bays and three stations of keyboard/mouse/monitor. But overall it is happening more and more often that it is easiest to be lazy and not bother hooking up VGA and PS/2... the shop KVM switch is a 4x16 dealie and there's only one uplink to the KVM-over-IP infrastructure. So it used to be that you'd be in the shop working at a bay. It used to be there'd be a pile of floppies, some USB floppy drives, a few floppy drives on the end of a FDD cable, piles of CD's, some USB CDROM's, etc. etc laying around. But of course you could never find the resource you needed, because it'd be in use or misplaced or whatever... and you'd be standing there scratching your head, trying to decide whether to go burn a new ISO of the missing item. What a pain!

But with the advent of virtualization, we had to build a consolidated repository on-line of ISO images. And once you have that, it turns out that with IPMI or ILO, it becomes easier to plug in an ethernet, go back to my desk, and then I can remotely command an ISO file to be mounted, and to manage the KVM over the management ethernet.

What's the alternative? Lots of time used to be spent downloading the latest software ISO's, burning CD's, hooking up CDROM's, cursing and swearing that boot ordering or other BIOS problems weren't letting it boot, hitting RESET, etc. To be sure, there's still some of that today, but basically I can sit here in the office and tell a box to do a lot of those things from remote.

And here's the kicker....

I can be doing other stuff - like browsing the FreeNAS forums - at the same time.

So I'm firmly in the "headless rocks" camp. I'll hook up a crash cart or KVM when necessary, but especially with virtualization, the similarity in workflows between managing a physical server and managing a VM from my desk is pretty compelling.
 

pirateghost

Unintelligible Geek
Joined
Feb 29, 2012
Messages
4,219
I am also firmly in the 'headless rocks' camp, but my experience over the last 6 years with all the servers I have supported have been with HP and Dell (these by default did not have DHCP enabled on the remote access controller), so I have become accustomed to hooking up the monitor and keyboard for just the initial configuration of the remote access. Once configuration of the remote access is done, its time to stack, rack, and power. Then the fun happens from the comfort of my own chair...this I do love.

Having the remote access controller get an IP on its own would be a godsend in some of my situations where I have had to have someone be my eyes, hands, and feet in the datacenter (last year had to deal with this situation in Comodoro Rivadavia, Argentina. Not an easy task walking someone (at the desktop support level and at least speaks better english than I spanish) how to boot and configure the remote access.

The question of why make it more complicated than necessary comes from the fact that you have to log into your DHCP server to find out what IP your IPMI just grabbed. Based on the question posted in the original thread, it was apparent that going through the extra steps of finding out which IP it had was more complicated than was necessary.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Well yeah but since we're going to wire all that in to the management infrastructure anyways, you can either:

1) Note the MAC and program it into DHCP so it has a fixed IP, then not have to wonder

2) Putz around trying to determine what was just allocated by DHCP, then still have to go back and program it into DHCP so it has a fixed IP, and then have to also log in to the dynamic IP, tell it to reboot itself, so that it will wind up at the fixed IP.

From my point of view you're suggesting that #2 is what happens. If so: Yes. That's more complicated than necessary, but hey we sometimes screw around for a lot longer than that with annoying issues. That's one of the lesser ones.

My life became simpler once I got a nice phone with a macro zoom. The process of "Note the MAC" becomes "get the camera aimed and focused, click. Oh hey cool look there's a free magnification effect so I don't ruin my aging eyes trying to read tiny print embedded on a board deep in a chassis, plus no need for pencil and paper."

Odd though, I could've sworn our HP's with iLO2 do DHCP by default.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
It is only getting better. I never used KVMoIP or IPMI before 2013 and now I wish it came with everything. I just worry about the security aspects of the technology, or perhaps better phrased as, vulnerabilities.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
You learn to cover all your bases when the company you run owns gear on the other side of the continent, and for years I had a number of boxes with both KVMoIP and PDU *alongside* the system's IPMI. When the cost for an on-site visit starts around ~$500 and is at least several hours in the future (meaning potentially several hours downtime), ...

The security aspects that should be more worrying are the ones that aren't as visible and obvious. Many PC's these days include Intel's Management Engine, as an example, or Microsoft Windows 10 in it's default "jezebel Cortana reports everything to daddy" mode. These days I am mostly concerned about the management interfaces of things like hypervisors, which have that strange combination of needing to be somewhat more accessible while also being potentially more dangerous and damaging.
 

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
It is only getting better. I never used KVMoIP or IPMI before 2013 and now I wish it came with everything. I just worry about the security aspects of the technology, or perhaps better phrased as, vulnerabilities.
I think the security implications of this are something that are not well addressed, particularly by smaller shops. As an industry, we talk a lot about how console access is basically root access, so we lock our servers up in a secured closet, but leave the IPMI interface unpatched, unsegregated, or god-forbid, with the default password. I've done some penetration testing for smaller shops, and without fail I see poor management of IPMI.

If we could get people to segregate access to only a couple trusted access machines, that would go a long way to addressing these security issues.
 
Joined
Apr 9, 2015
Messages
1,258
I haven't done a ton of KVM over IP or IPMI before my current FreeNAS was setup but I have to agree that it is great to have. About the only time I actually hook something up to the server is if something is very wrong and I don't want any possibility of the IMPI getting in between me and what I am doing.

I have however used remote desktop software for quite a long time now. My previous home server was windows based and I ran headless and just used remote desktop to access. I use something similar when I have to help my father or step daughter with their systems since it has been easier a lot of times to just remote in to their desktop and run everything so they can learn while i am doing things.

I have however set static IP addresses for the IMPI and kept the last three digits the same for the FreeNAS systems and jails between the three installations just so it is easier for me to keep track of what they are. Kinda like Plex is always .151 and syncthing is always .190 The IPMI is the same on each of the networks as well. I do worry about the IMPI security but found a simple solution is to just unplug the Cat5 cable unless I intend to use it. It's not that hard to walk over and plug it in when I need to use it and so far I have not needed to at the remote installations.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
I haven't done a ton of KVM over IP or IPMI before my current FreeNAS was setup but I have to agree that it is great to have. About the only time I actually hook something up to the server is if something is very wrong and I don't want any possibility of the IMPI getting in between me and what I am doing.

I have however used remote desktop software for quite a long time now. My previous home server was windows based and I ran headless and just used remote desktop to access. I use something similar when I have to help my father or step daughter with their systems since it has been easier a lot of times to just remote in to their desktop and run everything so they can learn while i am doing things.

I have however set static IP addresses for the IMPI and kept the last three digits the same for the FreeNAS systems and jails between the three installations just so it is easier for me to keep track of what they are. Kinda like Plex is always .151 and syncthing is always .190 The IPMI is the same on each of the networks as well. I do worry about the IMPI security but found a simple solution is to just unplug the Cat5 cable unless I intend to use it. It's not that hard to walk over and plug it in when I need to use it and so far I have not needed to at the remote installations.

Be careful you don’t have the bios set to auto-fail over the IPMI port to one of the LAN ports in that situation ;)

It is the default on most supermicro boards I’ve worked with.
 
Joined
Apr 9, 2015
Messages
1,258
Be careful you don’t have the bios set to auto-fail over the IPMI port to one of the LAN ports in that situation ;)

It is the default on most supermicro boards I’ve worked with.

Yes, I realized this at one point. Now that I am using fiber it is not an issue as I don't even have a need to connect the Cat5 at all. And my router has an IPMI failure, the board was borked when I got it so the IPMI just doesn't work, the video doesn't work as well but it's simple enough to run headless and if I need to actually make a change with a keyboard and mouse I just insert a video card.
 
Status
Not open for further replies.
Top