SOLVED Running AMSD service for HPE Microserver Gen10+ iLO

[amihal]

Hi!
I'm running TrueNAS Scale on a HPE Microserver Gen10+. The iLO5 BMC on the system can integrate with a client-side service called AMS (Agentless Management Service) in order to pass some client OS information through to the iLO gui. Things like networking config and SATA drive info. AMS is available as an Ubuntu package containing the amsd daemon and the systemd config files to run it. Out of curiosity and not necessity, I am trying to run the amsd daemon on my system.

I have tried two things so far: running AMSD in a docker container and running in an Ubuntu VM.

The docker approach looks like this:

Code:

FROM ubuntu:20.04
RUN apt-get update && apt install -y curl gnupg
RUN echo "deb http://downloads.linux.hpe.com/SDR/repo/mcp focal/current non-free" > /etc/apt/sources.list.d/mcp.list
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey1024.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add -
RUN apt-get update && apt install -y amsd && rm -rf /var/lib/apt/lists/*

Launch the container interactively for testing, and try to start amsd to see if it works:

Code:

# docker build --network=host -t amihal/ams .
# docker run --device /dev/mem --cap-add SYS_RAWIO -i -t amihal/ams bash
root@79964ccb4e69:/# amsd
SM BIOS initialization failed, unable to determine system type

The daemon starts, but it's not able to access SMBIOS from inside the container.

For the VM approach, I created a normal Ubuntu 20.04 VM and ran the same commands as in the Dockerfile above to install the amsd daemon. Launching it inside the VM I get a different result:

Code:

ubuntu-mate@ubuntu-mate:~$ sudo amsd
This program requires the host to have a HP Integrated Lights Out 4/5 (iLO 4 or iLO 5) BMC

So in this case the BMC is not visible from inside the VM.

A third approach would be to install and run amsd directly on the TrueNAS Debian system. I think this would work at least temporarily but I have not tried it since it is not the intended usage scenario for running services on TrueNAS. My favored approach would be to get it working as a docker container and eventually publish to TrueCharts so that other Microserver users can install it with one click.

Any ideas?

Thanks,
Andrew

 

[amihal]

Updated Dockerfile:

Code:

FROM ubuntu:20.04
RUN apt-get update && apt install -y curl gnupg
RUN echo "deb http://downloads.linux.hpe.com/SDR/repo/mcp focal/current non-free" > /etc/apt/sources.list.d/mcp.list
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey1024.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add -
RUN apt-get update && apt install -y amsd storcli ssa ssacli ssaducli hponcfg && rm -rf /var/lib/apt/lists/*

Run with privileged instead of cap-add:

Code:

# docker build --network=host -t amihal/ams .
# docker run --device /dev/mem --privileged --network=host -i -t amihal/ams bash
# hponcfg -w ilo.cfg
HP Lights-Out Online Configuration utility
Version 5.6.0 Date 6/30/2020 (c) 2005,2020 Hewlett Packard Enterprise Development LP
sh: 1: lsmod: not found
sh: 1: lsmod: not found
sh: 1: lsmod: not found
Firmware Revision = 2.55 Device type = iLO 5 Driver name = ^


Management Processor configuration is successfully written to file "ilo.cfg"
# cat ilo.cfg
<!-- HPONCFG VERSION = "5.6.0" -->
<!-- Generated 4/9/2022 16:51:55 -->
<RIBCL VERSION="2.1">
 <LOGIN USER_LOGIN="Administrator" PASSWORD="password">
  <DIR_INFO MODE="write">
  <MOD_DIR_CONFIG>
    <DIR_AUTHENTICATION_ENABLED VALUE = "N"/>
    <DIR_LOCAL_USER_ACCT VALUE = "Y"/>
    <DIR_SERVER_ADDRESS VALUE = ""/>
    <DIR_SERVER_PORT VALUE = "636"/>
    <DIR_OBJECT_DN VALUE = ""/>
    <DIR_OBJECT_PASSWORD VALUE = ""/>
    <DIR_USER_CONTEXT_1 VALUE = ""/>
    <DIR_USER_CONTEXT_2 VALUE = ""/>
    <DIR_USER_CONTEXT_3 VALUE = ""/>
  </MOD_DIR_CONFIG>
  </DIR_INFO>
  <RIB_INFO MODE="write">
  <MOD_NETWORK_SETTINGS>
    <SPEED_AUTOSELECT VALUE = "Y"/>
    <NIC_SPEED VALUE = "10"/>
    <FULL_DUPLEX VALUE = "N"/>
    <IP_ADDRESS VALUE = "192.168.1.39"/>
    <SUBNET_MASK VALUE = "255.255.255.0"/>
    <GATEWAY_IP_ADDRESS VALUE = "192.168.1.1"/>
    <DNS_NAME VALUE = "iceberg-ilo"/>
    <PRIM_DNS_SERVER value = "192.168.1.1"/>
    <DHCP_ENABLE VALUE = "Y"/>
    <DOMAIN_NAME VALUE = "localdomain"/>
    <DHCP_GATEWAY VALUE = "Y"/>
    <DHCP_DNS_SERVER VALUE = "Y"/>
    <DHCP_STATIC_ROUTE VALUE = "Y"/>
    <DHCP_WINS_SERVER VALUE = "Y"/>
    <REG_WINS_SERVER VALUE = "Y"/>
    <PRIM_WINS_SERVER value = "0.0.0.0"/>
    <STATIC_ROUTE_1 DEST = "0.0.0.0" GATEWAY = "0.0.0.0"/>
    <STATIC_ROUTE_2 DEST = "0.0.0.0" GATEWAY = "0.0.0.0"/>
    <STATIC_ROUTE_3 DEST = "0.0.0.0" GATEWAY = "0.0.0.0"/>
  </MOD_NETWORK_SETTINGS>
  </RIB_INFO>
  <USER_INFO MODE="write">
  </USER_INFO>
 </LOGIN>
</RIBCL>

Success! This returned the IP and hostname of the iLO. It looks like the hponcfg utility is able to talk to the iLO system ok.
Start AMSD:

Code:

# amsd -f -L
Warning: Failed to connect to the agentx master agent (/var/agentx/bmc): 
amsd[78]: amsd Started . . 

Unfortunately the iLO web interface still says that the Agentless Management Service was not detected, and does not show any of the data that AMS is supposed to be providing. I don't know enough about SNMP to understand if the agentx connection warning is part of the problem. The AMS default is to do a RFC 2741 PING every 15 seconds and it prints this warning each time.

 

[amihal]

I have a solution. I rebooted into an Ubuntu live iso, installed the HPE packages listed in my Dockerfile on bare metal, and used systemctl to show all of the services that those HPE packages actually started. This experiment showed that I needed to start more than just the amsd daemon.

Working Dockerfile:

Code:

FROM ubuntu:20.04
RUN apt-get update && apt install -y curl gnupg
RUN echo "deb http://downloads.linux.hpe.com/SDR/repo/mcp focal/current non-free" > /etc/apt/sources.list.d/mcp.list
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey1024.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add -
RUN apt-get update && apt install -y amsd storcli ssa ssacli ssaducli hponcfg && rm -rf /var/lib/apt/lists/*
ADD start.sh /
RUN chmod +x /start.sh
CMD ["/start.sh"]

start.sh script:

Code:

#!/bin/bash
cpqIde -f -L &
cpqiScsi -f -L &
amsd -f -L &
smad

Docker build command:

Code:

# docker build --network=host -t amihal/ams .

App setup:
* Image repository: amihal/ams
* Image tag: latest
* Image Pull Policy: Never...
* DNS Policy: Use Default...
* Provide access to node network namespace for the workload: checked
* Privileged Mode: checked
* Update Strategy: Kill existing...

The app should start successfully. If you view the app logs, there are messages like this:

Code:

2022-04-15 01:31:24.108419+00:00[INFO  ]: BMC device is /dev/hpilo/d0ccb3
2022-04-15 01:31:24.123241+00:00[NOTICE]: iLO5: 2.55 (2021/10/1)
2022-04-15 01:31:24.123259+00:00[INFO  ]: Reset iLO AgentX session
2022-04-15 01:31:24.525395+00:00[INFO  ]: Open iLO AgentX session in Regular Mode
2022-04-15 01:31:24.525996+00:00[NOTICE]: 'AMS Linux AgentX sub-agent' connected to AgentX master in regular mode
2022-04-15 01:31:24.529621+00:00[NOTICE]: Session initialized with ID: 18
2022-04-15 01:31:24.529691+00:00NET-SNMP version 5.9 AgentX subagent connected
2022-04-15 01:31:24.541471+00:00[INFO  ]: Completed the iLO Socket Init
2022-04-15 01:31:24.541493+00:00[NOTICE]: Start proxy loop
2022-04-15 01:31:24.670397+00:00[INFO  ]: AgentX trap received
2022-04-15 01:31:24.670446+00:00[NOTICE]: AgentX trap MIB-II (.1.3.6.1.6.3.1.1.4.1.0:.1.3.6.1.6.3.1.1.5.1)

The iLO interface should now show that the Agentless Managment Service status is OK. You can see the status of the ethernet ports, the kube-bridge logical network adapter, the storage devices, and even a list of the running processes in the container.

 

[NightShaman]

I have a solution. I rebooted into an Ubuntu live iso, installed the HPE packages listed in my Dockerfile on bare metal, and used systemctl to show all of the services that those HPE packages actually started. This experiment showed that I needed to start more than just the amsd daemon.

Working Dockerfile:

Code:

FROM ubuntu:20.04
RUN apt-get update && apt install -y curl gnupg
RUN echo "deb http://downloads.linux.hpe.com/SDR/repo/mcp focal/current non-free" > /etc/apt/sources.list.d/mcp.list
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey1024.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add -
RUN apt-get update && apt install -y amsd storcli ssa ssacli ssaducli hponcfg && rm -rf /var/lib/apt/lists/*
ADD start.sh /
RUN chmod +x /start.sh
CMD ["/start.sh"]

start.sh script:

Code:

#!/bin/bash
cpqIde -f -L &
cpqiScsi -f -L &
amsd -f -L &
smad

Docker build command:

Code:

# docker build --network=host -t amihal/ams .

App setup:
* Image repository: amihal/ams
* Image tag: latest
* Image Pull Policy: Never...
* DNS Policy: Use Default...
* Provide access to node network namespace for the workload: checked
* Privileged Mode: checked
* Update Strategy: Kill existing...

The app should start successfully. If you view the app logs, there are messages like this:

Code:

2022-04-15 01:31:24.108419+00:00[INFO  ]: BMC device is /dev/hpilo/d0ccb3
2022-04-15 01:31:24.123241+00:00[NOTICE]: iLO5: 2.55 (2021/10/1)
2022-04-15 01:31:24.123259+00:00[INFO  ]: Reset iLO AgentX session
2022-04-15 01:31:24.525395+00:00[INFO  ]: Open iLO AgentX session in Regular Mode
2022-04-15 01:31:24.525996+00:00[NOTICE]: 'AMS Linux AgentX sub-agent' connected to AgentX master in regular mode
2022-04-15 01:31:24.529621+00:00[NOTICE]: Session initialized with ID: 18
2022-04-15 01:31:24.529691+00:00NET-SNMP version 5.9 AgentX subagent connected
2022-04-15 01:31:24.541471+00:00[INFO  ]: Completed the iLO Socket Init
2022-04-15 01:31:24.541493+00:00[NOTICE]: Start proxy loop
2022-04-15 01:31:24.670397+00:00[INFO  ]: AgentX trap received
2022-04-15 01:31:24.670446+00:00[NOTICE]: AgentX trap MIB-II (.1.3.6.1.6.3.1.1.4.1.0:.1.3.6.1.6.3.1.1.5.1)

The iLO interface should now show that the Agentless Managment Service status is OK. You can see the status of the ethernet ports, the kube-bridge logical network adapter, the storage devices, and even a list of the running processes in the container.

Did you ever make this image available? I would be interested in it.

 

[amihal]

Sorry, no. If someone knows how to publish to Truecharts that would be great.

 

[NightShaman]

Did you try putting in a new app request here? https://github.com/truecharts/charts

Or ask about it on the official discord? https://discord.com/invite/tVsPTHWTtr

@stavros-k You think this would be doable? It works, I just do not know enough publishing a container or even how to go about getting it into a chart. I can put in the request later.

 

[vampirebyte]

Thank you for this, I'm in the same situation with a G9... but, wouldn't it make more sense to run it on the bare metal (even tho it's forbidden) so that the real SAS disks and network interfaces are shown?

 

[jgreco]

Thank you for this, I'm in the same situation with a G9... but, wouldn't it make more sense to run it on the bare metal (even tho it's forbidden) so that the real SAS disks and network interfaces are shown?

It depends. Do you want to fight with the system after every upgrade, and need to work this out for yourself when it breaks? Unfortunately that might be your only other option. You MIGHT be able to convince iXsystems to include it in the base system if you put in a feature request, but they have been reluctant to veer off into custom daemons to support this kind of stuff in the past.

 

[vampirebyte]

I know the downsides, I just think that it's worth the trouble of reinstalling it on every update, now I can see the SATA drive details in iLO.

I just tried this on our TrueNAS-SCALE-22.02.4 and it works beautifully, just please NOTE that amsd service is for Gen 10 and above, for Gen 9 like mine and below, you need to install hp-ams package instead.

Solution found here, but the TL/DR version is that you need to add one more extra line on the apt sources:

deb http://downloads.linux.hpe.com/SDR/repo/mcp bullseye/current non-free
deb http://downloads.linux.hpe.com/SDR/repo/mcp stretch/10.62 non-free

reason is stated in the HPE repository:

NOTICE: The health/snmp functionality was moved to the iLO card on HPE ProLiant Gen10 servers. The hp-health, hp-snmp-agents, hp-smh* and hp-ams debs are only to be installed on Gen9 servers and earlier. Gen10 users, please subscribe to "11.xx" or "current" repositories. Gen9 users, please use "10.xx" or earlier.

then you can install everything with:

apt update && apt install hp-ams ssa ssacli ssaducli storcli

The result:

 

[amihal]

I made some updates to the app after upgrading to Cobia:

Dockerfile

Code:

FROM ubuntu:22.04
RUN apt-get update && apt install -y curl gnupg
RUN echo "deb http://downloads.linux.hpe.com/SDR/repo/mcp jammy/current non-free" > /etc/apt/sources.list.d/mcp.list
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | apt-key add -
RUN curl http://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add -
RUN apt-get clean && apt-get update && apt install -y amsd storcli ssa ssacli ssaducli && rm -rf /var/lib/apt/lists/*
ADD start.sh /
RUN chmod +x /start.sh
CMD ["/start.sh"]

start.sh

Code:

#!/bin/bash
cpqIde -f -L &
cpqiScsi -f -L &
amsd -f -L &
smad

Image built on WSL2 Ubuntu:

Code:

docker build --network=host -t amihal/ams .
docker push amihal/ams:latest

https://hub.docker.com/r/amihal/ams

App setup:
Application Name: ams
Image repository: amihal/ams
Image Tag: latest
Image Pull Policy: Always pull image even if present on host
DNS Policy: Use Default DNS Policy where Pod will inherit...
Provide access to node network namespace for the workload: checked
Privileged mode: checked