SOLVED USB boot on MicroServer Gen8?

[danb35]

I just picked up a used HPE Proliant Microserver Gen8 to use as a second TrueNAS server at my parents' home--Time Capsule replacement for them, replication target for me. Seems like a nice little piece of kit, compact, has remote management, four drive bays, and reasonably economical for a Xeon-based server. Problem is that I can't get it to boot from USB. Like, at all. It won't boot the TrueNAS installer from a USB stick (and that same stick will boot other computers into the installer), whether in an external USB2 port, an external USB3 port, or the internal USB2 port. It will boot the ISO through the iLO virtual media feature, so I installed TrueNAS onto a USB SSD that way--which worked fine, but was very slow (almost an hour and a half to install). But the system won't boot from the USB SSD either; it says "Non-System disk or disk error"--same as it does for a USB flash drive.

I tried resetting the BIOS to defaults, no change.

I'd like to get this working. Other options are:

• Put a standard SATA SSD into one of the drive bays. In the short term, I only plan on using two spinners, so I'd have two bays free--but that would mean I wouldn't be able to add a second pair later on without making adjustments. I'm installing TN right now this way to test, but I expect this to work (but then, I expected the USB boot to work--but I was able to boot a different OS from the SSD, so I expect this will be fine).
• Connect a SSD to the SATA port for the optical drive, and follow the instructions here (https://www.admfactory.com/hp-microserver-gen8-boot-from-ssd-install-on-odd-bay/) to boot from it. This doesn't take a drive bay, but it's turning on the onboard hardware RAID controller--bad juju for TrueNAS. I wouldn't mind it too much for just the boot pool, but it seems like an all-or-nothing situation.

If USB can't boot, the options don't seem great. Any suggestions to get this to work?

 

[KrisBee]

@danb35 Have a look at this thread: https://www.truenas.com/community/threads/bootloader-on-usb-freenas-on-ssd.56430/

In short, create a custom grub boot usb mem stick which boots an SSD attached to SATA port for the optical drive on which you install TrueNAS via ILO. In this config the onboard fake RAID B120i is set to work in AHCI mode in the BIOS. Do check what the boot order is set to in the BIOS as it might explain your odd USB boot behaviour.

If problem persists, try installing FreeNAS first and then upgrade to TrueNAS. Only the first two leftmost bays operate at SATA III speeds.

 

[danb35]

Have a look at this thread: https://www.truenas.com/community/threads/bootloader-on-usb-freenas-on-ssd.56430/

Thanks. I think I'd seen it before (and as I look back at it, I obviously had, as I replied there), but hadn't bookmarked it, so...

I like that it gives a way to boot from the SSD on the ODD port without invoking the RAID controller. I'm not too keen on the fact that it requires manual configuration for which "disk" is going to boot, but I don't anticipate making frequent changes to the disk configuration in the system such that this would be a big problem.

My biggest concern with this method, though, is that it requires the system to boot from USB, and that's the thing it doesn't seem to want to do. Even with nothing else connected to the system--no hard drives, no SSDs, no other USB devices, just the TrueNAS installer USB, which did boot on another system. I haven't yet tried booting anything else from USB; that may be a logical next troubleshooting step. I wouldn't think it would be an issue with the FreeBSD bootloader, as the installer ISO booted just fine (if slowly) as virtual media. But boot order ought to be fine:

For right now, I installed a SATA SSD in the first drive bay and installed to that. It boots and runs just fine. So the system will run TrueNAS, anyway--not that I'd expected otherwise. I think I'll try burning something like a Ubuntu live USB and see if it boots that.

OK, further testing shows that the system will boot a Ubuntu 18.04 live USB, but will not boot a FreeNAS 11.3 installer USB. That does give some hope for the boot method discussed in the thread you linked.

 

[Elliot Dierksen]

I have a vague recollection of having USB install boot issues on another family of servers. I wonder if it would work if you booted from optical media, but installed on the internal USB drive.

 

[Basil Hendroff]

This is odd. I run CORE on Gen8's booting from USB and haven't ever come across this issue. Is the Gen8 firmware up-to-date?

 

[danb35]

Is the Gen8 firmware up-to-date?

Latest I can find from HPE:

And it gets curiouser and curiouser. The TrueNAS 12.0-U5.1 installer won't boot from a USB stick (it will boot as remote media through iLO). Neither will an installation of TrueNAS 12.0-U5.1 on a USB SSD, with BIOS boot enabled. The FreeNAS 11.3-U2.1 installer won't boot from a USB stick.

TrueNAS 12.0-U5.1 will boot just fine from a SSD in a drive bay, so it's nothing with TrueNAS as such.

But here's where it gets strange. It does boot from the FreeBSD 12.2 installer USB. It also boots from the Ubuntu 18.04 Desktop live USB. So it will definitely boot from USB, and it will even boot FreeBSD from USB--it just, for some reason, doesn't want to boot Free/TrueNAS from USB.

 

[Elliot Dierksen]

Does the BIOS have a setting for UEFI versus Legacy boot? If so, try changing that.

 

[danb35]

I haven't been able to find a setting for UEFI vs. Legacy. Other sources indicate it only supports legacy boot, and I know I had to install TrueNAS with legacy boot to get it to boot--it wouldn't if I installed it with UEFI.

 

[Basil Hendroff]

Latest I can find from HPE:

Your firmware is the most recent available.

And it gets curiouser and curiouser.

I think I've solved the mystery. To my surprise, I was able to replicate the behaviour you're describing. I wasn't able to run the TN CORE 12-U5.1 installer from a USB stick on a Gen8 microserver I've been using for testing TN SCALE. I then remembered that the Gen8 can be quite fussy when it comes to what brand/model of USB stick is used. USB sticks that worked for me on the Gen7 series failed on the Gen8. I had put the TN installer on a SanDisk Cruzer Facet for this test and the Gen8 failed to boot from that stick. I then put the installer on a Toshiba USB stick and Gen8 successfully booted from that stick. My suggestion... try a different model or brand of USB stick until you find one that works.

 

[danb35]

I then remembered that the Gen8 can be quite fussy when it comes to what brand/model of USB stick is used.

Interesting, and that's something I hadn't messed with. In fact, all the "USB stick" boot experiments were done with the same stick, a Sandisk Cruzer. I'll have to try messing with this variable as well.

 

[Basil Hendroff]

I'll have to try messing with this variable as well.

My signature block lists the sticks that work for me. You may find this helpful.

 

[danb35]

Right now, the system's doing burn-in on a pair of 16TB disks, so I'll investigate further once that's finished. It'll be a few days.

 

[danb35]

OK, disk testing is complete (badblocks takes a while to run on a 16TB disk), so on to trying to boot the system from the ODD SATA port. Since the system isn't in production yet, I figure I'll generate the Grub boot device on it. Put the Ubuntu installer back on the Sandisk Cruzer 8GB I'd used before, from which the system again booted just fine, and installed it to the JMicron USB SSD I'd tried previously. Nope, still doesn't want to boot from that.

Well, the system shipped with a 1TB spinner I won't otherwise be using, so let's install to that--that works fine. Booted Ubuntu from that, used Ubuntu to install Grub to a microSD card following https://github.com/ndeineko/grub2-bios-uefi-usb (and ignoring the UEFI stuff, as the Gen8 doesn't support UEFI), created the grub.cfg (adding a default and timeout), and it works. SSD is connected to the ODD port, the system boots from the microSD, loads Grub, and Grub loads the FreeBSD loader from the SSD.

 

[danb35]

OK, I think I've finally reached where I'm going to be with this. The "bootloader on microSD, chain to SSD" worked, but I couldn't come up with a way for it to automatically work with the boot pool regardless of its drive number (i.e., in case I add another pair of spinners to the system). I could, of course, edit the grub.cfg, but what are the odds I'd remember what was going on by then?

So, following a suggestion from @Patrick M. Hausen in another thread, I bought a m.2 SSD, a m.2-to-USB adapter, and installed to that. And it's booting fine, and I can take back the 250GB SSD I'd been using as a boot device for other uses. And I have a spare 32GB microSD card that I'll use with one of my cameras.