Any Grub gurus?

[danb35]

tl;dr: I'm trying to refine this:

Bootloader on USB, FreeNAS on SSD

Hello! I got an HP Proliant gen8 and i want to install FreeNAS on an SSD (The SSD is in the ODD slot). The Problem is that the proliant can't boot from the ODD slot with activated AHCI mode, but it can boot from an USB Stick. So i thought it would be a nice idea to have to bootloader on the USB...

www.truenas.com

I recently bought a HPE Microserver Gen8 to set up at my parents' place, and want to follow the recommendation to not boot from USB sticks (aside from that system being pretty picky on what USB sticks it will boot from anyway), but also don't want to take up one of the only four drive bays with a boot SSD. I've thus connected it to the fifth SATA port, designated for the optical drive, and following the method at the link above, have installed Grub on a micro SD card which will then boot TrueNAS from that drive (that SATA port isn't bootable on the Microserver Gen8). This all works well. Except...

The grub.cfg used there hard-codes the boot to a specific device, hd3 in my case (I only have one pair of drives in the system to start). It'd be nice to have the .cfg file written such that it wouldn't break on adding a couple of drives (or even removing a drive). A couple of others in that thread were trying to figure out if it were possible to have Grub determine the highest available drive number, and then boot from that--and that doesn't seem to be possible (though I'd be happy to hear I'm wrong here, with examples). But I'm finding a number of sources indicating that it should be able to find the boot pool by name, and then boot from it. E.g.,

How to create a GRUB entry for a FreeBSD on ZFS installation?

To be clear what I want is a FreeBSD GRUB2 entry that boots FreeBSD on a ZFS (name: zroot) with BIOS firmware (not UEFI). The operating system GRUB is installed on is Arch Linux if it helps, Arch is

unix.stackexchange.com

But it doesn't work (and neither do the suggestions at https://itectec.com/unixlinux/ubunt...u-tools-or-liveusb-to-find-what-partitions/); that menu entry gives me this:

The Grub manual also suggests it ought to work, though its example doesn't seem to be dealing with quite this scenario:

GNU GRUB Manual 2.12

GNU GRUB Manual 2.12

www.gnu.org

 

[sretalla]

Just a thought, but maybe the method 2 has some clues there... although I don't have the HP box to test anything on. https://www.thesecmaster.com/three-different-ways-to-boot-a-raspberry-pi-from-a-usb-drive/

 

[KrisBee]

@danb35 What exactly was "it" grub stanza that did not work? The format taken from here https://forums.freebsd.org/threads/grub2-has-no-zfs-support.25164/#post-352773 seems to work for me. e.g:

Code:

menuentry 'FreeNAS BSD loader' --class freebsd --class bsd --class os {
    insmod zfs
    search --set --label freenas-boot
    kfreebsd /ROOT/12.0-U5/@/boot/zfsloader
}

But isn't this swings and roundabouts if you have to reference a given boot environment? Your grub.cfg is not going to cope with TrueNAS updates and rollbacks.

IIRC GRUB doesn't care which two drive bays are used when just two HDDs are in use, the SSD in the OD bay always enumerates to the same number. So what about creating a grub.cfg with multiple entries and different labels to cater for each possible number of HDDs where the set root command changes appropriately?

If your grub.cfg contains these two lines at the top, then via iLo you'd have the chance to over ride the "default" grub boot option during the boot process:

Code:

set default="0"
set timeout="20"

 

[danb35]

The block I'd tried that produced the result in the screen shot was from the first link in my OP (https://unix.stackexchange.com/ques...-grub-entry-for-a-freebsd-on-zfs-installation):

Code:

menuentry "TrueNAS Core 12" --class freebsd --class bsd --class os {
    insmod zfs
    insmod bsd
    search -s -l boot-pool # With 12.0, the name of the boot pool has changed
    kfreebsd /@/boot/zfsloader
    kfreebsd_loadenv /@/boot/device.hints
}

My web searching hadn't found that thread on the FreeBSD forum. I could try that, but if it's mandatory to specify a boot environment, this method just trades one problem for another.

So what about creating a grub.cfg with multiple entries and different labels to cater for each possible number of HDDs where the set root command changes appropriately?

That's a simple solution I hadn't considered. It still doesn't quite accomplish what I'd like to do (specifically, automatically boot from the SSD regardless of how many spinners are in the system), but it gives an easy way to tell it to boot from a desired drive number.

 

[Patrick M. Hausen]

Why don't you use a proper SSD in an M.2 to USB case and the internal USB header?

 

[danb35]

Why don't you use a proper SSD in an M.2 to USB case and the internal USB header?

Kind of an orthogonal solution, but I suppose it could work. Concerns I have are (1) the M2/USB cases I've seen are a little chunky, so I'm not sure it would fit in there; and (2) as yet, I haven't been able to get anything FreeNAS or TrueNAS to boot from USB on that system (though the FreeBSD 12 installer booted from USB just fine).

 

[Basil Hendroff]

Concerns I have are (1) the M2/USB cases I've seen are a little chunky, so I'm not sure it would fit in there; and (2) as yet, I haven't been able to get anything FreeNAS or TrueNAS to boot from USB on that system (though the FreeBSD 12 installer booted from USB just fine).

I have a Gen8 that I'm testing SCALE on. I've not been able to boot SCALE from a USB stick. I don't think it's possible. However, the solution that @Patrick M. Hausen proposed in this post solved the problem for me. I've just this evening transitioned a primary CORE server on a Gen8 to use a mirrored pair of these drives for booting TrueNAS CORE. They work flawlessly and are super fast compared to USB sticks. The downside is, compared to USB sticks, they're not cheap.

A limitation of the Gen8 is that it only supports 16GB of RAM. With lots of jails and services running, the RAM available for ARC becomes a limiting factor. I have an SSD in the ODD bay that I have set up as an L2ARC cache. I've managed to achieve considerable performance gains with this configuration.

 

[danb35]

With lots of jails and services running,

Main purpose of this system is backup--a Time Machine share for my parents (replacing their ancient Apple Time Capsule); a replication target for me. One jail (xmrig, might as well use spare cycles for a little crypto), and one VM for Pi-Hole allocated 1 GB of RAM. I won't have a ton of ARC, but it should be adequate for its needs.

Now, the Gen10+ I have on the way will be a different story...

 

[Basil Hendroff]

Now, the Gen10+ I have on the way will be a different story...

I'm jealous! Still new and way overpriced Down Under. Hope you included the iLO card in the configuration as well? That's the nice thing about the Gen8. For me, it's been the sweet spot of the HP microserver range with built-in iLO, choice of processor, and cheap and ready availability on the secondhand market.

 

[danb35]

Yeah, I'm annoyed that iLO needs an extra card, but it's under US$40 extra, so not too bad. I'm thinking it will be an addition to my Proxmox cluster.

 

[KrisBee]

@Basil Hendroff Useful to know SCALE is not able to boot from USB on the HP gen8 if I get round to testing it on real hardware. Maybe I should look again at the SSD in an USB enclosure thing.

If it's not too off-topic, I'd interested to know how a SDD in the OD, which is limited to sata2 speeds, gave performance gains when used for L2ARC read cache. What was the use case, etc?

 

[Basil Hendroff]

If it's not too off-topic, I'd interested to know how a SDD in the OD, which is limited to sata2 speeds, gave performance gains when used for L2ARC read cache. What was the use case, etc?

@KrisBee As I rebooted the server in question overnight after upgrading the boot drives, I'll need a few days for the ARC to reduce in size and for the L2ARC to warm up. I'll come back with some more quantitative data then. This is a point-in-time snapshot of the current situation. I expect the L2 cache to be hit more frequently as ARC continues to diminish in size and the L2 cache continues to grow.

There are 19 jails on this server of which nine are active WordPress sites. Of the remainder, heavy hitters include Plex and Nextcloud. I'm gradually maneuvering into the position of splitting the WP sites off onto their own Gen8 server, but the L2ARC cache has allowed me to extend the life of the current arrangement.

 

[KrisBee]

@Basil Hendroff Thanks for the explanation and it's interesting to see just how you've utilised your Gen8 microserver.

 

[Basil Hendroff]

@KrisBee Just a day later and ARC has reduced in size by another 2GiB to just 4GiB or 25% of available RAM. The L2 cache has kicked in nicely. It's doubled in size and is being hit on average 50% of the time.

Without L2ARC, some of the symptoms I experienced included sluggish WordPress performance, exhausted swap space and specific jails (Plex and rslsync) randomly stopping. With the L2 cache, the symptoms magically disappeared. I believe the symptoms stemmed from high storage I/O latency, which the L2 cache addressed.

 

[KrisBee]

@Basil Hendroff With all the warnings, I never considered using a RAIDZ1 pool on my microserver. I wonder if the extra IOPS of a stripe of mirrors would have performed better. Still off topic, anything particularly notable when testing SCALE on the E3-1220L V2 Gen8 you 're have, e.g CPU load, memory usage, share performance, etc. ?

 

[chruk]

I am not an expert, but I did find this page which helped me a lot to debug a grub issue a few days back.

https://www.csparks.com/BootFedoraZFS/GrubExpressions.md

 

[blanchet]

I cannot tell for your Grub issue, but I have HP Microserver Gen 8 for years.

• I boot from a small SSD SATA + USB3-to-SATA adapter that I plug to the internal USB connector.
• I put the SSD in the cdrom slot.
• It is very easy and very reliable.
• It works with any operating system and boot loader.

 

[danb35]

Why don't you use a proper SSD in an M.2 to USB case and the internal USB header?

The case you'd recommended in other threads seems to be discontinued, but Amazon still has this:

Amazon.com

A little dicey, as it won't fit into the internal USB at full length, so you have to break it off. But it works fine. A little disappointed to not have found a way to make Grub work as I'd wanted it to, but this should do the trick pretty well indefinitely.

 

[Basil Hendroff]

The case you'd recommended in other threads seems to be discontinued, but Amazon still has this:
Amazon.com

Available on Amazon here, but the solution you've stumbled upon is much cheaper.

With all the warnings, I never considered using a RAIDZ1 pool on my microserver. I wonder if the extra IOPS of a stripe of mirrors would have performed better.

While I love the small form factor of the HP microserver series, the challenge for me is how best to utilise the four drive bays. There are trade-offs when considering storage options.

I've little doubt you would see better performance and a quicker recovery with a stripe of mirrors. In my case, I opted to head towards the apex of the triangle initially and chose RAIDZ1 as the storage option. All pool datasets are replicated to other microservers. With larger disk capacities, resilvering can take 24 hours or more to complete. Replacing all four disks to increase pool capacity will take around four days. There's a non-negligible risk of losing the pool during those four days. Replication is my Alamo. The L2 cache gives me a performance benefit. With the combination of RAIDZ1, replication and L2 cache, I feel like I've nudged my microserver storage more towards the centre of the triangle.

Still off topic, anything particularly notable when testing SCALE on the E3-1220L V2 Gen8 you 're have, e.g CPU load, memory usage, share performance, etc. ?

I haven't done enough to comment on these aspects. My focus has been app-centric at this stage - moving simple apps (not requiring container orchestration) off a CORE Ubuntu VM running Docker and Docker Compose, and on to SCALE one-container Kubernetes pods.

Why don't you use a proper SSD in an M.2 to USB case and the internal USB header?

While this seems to work well, there are some bootup errors that I find disconcerting. Should I be concerned? This is an extract from dmesg...

Code:

ugen0.5: <ElecGear NG-2242A> at usbus0
umass2 on uhub5
umass2: <ElecGear NG-2242A, class 0/0, rev 2.10/0.00, addr 5> on usbus0
umass2:  SCSI over Bulk-Only; quirks = 0x0100
umass2:10:2: Attached to scbus10
(probe0:umass-sim2:2:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
(probe0:umass-sim2:2:0:0): CAM status: SCSI Status Error
(probe0:umass-sim2:2:0:0): SCSI status: Check Condition
(probe0:umass-sim2:2:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
(probe0:umass-sim2:2:0:0): Error 22, Unretryable error
da2 at umass-sim2 bus 2 scbus10 target 0 lun 0
da2: <TS32GMTS 400S S090> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number 20201013ABCD
da2: 40.000MB/s transfers
da2: 30533MB (62533296 512 byte sectors)
da2: quirks=0x2<NO_6_BYTE>

This seems to be specific to CORE. Under SCALE with its Debian Linux parentage, I see this...

Code:

truenas-s# dmesg | grep -i sdb
[    2.568977] sd 6:0:0:0: [sdb] 62533296 512-byte logical blocks: (32.0 GB/29.8 GiB)
[    2.569349] sd 6:0:0:0: [sdb] Write Protect is off
[    2.569419] sd 6:0:0:0: [sdb] Mode Sense: 2f 00 00 00
[    2.569973] sd 6:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    2.570598] sd 6:0:0:0: [sdb] Optimal transfer size 33553920 bytes
[    2.591676]  sdb: sdb1 sdb2 sdb3
[    2.595358] sd 6:0:0:0: [sdb] Attached SCSI disk

 

[danb35]

Available on Amazon here,

Not exactly the same, but looks as though it should do. There seem to be a few issues: (1) the chassis is rather space-constrained, particularly in the area of the internal USB port, so (at least) too-long devices just won't fit; (2) most of the converters/enclosures I was finding wanted to run a cable between the enclosure and the USB port--great for external, obviously not good for internal; and (3) NVMe seems to be more popular than m.2. But still, up and running now.

And for posterity, since the Gen8 is so picky with boot devices, the TrueNAS 12 installer booted fine from an old SanDisk Cruzer micro 16GB. It wouldn't boot (though Ubuntu would boot) from a newer SanDisk Cruzer Glide 8GB. And nothing would boot from my Jmicron 128GB USB SSD.