SOLVED New install of 9.10 boots only to GRUB prompt after adding internal RAID to config in HP ProLiant

[allegiance]

ProLiant DL580 G5, clean new install into internal 64Gb USB flash. First round of install went fine, rebooted after install finished, connected browser and went through wizard. I think the hiccup was when the wizard added the built-in RAID as a drive to the system. Now on reboot, I only get a blank GRUB prompt, no boot menu. I've even tried re-installing, but now it won't even come up after the initial install (I'm assuming because the RAID is now formatted file system). I've been using FreeNAS for a while, but am still not familiar with GRUB. Not sure what to do at the prompt or how to fix it.


HW specs:
Intel Xeon Quad or Hexa-Core Processors: 4x 2.40GHz Intel E7440 Quad Core

Memory 74Gb
Controller: P.400i Smart Array (plan to use for Jails):

Hard Drive | Bay 1: 73GB 15K SAS 2.5"

Hard Drive | Bay 2: 73GB 15K SAS 2.5"

Hard Drive | Bay 3: 73GB 15K SAS 2.5"

Hard Drive | Bay 4: 73GB 15K SAS 2.5"

Optical: + DVD-ROM Optical Drive
RAID Card: SAS9211-8I 8PORT Int 6GB Sata+sas Pcie 2.0 (for ZFS data RAID raidz2)
Intel RAID Expander Card (RES2SV240)
WD Red 4Tb drives

 

[allegiance]

Update, more data - I've tried loading FreeNAS 10, just to see if it can handle the ProLiant, but it will not boot. I get the correct grub boot loader menu (good), but when it tries to boot the OS it gives an "error: overlap detected", and then cycles back to the boot menu. I'm next going to try FreeNAS 9.3 - I've been using that on my custom-built NAS for a while so I have hopes it will work fine on the HP ProLiant.

 

[depasseg]

I think the hiccup was when the wizard added the built-in RAID as a drive to the system

What is this built in RAID that you speak of?

Could you please edit the OP to add the HW system specs of what's in the DL580-G5?

 

[allegiance]

Sorry, the built-in RAID is a P.400i Smart Array. I was planning on using it for a jails volume, letting the RAID controller handle the RAID for it and presenting to the FreeNAS system as a single drive. I will be using SAS9211-8I RAID card with Intel RAID Expander Card (RES2SV240) for the ZFS data raid (using WD Red 4Tb drives). The ZFS pool will be moved from my existing machine into this "new" one once I get it running.


One more bit of new information. Doing a clean load of FreeNAS 9.3 boots fine, even after I create a volume on the P400 RAID. Several reboots were OK. So, I then tried to upgrade to 9.10. Back to not booting. BUT, it's slightly different now - it not boots to a grub rescue prompt rather that a regular grub prompt like it did when I did a clean load of 9.10. But again, I do not know much about grub so don't know exactly how to fix it, if possible.

 

[SweetAndLow]

The word raid is being used way to much in this thread. Just so you know with zfs you should never use raid.

Sent from my Nexus 5X using Tapatalk

 

[F!$hp0nd]

One thing you might want to try mate, is seeing if you can put the card into IT passthrough mode. This way the drives are directly passed to FreeNAS at the bit section of the drives. ie, get rid of RAID card, ZFS does its own RAID called RaidZ*. Also, I got a bit confused reading your post, but did you install to an external USB or to your RAID set? Because it sounds like you thought you installed to your internal/external USB, but more than likely it installed to your RAID array on the LSI card. In alot of the LSI controllers, you have to choose which virtual disk should be set as the boot device. If you boot into the LSI webbios and choose your virtual drive you are installing to, then you should be able to see the properties and then choose "set boot drive".

If you installed to your USB, make sure that your UEFI options are all disabled inside your bios and then make sure your FreeNAS USB (not the installer) is selected for primary boot as non UEFI.

I know the above is a bit confusing, but it sounds like your internal USB is actually not selected as the install location and it is also not the primary boot drive. Also, if you have to use your LSI SAS card to show the drives in freenas, atleast put each drive into its own single disk Raid 0 stripe. This way it shows as individual drives to freenas. Then from their configure RaidZ(whatever level) with the shown drives.


Hope this helps, if not post questions

 

[allegiance]

Again, I apologize for leaving out key information. I am using an internal 64gb Sandisk USB drive for the OS drive. That is where I installed. It's is selected as first boot device, this machine is bios only, no uefi. I installed using CD-ROM.

 

[allegiance]

The LSI raid cards are not installed as of yet - they will be moved from another system once I get this one working. I included them in the system profile to illustrate that I do plan on having a ZFS pool in this machine, but not on the HP raid controller.

To reiterate. The system will boot in 9.10 if the OS is installed on the flash drive and the HP raid is in configured. If I configure the hp raid in freenas as an empty stripe volume, then the machine no longer boots but stops at grub. If I load 9.3 on the USB drive, it boots fine even with the hp raid configured as a stripe volume.

 

[F!$hp0nd]

Ok, first here is the easiest thing I can say....DONT USE HARDWARE RAID!!!!!!!!! in ZPOOL. ZFS is a dataset/volume manager in itself (in plain text a Software RAID). Therefore, having a hardware raid controller anywhere in your setup is a no go. The issue you are having is, that when you install freenas on your USB and then put the USB inside your server, then boot off the usb, then initialize the hardware RAID (HP RAID) on freenas, the server does not care your USB is plugged in. It will try and boot off the hardware (HP) Raid setup. One thing you might want to do, is see if your HP RAID card has an HBA or passthrough IT mode.

If not try this, after installing freeNAS on your USB boot using your USB, configure your system in the webGUI how you want, then reboot the server. If you still get a grub screen, try using the HP boot menu in bios to select your USB. If it boots, your HP server is still defaulting to the HP hardware RAID array as the primary boot device, which you will have to select under the boot tab in the HP Server BIOS.

When you are in the GUI, creating a stripe volume (ie. a single drive dataset) on a virtual Hardware RAID drive, should in theory work, but ZFS is designed to connect with the drives without the raid controller in between, because RAID controllers stop SMART data from being correctly passed through.

The biggest thing I can stress is dont use hardware raid with FreeNAS. There are numerous posts and forum topics about the issues.

 

[allegiance]

There does not seem to be pass through on this hardware raid. I understand in principle that I should use ZFS and not hardware raid, and I am for my data. I've discovered even if I disable the raid controller in bios and pull all the drives from the hardware raid, it still boots to grub

 

[allegiance]

Also, I installed freenas onto the USB drive while in the server, not installed before hand and inserted it later. Not sure if that makes a difference, but I thought I'd mention that since it seems like that was the impression I gave for how I installed

 

[allegiance]

If I cannot put the hardware raid into pass through mode, maybe I'll remove it. It was not installed intentionally for any purpose, but I thought since it was already in the server I could use it for the jails - I've had a separate single drive for raids in my other system for quite some time with absolutely no issue so I assumed that the hardware raid would present itself to the OS as a single drive similar to my other system and I'd have the same setup basically, aside from the added benefit of the hardware raid instead of a single physical drive for the jails. If I can't get the hardware raid in pass through I'll see if I can just replace it with another single drive for the jails.

 

[MatthewSteinhoff]

To reiterate. The system will boot in 9.10 if the OS is installed on the flash drive and the HP raid is in configured. If I configure the hp raid in freenas as an empty stripe volume, then the machine no longer boots but stops at grub.

1. I wouldn't be horrified if you used internal RAID as the FreeNAS boot device. I'm in the minority but I think RAID is at least as reliable as a USB thumb drive. We backup our FreeNAS configuration daily and consider the boot device (USB or whatever) disposable. It is the component most apt to fail and the most easily replaced.

2. If you're using a USB drive as your boot device and you're planning on using the internal RAID controller in a RAID configuration for your jails and in that configuration it will boot, why not just configure it that way now? You're not going to use an empty stripe in your final configuration, right?

I'm really not sure what you're trying to accomplish but it sounds like your intended configuration is one that actually boots and it is only this interim configuration with which you're having problems?

Cheers,
Matt

 

[allegiance]

1. I may end up using the RAID as the boot device if that solves my problems. I was trying to use the USB since that is what seems to be generally recommended. Configuring the boot as USB and the jail as the hardware RAID does boot in 9.3, but when I try to clean install or upgrade to 9.10, it no longer boots that way. I'd like to run 9.10, since that is the current version. Again, I am grub ignorant, but it looks to me like once 9.10 is installed, it does something to grub that is preventing the system from finding the Usb drive as the boot drive. This piece of data I collected seems odd to me, but maybe it's not - when FreeNAS 9.10 is running, when I go to add the hardware RAID as a volume, it's listing the USB drive as sd1 and the hardware RAID as sd0. But, when I do an "ls" from the grub prompt, it shows my USB drive as HD0 and the hardware RAID as HD1. Are FreeNAS 9.10 & grub miscommunication on which drive is which?

 

[allegiance]

OK. More data. More interesting - I'm still thinking it's a grub config issue with 9.10. Even with the HP StartStart CD, I cannot put the RAID into passthrough mode. Even if I destroy the RAID and leave all drives unconfigured, on boot the controller automatically sets them up as RAID 5. So, the hardware RAID will remain a hardware RAID. Taking Matthew Steinhoff's advice, I thought I'd try setting up the hardware RAID as the boot volume. Here's where I think it gets interesting:

FreeNas 9.3:
-With BIOS set to use USB drive as boot volume and FreeNAS 9.3 installed on USB drive, system boots fine with or without the hardware RAID setup as an empty stripe volume (setup within the FreeNAS GUI).

FreeNAS 9.10:
-With the BIOS set to use the USB drive as boot volume and FreeNAS 9.10 installed on USB drive, system boots fine with the hardware RAID not configured inside the FreeNAS GUI
-With the BIOS set to use the USB drive as boot volume and FreeNAS 9.10 installed on USB drive, system will stop boot at grub with the hardware RAID configured as empty stripe volume (setup within the FreeNAS GUI)
-With the BIOS set to use the USB drive as boot volume and FreeNAS 9.10 installed on USB drive AND hardware RAID, system will stop boot at grub (this was after attempting a clean load of 9.10 on the hardware RAID without reformatting the USB drive)
-With the BIOS set to use the hardware RAID as the boot volume and FreeNAS 9.10 installed on USB drive AND hardware RAID, system will boot completely - BUT, it boots to the USB drive, not to the hardware RAID

So, from what I can tell (and I may be wrong since again I know basically nothing about grub), it looks like the server is looking to whichever drive the BIOS tells it is the boot drive for boot info, but the 9.10 grub config on the USB drive is telling the system to boot from the hardware RAID, and the grub config on the hardware RAID is telling the system to boot from the USB drive. I'm going to attempt to copy off the grub config files from both drives and see if I can figure out what needs changed on them.

 

[allegiance]

Addition - With the BIOS set to use the USB drive as the boot volume and FreeNAS 9.10 installed on USB drive AND hardware RAID, system will stop at grub rescue prompt. :-/

I'm stumped

 

[allegiance]

I think I solved it. I replaced the new SanDisk 64Gb flash drive with an old Kingston 16Gb flash drive I've had laying around. So far, I'm on my 3rd reboot with the USB set as my boot volume and the hardware RAID configured as my stripe Jail "drive". This is with FreeNAS 9.10. I'll post again after running for a bit, but thank you to all who tried to help me out with thoughtful suggestions.

 

[allegiance]

I have been running for several days now, rebooted at least a dozen time, and imported in my ZFS pool from my now retired system. No more boot problems - it does seem it was the 64Gb SanDisk USB drive all along.

 

[Bruin]

Same problem here with a Sandisk 64GB USB drive, "SanDisk Ultra Fit CZ43 64GB USB 3.0 Low-Profile Flash Drive
SDCZ43-064G-G46 [Older Version]" (bought years ago) installed to a HP Proliant ML10. Boots to grub menu only after upgrade or clean install. Tried reformatting and zeroing out USB drive with same end result. Installed 11.1U1 to generic 16GB USB 3 drive afterwards without difficulty.