The Mysterious Case of the Unbootable NAS

[raw]

While you may find it hard to believe the story you are about to read is true. Any you, dear reader, may be the one to solve the mystery!

Cliff Note Version: FreeNAS 11.1-U6 system running for a year suddenly fails to boot with "no boot drive available", can't reinstall even on a new boot disk.

The story begins a little over a year ago. I installed FreeNAS 11.1-U6 on the following system:

Dell PowerEdge R510
BIOS 1.14.0
Firmware 1.52 Build 10
32GB RAM
Dual Pentium E5640
Perc H200i, FW 7.15.08.00-IR, LSI Config v.7.11.10.00

I installed it on a dedicated SSD boot drive. It's been up and running for the past year without issue, until....

One day recently I noticed FreeNAS was down. I investigated only to discover "no boot device available" on the console of the FreeNAS server. This occurred at or about the the time we had a power glitch - a series of brownouts.

I assumed the cheap SSD had failed and went out and purchased a new SSD (two in fact, so I could set up a redundant boot drive this time).

Much to my surprise and dismay, I was unable to reinstall FreeNAS on the new boot drive (an HP 120GB SSD). The install appears to work fine, then near what I assume is the end of the install process I see:

/usr/local/sbin/grub-install: error: unknown filesystem.
The FreeNAS installation on da0 has failed. Press enter to continue...

I tried build 11.2-U6, and I even tried switching from BIOS to UEFI. Nothing seemed to make any difference. The error reported by 11.2-U6 is:

ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool freenas-boot
gptzfsboot: failed to mount default pool freenas-boot

After several attempts I began to suspect a BIOS setting had been reset by the power outage, or perhaps the server had developed a hardware issue.

And it is here that our story takes a strange and mysterious turn. On a whim, I attempted to install Ubuntu Server. Lo and Behold, the install worked! The system booted without issue.

So, the server hardware appears to be in working order. I am able to install and boot another OS (Ubuntu), but I cannot get FreeNAS re-installed.

Who will be the super sleuth that will solve this dark mystery and restore FreeNAS to its rightful place in my home lab? Sherlock are you out there? Poirot? Columbo?

 

[raw]

Here's a little additional detail. I CAN install FreeBSD 11.1. I still CANNOT install FreeNAS 11.1 (even though I was able to install it a year ago, and it ran for the past year.)

If I press enter after the "Press enter to continue..." prompt, I am returned to the install menu where I can open a shell. From the shell I can see that a ZFS pool has been created and appears to be properly configured:

 

[Jessep]

Did you start with the basics?

• How are you booting the install media? USB? DVD? Virtual Media?
• Did you try a different method?
• Is your install media image corrupt?
• If you are installing by USB how did you image the USB drive?
• Did you try a different imaging program?

It may have nothing to do with your server or settings.

If you have iDRAC enterprise I would suggest virtual media.

 

[raw]

I am currently booting from iDRAC 6 Virtual Media. The FreeNAS 11.1 iso is the original ISO I downloaded and installed from last year. The FreeNAS 11.2 iso I tested is a fresh download. I successfully installed FreeBSD via Virtual Media. I will attempt to install to USB later this afternoon.

 

[Plato]

maybe secure boot is activated in BIOS.. This is like the problems from 90s.. I had problems like these with HDDs with bad sectors. But as you said this is weird.