Weird Installation Problems

[Stephen Hayne]

Hi - I'm trying to install freeNAS (9.3, 9.10, 10, any version really), on a SuperMicro 4U system that is a duplicate of another SM 4U system that I am successfully running 9.10 on (~90TB storage) and have used for over a year. On this particular hardware, I ran Nexenta 4 for several months and have decided that freeNAS is where we need to be. Here is the system description:

4U Supermicro 847A, X9DRH-iF, 2x Xeon E5-2603, 64GB RAM, FW 3.36, BIOS 3.0b, IPMI config'd
2x 8GB Patriot USB Boot drives (mirrored) for FreeNAS OS
1x Quad mSATA PCIe card with 2x Crucial 250 GB Internal SSD (for cache)
5x LSI 9207-8i HBAs (running firmware 16.00.00.00-IT - I will update this after install)
various SATA drives (on HCL list), ranging from 6TB/8TB to 1TB SSD
1x Chelsio PCIe Dual 10gig Intel X520 (SPF+)

I cannot for the life of me get freeNAS successfully installed. When using a USB cd or IPMI iso, I get umass errors as in the first screen cap. And the install eventually runs, but then when booting to the mirror'd (NEW!) usb drives, I get the errors in the second screen cap. I have also tried 2 mirrored SSDs connected to the SATA on the MB with no joy, i.e., I get the "grub rescue" error.

Can someone suggest how I might proceed with a good install? I think the MB is OK, and have tried all combinations of the 4x USB 2.0 ports. Tried removing all HDD/SDD and just install to usb. Just running out of ideas... I've search and read as much as I can handle at this point!

We use these storage systems to serve 24 HP G6/G7 blades hosting VMware 6.0 (500+cores, 2TB RAM), for both teaching (800+ student VMs) and data analytics research projects.

Stephen

--
Dr. Stephen C. Hayne, Professor, CIS, Colorado State University
__!__ (970)491-7511(w) (970)491-5205(f) (970)204-4040(h)
___(_)___ "I love to fly AngelFlights! 310I - N8109M
"http://selfsynchronize.com/hayne/"

 

[Spearfoot]

The first screen dump looks like a bad CD, perhaps a corrupted installation image? Might be worthwhile to download another copy.

Did you try creating a FreeNAS installation USB stick and installing from that? With 4 x USB 2.0 ports, you should be able to boot from one of them and install to a pair of mirrored USB sticks (if that's where you want FreeNAS to live).

 

[Robert Trevellyan]

Sanity check: did you checksum the downloaded ISO?

 

[Stephen Hayne]

The first screen dump looks like a bad CD, perhaps a corrupted installation image? Might be worthwhile to download another copy.

Did you try creating a FreeNAS installation USB stick and installing from that? With 4 x USB 2.0 ports, you should be able to boot from one of them and install to a pair of mirrored USB sticks (if that's where you want FreeNAS to live).

I've done all of these things. I've burned CDs (on different burners) of 9.3, 9.10 and 10, installed same to USB sticks and run into various errors... remember, I have another one of these systems that is an exact duplicate that is already running 9.10 - but the initial install was over 16 months ago. I don't remember any of these issues on that box.

 

[Spearfoot]

You've done everything I would've done:

1> IPMI-attached remote images (USB & ISO) didn't work, so you tried:
2> Directly attached USB drive & CD, which also didn't work.

And I assume you verified the checksum of your installation image(s).

@jgreco, @cyberjock, @joeschmuck , @Mirfster, @Bidule0hm : any idea how @Stephen Hayne should proceed?

@Stephen Hayne , would you please post as much detailed error information as possible?

 

[jgreco]

I think the MB is OK, and have tried all combinations of the 4x USB 2.0 ports.

Okay, well, I'd suggest you've jumped way ahead in the process here. If you merely "think" the MB is okay, you need to do basic burn-in and testing. Loading FreeNAS onto a new system is usually not the right thing to do.

https://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/

Go into the BIOS, and select the option to "Reset to factory defaults." Then pick up in the above-linked post at "Configuration."

Pay particular attention to running other testing utilities, which should load easily and seamlessly via IPMI or CD or USB key. If they don't, you have a bad system and the system needs to be ripped apart and tested. If you're able to run all the usual burn-in tools but FreeNAS still fails to load, that's ... very unusual but we could look at that too.

Best bet at this point is that there's something wrong with the system.

 

[Mirfster]

Sanity check: did you checksum the downloaded ISO?

Did you do this?

If desired, grab a slightly older version of the ISO from: http://download.freenas.org/9.10/STABLE/

Try installing FreeNas to a USB/SSD on a different system (Like your desktop/laptop for example). If it installs fine, then disconnect that USB/SSD; plug it into the Server (without any other drives attached). See if it boots up fine then; if not then the issue lies with the SuperMicro and I would start focusing there. If it doesn't even install and work from the USB/SSD attached to your other system then let us know.

Don't worry about mirroring right now, you can always do that later...

Pondering this right now:

 

[Stephen Hayne]

Okay, well, I'd suggest you've jumped way ahead in the process here. If you merely "think" the MB is okay, you need to do basic burn-in and testing. Loading FreeNAS onto a new system is usually not the right thing to do.

https://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/

Go into the BIOS, and select the option to "Reset to factory defaults." Then pick up in the above-linked post at "Configuration."

Pay particular attention to running other testing utilities, which should load easily and seamlessly via IPMI or CD or USB key. If they don't, you have a bad system and the system needs to be ripped apart and tested. If you're able to run all the usual burn-in tools but FreeNAS still fails to load, that's ... very unusual but we could look at that too.

Best bet at this point is that there's something wrong with the system.

OK - I'll back up a few steps, but remember, this system is a year old and was running Nexenta CE for the last year as we checked that NAS out. A few days ago, I plugged in the Nexenta SSD and it boot up just fine. Also, I've booted to Ubuntu 14.o4 and run memtest for 3 days (over the weekend) with no errors.

I will try a different USB DVD player, and also try the steps where I don't put the new boot usb drives in until it gets to the install screen - someone complained that might be a problem (on their system).

 

[Stephen Hayne]

Did you do this?

If desired, grab a slightly older version of the ISO from: http://download.freenas.org/9.10/STABLE/

Try installing FreeNas to a USB/SSD on a different system (Like your desktop/laptop for example). If it installs fine, then disconnect that USB/SSD; plug it into the Server (without any other drives attached). See if it boots up fine then; if not then the issue lies with the SuperMicro and I would start focusing there. If it doesn't even install and work from the USB/SSD attached to your other system then let us know.

Don't worry about mirroring right now, you can always do that later...

Pondering this right now:
View attachment 12341

Uhhhh - even though I teach a graduate course in security, I am embarrassed to say I did not... I just downloaded all three stable versions (at different times) to my work desktop. I suppose that could be an issue, and I will check the hash now.

 

[jgreco]

OK - I'll back up a few steps, but remember, this system is a year old and was running Nexenta CE for the last year as we checked that NAS out. A few days ago, I plugged in the Nexenta SSD and it boot up just fine. Also, I've booted to Ubuntu 14.o4 and run memtest for 3 days (over the weekend) with no errors.

I will try a different USB DVD player, and also try the steps where I don't put the new boot usb drives in until it gets to the install screen - someone complained that might be a problem (on their system).

Okay, then my next guess also moves on to the "check your download". Obviously something's amiss.

 

[Stephen Hayne]

Okay, then my next guess also moves on to the "check your download". :) Obviously something's amiss.

All hashes are fine. Tried 3 new/different USB drives and all failed. I wonder if we are having difficulties with this partition problem and the SuperMicro X9DRH:

https://forums.freenas.org/index.php?threads/freenas-stable-9-3-unable-to-boot-usb.28103/

Our other identical box boots from mirror'd SanDisk USB drives, but we started with 9.1 on that system... so maybe we still have the proper partitioning for boot for this particular MB. 9.3, 9.10, and 10.x will NOT install and boot onto USB...

Still digging.

Stephen

 

[Mirfster]

Our other identical box boots from mirror'd SanDisk USB drives

Hmm, would it be possible to take a new/clean USB stick and try the install on the other machine? Perhaps only have the USB stick connected and all other drives disconnected (just to be safe). That would provide some level of comparison...

 

[Stephen Hayne]

Hmm, would it be possible to take a new/clean USB stick and try the install on the other machine? Perhaps only have the USB stick connected and all other drives disconnected (just to be safe). That would provide some level of comparison...

Sigh - likely not, as that box is used in "production", i.e., since it is summer, supporting a couple research projects. I am very hesitant to tinker with it until I can get more progress on this current system. I would hate to have TWO inop systems and be completely dead in the water!

A diagnostic that might be useful is to try to install 9.1 on the current system (like we did over a year ago to the working system), onto new USB drives and then see if we can "upgrade" through the versions. Does anyone know if the upgrade path changes the boot partitions, etc.?

 

[Sakuru]

Have you tried removing all of your PCIe cards?