Cant boot into any version of FreeNAS after 11.1 upgrade

[drwoodcomb]

Last night I updated Freenas from 11.0 U4 to 11.1

Something odd has happened and now I am unable to boot up into any version of FreeNAS. I dont know of any way to copy and paste the output during boot up so I took pics with my phone and will try typing out what the problem seems to be.

ada0 is a SATA3 hard drive connected directly to the motherboard
ada1 is a SATA3 hard drive connected directly to the motherboard
da1 to da7 are SATA3 hard drives connected to an LSI card in IT mode
da8 and da9 are the two samsung USB drives I use for the operating system. They are set to be redundant.

Here is where I think the problem seems to be:


da9 at umass-sim1 bus 1 scbus9 target 0 lun 0
da8 at umass-sim0 bus 0 scbus8 target 0 lun 0


a bit further in the booting process it gets stuck with a "CAM status: CCB request completed with an error"


(da9:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 01 2b 45 31 00 00 80 00
(da9:umass-sim1:1:0:0): CAM status: CCB request completed with an error
(da9:umass-sim1:1:0:0): Retrying command


This will go on for a bit and then it does the same thing with the other USB drive (da8:umass-sim0:0:0:0)

I am not sure how to proceed from here. Did both my boot USBs fail at the same time? That seems hard to believe. Any help would be greatly appreciated. Everything was fine until I upgraded to FreeNAS 11.1 and even trying to boot into previous versions of FreeNAS gives me the same problems.

 

[joeschmuck]

Please realize that having a redundant/mirroed boot device does not mean that when the primary one fails that the secondary one will take over. You need to use true RAID Mirroring for that to happen. I suspect that da9 is corrupt, probably dying. Swap the USB drives and try to bootstrap your system. If it works then you know one of your devices is bad. The fastest and probably best solution is to replace the boot devices with a single SSD (or any small hard drive or another USB flash drive if you must) and reinstall the OS and then restore the config file you backup periodically.

 

[drwoodcomb]

Please realize that having a redundant/mirroed boot device does not mean that when the primary one fails that the secondary one will take over. You need to use true RAID Mirroring for that to happen. I suspect that da9 is corrupt, probably dying. Swap the USB drives and try to bootstrap your system. If it works then you know one of your devices is bad. The fastest and probably best solution is to replace the boot devices with a single SSD (or any small hard drive or another USB flash drive if you must) and reinstall the OS and then restore the config file you backup periodically.

Thanks for the quick reply (Im panicing inside right now). I am a bit confused by your explanation about the difference between the way I have two redundant usb boot devices and true RAID mirroring. When I originally installed freeNas I had it install to both devices. I ask this because I want to learn from my mistakes and avoid this in the future, what is the point of installing the OS to two usb drives if one doesnt kick in when the other fails? Is there a way to make the OS use the other usb drive in the event that one fails?

Thanks

 

[Ericloewe]

what is the point of installing the OS to two usb drives if one doesnt kick in when the other fails?

It's more complicated than that. Basically, booting from ZFS is far outside what anyone working on the UEFI spec (to say nothing about legacy BIOS) imagined people might want to do. At the end of the day, you need a single device to reliably provide enough data for something ZFS-aware to take over and that is not always the case.

Mirrored boot drives do allow for higher reliability and uptime, but they're not as elegant a solution as we'd like them to be. That's why a single SSD is generally recommended - the base reliability is much higher, so you don't have to worry about these things.

 

[joeschmuck]

(Im panicing inside right now).

Don't panic, your data is very likely all there, just don't do something stupid.

1) Do you have a backup of your configuration file? If not you can recover it but let's assume you did what you should be doing and have a backup.
2) Grab yourself a single USB Flash Drive (a good one) and reinstall FreeNAS on it.
3) After you boot up just cancel the configuration questions and then restore your configuration file and Reboot for good measure.

If you can support a SSD then you are best to use one. USB Flash drives are unreliable in the long run.

If you do not have a copy of your configuration file, don't worry, it should be on your pool. Do a Google search for something like "freenas recover freenas-v1.db" and you should find some instructions.

You want to know what freaked out is? Me having 12 hard drives connected to my ESXi server and trying to migrate all my data to my new pool from the old pool. Nothing has failed yet on my old pool but I want to ensure I don't leave any data behind. I procrastinated 2 weeks before connecting those drives up. Now I have four drives sitting on the floor with a 12" fan blowing air across the entire hard drive section ensuring they stay cool and hoping all continues to work. My PS is only 550 Watts, not really meant for 12 hard drives. I hope to power down and migrate everything tomorrow or Wednesday.

 

[drwoodcomb]

Don't panic, your data is very likely all there, just don't do something stupid.

Stupid is my middle name

You want to know what freaked out is? Me having 12 hard drives connected to my ESXi server and trying to migrate all my data to my new pool from the old pool. Nothing has failed yet on my old pool but I want to ensure I don't leave any data behind. I procrastinated 2 weeks before connecting those drives up. Now I have four drives sitting on the floor with a 12" fan blowing air across the entire hard drive section ensuring they stay cool and hoping all continues to work. My PS is only 550 Watts, not really meant for 12 hard drives. I hope to power down and migrate everything tomorrow or Wednesday.

Oh god that sounds terrifying. Hopefully you dont have any pets or kids playing around the hard drives. Its never fun waiting around with your fingers crossed that something doesn't fail

1) Do you have a backup of your configuration file? If not you can recover it but let's assume you did what you should be doing and have a backup.
2) Grab yourself a single USB Flash Drive (a good one) and reinstall FreeNAS on it.
3) After you boot up just cancel the configuration questions and then restore your configuration file and Reboot for good measure.

If you can support a SSD then you are best to use one. USB Flash drives are unreliable in the long run.

If you do not have a copy of your configuration file, don't worry, it should be on your pool. Do a Google search for something like "freenas recover freenas-v1.db" and you should find some instructions.

I took your advice and reinstalled freeNAS. For now Im using a flash drive but after this ordeal I will be going with an SSD for sure. Originally I dried booting up from the old flash drives by only having one at a time but both gave the same error. It seems odd to me that they would both fail at the same time when I happen to update from 11.0 U4 to 11.1

I was wondering if this could maybe a bug? Either way I reinstalled using version 11.0-u1 since I had that installed on a flash drive from a while ago. I did have a config backup but it was a bit old. Luckily I haven't made many changes since that config back up so I was able to get myself back up and running in short order. Im worried about upgrading from 11.0-u1 to 11.1

If the same thing happens again I guess I can be certain that its a bug. Eitherway I will have to be more consistent with config backups as well as making sure I do a config backup before every update...which I probably should have been doing in the first place. Thanks for all your help

 

[drwoodcomb]

It's more complicated than that. Basically, booting from ZFS is far outside what anyone working on the UEFI spec (to say nothing about legacy BIOS) imagined people might want to do. At the end of the day, you need a single device to reliably provide enough data for something ZFS-aware to take over and that is not always the case.

Mirrored boot drives do allow for higher reliability and uptime, but they're not as elegant a solution as we'd like them to be. That's why a single SSD is generally recommended - the base reliability is much higher, so you don't have to worry about these things.

Hi Ericloewe, thanks for the info. After hearing both you and joeschmuck saying the same thing I will definitely be investing in an SSD

 

[joeschmuck]

Hopefully you dont have any pets or kids playing around the hard drives.

Two dogs and one cat. They are pretty good around my stuff.

I was wondering if this could maybe a bug?

Absolutely it could be a bug but the problem is very few people may see it. So are your USB Flash Drives good? Maybe. You could use them just to see.

Im worried about upgrading from 11.0-u1 to 11.1

If you have conserns then don't upgrade. I would recommend upgrading to 11.0-U4 however if your system is working fine and you are not exposing it to the internet then you don't need to make any upgrades. I don't plan to upgrade to 11.1 for at least 1 month. I've been on the cutting edge dmost of my life and now it's time to take a step back and let the youngsters be adventurious and put thier data at risk. I can wait. When I want to play with the latest stuff, I create a virtual machine and play with that, it's much safer.

I did have a config backup but it was a bit old. Luckily I haven't made many changes since that config back up so I was able to get myself back up and running in short order.

Same here, my configuration just rarely changes. I only make a backup of the configuration file once every 3 to 6 months just for good measure, but it's only when I think about it or unless I make a change and I know I need to backup the config file.

Stupid is my middle name

Mine use to be but now it's whatever my wife feels like calling me. Sometimes it's good, sometimes not.