jlw52761
Explorer
- Joined
- Jan 6, 2020
- Messages
- 87
In doing the upgrade from 11.3-RELEASE to 11.3-U1, after about 30 minutes I fired up the monitor connected to the box and saw a kernel panic on the console. I figured ok, so problem, I can just reboot, and using the console I used the reboot command. After the crazy POST process, I was hopeful when I saw the bootloader startup, then I started to see this stream of "CCB request completed with an error" messages when trying to mount the root for 11.3-U1.
Ok, pucker factor now goes up, I did have a backup of the config right before the upgrade, but urgghhhh, what would not work from that type of recovery.?.?
So I rebooted the server again, holding the power button and waiting for all the fans (all 20 of them) to spin down and go silent, like the end of flight ops. Counted to 10 (old silly habits, capacitors discharge much faster nowadays) and powered back on. After waiting for POST, same stream of messages again.
Second power cycle and decided to try and boot the previous boot environment thinking maybe this was just a bad upgrade. Can you guess it? Yep, another stream of CCB request messages. So I decided to look up what this error actually was, and my research basically said "Your partition is borked and unrecoverable, have a nice day".
Ok, so I guess a reinstall is in order, but not wanting to give up I decided to see if maybe, on this older piece of hardware I had fried the USB port, since I'm booting off a USB flash drive. I switched the port around with my USB cable from my UPS and tried the boot again, only to be greeted with the CCB request failed that I have come to loathe.
So, I resigned myself to needing to reinstall, so I put my boot drive back in the original port and did not plug the UPS back in. I began downloading the ISO to burn to a second USB stick to start the install process.
While waiting for the download, I decided to try the boot again, just to make sure it wasn't me loosing my mind. Can you guess what I got from this effort? Well, if you guessed I got those CCB request failed messages you would be wrong. I got a fully booting system in 11.3-U1! Well I'll be, that was great news!
At this point, I don't know how the UPS could've caused the failure of the upgrade, but apparently it did. After the successful boot, I plugged the UPS back in and decided to tempt fate, because you know at 12:30AM when you have to be up for work in 5 hours is always a great time to do these things, I rebooted. Thankfully, the system booted normally and all is well.
So, what I have learned, and am passing on, is that for some reason a USB connected UPS can interfere with a boot/upgrade when using a USB flash drive as your boot device, and that you should really use a SATA drive if possible, or two USB flash drives mirrored for your boot.
Personally, I am on the lookout for a module similar to how Dell and HP do ESXi boot on their servers, a module that connects to USB and presents as a flash drive, but is a dual SD card device with RAID 1 between the SD cards. I don't have any free SATA ports in my server (14 will eat that up) and no space for any additional drives, but I do have a free full height PCIx8 slot, so maybe something there? A small m.2 drive or something possible, not sure.
Ok, pucker factor now goes up, I did have a backup of the config right before the upgrade, but urgghhhh, what would not work from that type of recovery.?.?
So I rebooted the server again, holding the power button and waiting for all the fans (all 20 of them) to spin down and go silent, like the end of flight ops. Counted to 10 (old silly habits, capacitors discharge much faster nowadays) and powered back on. After waiting for POST, same stream of messages again.
Second power cycle and decided to try and boot the previous boot environment thinking maybe this was just a bad upgrade. Can you guess it? Yep, another stream of CCB request messages. So I decided to look up what this error actually was, and my research basically said "Your partition is borked and unrecoverable, have a nice day".
Ok, so I guess a reinstall is in order, but not wanting to give up I decided to see if maybe, on this older piece of hardware I had fried the USB port, since I'm booting off a USB flash drive. I switched the port around with my USB cable from my UPS and tried the boot again, only to be greeted with the CCB request failed that I have come to loathe.
So, I resigned myself to needing to reinstall, so I put my boot drive back in the original port and did not plug the UPS back in. I began downloading the ISO to burn to a second USB stick to start the install process.
While waiting for the download, I decided to try the boot again, just to make sure it wasn't me loosing my mind. Can you guess what I got from this effort? Well, if you guessed I got those CCB request failed messages you would be wrong. I got a fully booting system in 11.3-U1! Well I'll be, that was great news!
At this point, I don't know how the UPS could've caused the failure of the upgrade, but apparently it did. After the successful boot, I plugged the UPS back in and decided to tempt fate, because you know at 12:30AM when you have to be up for work in 5 hours is always a great time to do these things, I rebooted. Thankfully, the system booted normally and all is well.
So, what I have learned, and am passing on, is that for some reason a USB connected UPS can interfere with a boot/upgrade when using a USB flash drive as your boot device, and that you should really use a SATA drive if possible, or two USB flash drives mirrored for your boot.
Personally, I am on the lookout for a module similar to how Dell and HP do ESXi boot on their servers, a module that connects to USB and presents as a flash drive, but is a dual SD card device with RAID 1 between the SD cards. I don't have any free SATA ports in my server (14 will eat that up) and no space for any additional drives, but I do have a free full height PCIx8 slot, so maybe something there? A small m.2 drive or something possible, not sure.
