11.2 upgrade problems

[crusader27529]

First of all, here's my HW setup.....

I'm running on a SuperMicro X7-DBU in a 2U SuperMicro SC-825 chassis. The MB has dual 8-core processors, and 32GB of ECC memory.
I have an LSI 9211-8i HBA set to IT mode, am booting from a 16GB PNY USB memory stick, and
have 6 drives connected to the HBA, along with 2 drives attached to the internal ports
on the MB. Those 2 drives will eventually be mirrored boot drives.

So, here's the history....I originally had been running on 9.3 on an old Dell box, and after it died, I moved the 3 drives I had at that time
to an external enclosure with a USB interface, attached to a laptop with 8GB of RAM. I know that it's far
from an optimal system, but it's what I had. The performance sucked, but it functioned
as a NAS just fine for my personal use. Along the way, I updated all the drives to
give me a RaidZ1 with 3 2TB drives. It all worked fine, except for the performance.

I was able to acquire the SuperMicro server at a good price, and decided to move to the faster box, and then update to something more current. The
transition to the new server went smoothly, as all I had to do was boot from the old USB stick.
I've never been able to update with the 9.3 server for some reason, so I decided to
re-install using boot media to 11.2u6 or u7(I don't remember), but that was a complkete disaster.
The install process NEVER comp[leteed, and continually would reboot the server while
updating at various point, never with an obvious pattern. I tried everything that I could think of, but no success.
At this time, I was able to find a copy of the install media for 9.10, and that worked fine. I added some drive and
was functional. I didn't change anything, but just added some vdevs and played with snapshots, plus
mounter the system drive to a spinning disk so as not to write too much to a USB stick.

So, after a while, I kept getting notifications that there were updates, so I dumped a backup to an external drive, and tried an update. It originally showed the 9.10
train, so I tried that, and I've no idea what actually was accomplished. I eventually
selected the 11.2 stable train, and did the update. The result was again notb what I wanted.
The system booted, but threw MANY errors, which I had no way of recording, and
eventually hung during the boot. I'm not sure how long I waited, but it was at least 10 minutes
for the process to continue, but it didn't. I rebooted, and selected the 9.10 version, and it
rebooted, threw the same errors(more or less), but only hung for about 45 seconds, then it continued, successfully,
and the sysrem was operational. I think I tried to reboot 11.2 at least another time before giving up and
going back to 9.10.
I'm not very good with BSD or Linux, so I waited until the daily freenas security
output was emailed to me, and it has the majority of the errors included. I
can append the 'real' error list if I knew how, but for now here's the contect of the email. I sure woulkd like some
direction........


I think this is from the last 9.10 boot, as the 11.2 boot seemed to say something about the retry count was exhausted, and then hung.

Because of the limit in size of this email, I'll try to split it into pieces.....it's easier to attach the text file.

Thanks in advance.

 

[crusader27529]

Surely someone can tell me what those error messages mean!?!?!?

HELP!

 

[pschatz100]

So, you are running successfully on 9.10? Are you still booting from the same flash drive as you were when on 9.3? I have seen errors like that when the boot device is failing. Using flash drives as boot devices is notoriously unreliable - especially for newer versions of FreeNAS.

I would pick up a small SSD and set it up as your boot device. As you have a rather simple FreeNAS setup at this point, I would just do a fresh install of 11.2. Since you have an older system, try setting up the boot image for BIOS boot (some systems call this LEGACY boot.) On my Supermicro X9SCM system, BIOS boot was more reliable than UEFI when I moved to SSD.

After you have the system booting and running reliably, you can import your data volumes.

 

[crusader27529]

Well, I don't have any SSD devices, so I selected 2 160GB spinning drives for boot that are connected to the USB ports on the MB. There's no USB stick even near the system, and I'm booting from a CD with the 11.2u7 install image on it.

The results are EXACTLY the same as when trying to install to a USB stick, giving the same error:

gmirror: invalid class name

It does that whether or not I try to mirror the drives or which of the 2 that I choose for a boot drive. The system hangs for several minutes with some fast, continuous activity on the CD, then completely reboots to try it all again.

One more time.......does anyone know what the errors in the file I uploaded signify, and can anyone tell me where the gmirror errors are generated?

It looks like I'll never be able to go beyond 9.10, which pisses me off, because I specifically bought a supported, plain vanalla SuperMicro server just to mitigate any issues.

 

[pschatz100]

Well... keep your cool. Losing your temper will not encourage people to help you. We're all volunteers trying to support the community.

I am not an expert at diagnosing these errors, but they seem to relate to SCSI accesses - which are similar to the protocols that USB interfaces use to communicate with SATA drives. So, if I had to guess, I would say the problem is related to the USB interfaces or your drives. If you want to use the spinning drives you have for boot devices, I would try connecting them to the onboard SATA and see if that works any better. You might also try a different boot device - 160Gb spinning disks are pretty old.

 

[crusader27529]

I'm sorry if I came across as losing my temper....it's quite the opposite, as I was sort of panicing.

There are NO USB drives in the system, and the errors are being reported on the relatively current drives connected to the LSI HBA. The boot drives are NOT connected to the LSI, but are connected to the MB ports.

The errors reported appear to be from the drives on the LSI, but the drive is functional, with no errors reported. I don't have SMART enabled, so maybe I'll try that to see what it reports, specifically on that 1st drive.

I had the exact same issues when I was booting from a USB stick.

 

[pschatz100]

I'm sorry if I came across as losing my temper....it's quite the opposite, as I was sort of panicing.

There are NO USB drives in the system, and the errors are being reported on the relatively current drives connected to the LSI HBA. The boot drives are NOT connected to the LSI, but are connected to the MB ports.

Well, in a previous post, you said "I selected 2 160GB spinning drives for boot that are connected to the USB ports on the MB. There's no USB stick even near the system, and I'm booting from a CD with the 11.2u7 install image on it."

The errors reported appear to be from the drives on the LSI, but the drive is functional, with no errors reported. I don't have SMART enabled, so maybe I'll try that to see what it reports, specifically on that 1st drive.

I had the exact same issues when I was booting from a USB stick.

Maybe you have an issue with the LSI card. I know there are some dependencies between FreeNAS versions and LSI card firmware versions. Some searching on the forum would turn up a few discussions about this.

Perhaps you should remove the LSI card and disconnect your data drives, then try configuring the system for 11.2u7 again. If the base hardware will boot OK, then you know the problem is related to the LSI card and/or the drives. If you still cannot get it to boot, then the problem is elsewhere.

 

[crusader27529]

When I talked about the 160GB drives, I said USB ports, but should have said SATA ports on ther MB......sorry.

The LSI HBA IT firmware is the 20......007 version(I don't have the exact version handy), but was a fix to the 20....004 version, so that shouldn't be a problem. Also, there were absolutely NO errors when I installed 9.10, even after the inability to install 11.2u7. The errors (besides the mysterious gmirror error during the 11.2u7 install failures) started after I did an update to 11.2u7 from the 9.10 GUI. That update completed, but at boot, it got the errors and just hung. Re-booting back to 9.10 got the same errors, but it didn't hang. So, it's sort of obvious that the errors were introduced during the 11.2u7 procedure.

I thought I read that something changed with 11.X update in that it caused problems if you booted back to an older version, but I'm not sure about that. I wonder if the file system is different enough to cause backlevel compatibility issues.

I setup SMART finally, and I should get results if there are any problems on the drives. The one 160GB drive is for the system data, and I'm not sure how to replace it, and any potential problems from replacing it. I originall added it just so I could use it to play with various volume settings and features. The raidz1 consists of 3 2TB drives, plus a forth that I plan to use to make it a raidz2. There's a single 4TB drive that I'll copy the contents of the current raidz1 onto before I delete the volume abd recreate it in its new configuration, but I'm waiting until I have more confidence in the 9.10 setup.

 

[pschatz100]

The version of FreeBSD that is used is different between FreeNAS 9.10 and 11. And the way the boot device is configured was updated going from 11.1 to 11.2. so, at the end of the day, there are a number of differences going from 9.10 to 11.2. So far as I know, there are no specific issues booting back to an older version - except that you might have ZFS problems if you have updated your ZFS volumes under the latest versions of FreeNAS and then decide to boot back into an older version. If you have not already done so, do not update the ZFS volumes. Older ZFS volumes will mount OK in newer FreeNAS.

If I were in your situation, I would do the following:

Remove the LSI card and disconnect all drives except the boot device. If the system boots into 9.10 without incident, then you will know that the problem lies somewhere with the LSI card or the drives. Assuming the system boots OK, then you have some choices:

1) If you have a complex FreeNAS installation, and want to update from the 9.10 version, I would save your 9.10 configuration (the system provides an easy way to do this) then update to 11.0. If that works, then update to 11.1. If that works OK, then update to 11.2. If that works, then you can start troubleshooting the LSI card and the drives. Let each update run to completion - it might take a while as there is a lot going on with these updates.

2) We know that booting from flash drive is notoriously unreliable, and this would be a good time to set up a more stable boot device. Get a small SSD and do a fresh install of 11.0. Once the new installation has completed, you can restore your saved configuration. If all is still going well, then you can update to 11.1 followed by an update to 11.2. Once everything is running OK, add back your LSI card and the drives then troubleshoot from there.

3) If your FreeNAS installation is relatively simple, it might be less work to just do a fresh install of 11.2 and reconfigure the system settings. As before, this would be a good time to set up a new boot device. Once everything is running OK, add back your LSI card and the drives then troubleshoot from there.

At the end of the day, the point of this exercise would be to pinpoint where the errors are coming from. Regardless of which approach you take to update the system, it would be good to know if the problems are being caused by something related to the motherboard or memory, or being caused by the LSI card and the drives.

 

[crusader27529]

Reference this post of mine:

No volumes in GUI, system completely functional

I was informed by my power provider of a planned shutdown, so I dutifully took down my FreeNAS system before the outage. Strangely, after the outage, the system if functional, but the GUI doesn't show any volumes, so it can't show the content of the system. The reporting tab shows the disk...

www.ixsystems.com

Essentially, after a planned power shutdown, the GUI doesn't show data related to volumes or drives, but does show activity and correct partition data. So, I decided to reinstall to 9.10, but that didn't fix the issue, but created a new one.

Now, I'm unable to map CIFS drives to the NAS no matter what I do. So, the data is there, but I can't get to it. I've tried everythiung I could, even making root the owner of all the data, and using root credentials, but I still get denied access.

So, I thought I'd try to upgrade to 11.3, since 11.2.7 failed. Well, nothing has changed, and I still get gmirror errors during install, and then a reboot.

I'm booting froma DVD, and have tried just about all the combinations that I could conceive. I tried no drives connected to the LSI card(IT mode, latest, correct firmware), with no change. Tried a single boot drive with the other drive(both connected to onboard SAS/SATA connectors on the MB), and even with the LSI card removed, and they all result with the same error(s). I even depopulated the memory, reducing it from 32GB to 16GB as a last ditch effort, with no positive results.

So, I reinstalled to 9.10, and the GUI still doesn't show volumes or drives. The GUI issue is likely related to something in the config.db, but I've no idea what or how to edit the config. What I need is a way to remove everything from the config except the minimal data to define the volumes.

I think that the inablilty to map drives is related to the config, because the sharing partmof the GUI doesn't show the shares that I create. I see them on the left side menu, but when I try to show the shares, the GUI shows nothing.

I have 2 copies of my data, one on a raidz2 and the other on a stripe drive. Both of these copies were made just before the planned shutdown. Maybe loading FreeBSD would allow me to map to one of the data copies and allow me to retrieve the data.......

The 11.X install stuff is unfortunate, but I could live with 9.10 if I had to........HELP(Again)!!!!

 

[pschatz100]

Are you really booting from a DVD? I always thought the boot device needs write capability. What happens if you try to boot from a small SSD or flash drive?

 

[crusader27529]

Are you really booting from a DVD? I always thought the boot device needs write capability. What happens if you try to boot from a small SSD or flash drive?

Yes, and no......I did boot from a DVD, but have tried multiple tries with 11.2.7 from a USB stick with the same results.

I started upgrading in sequence from 11.0.0, and I was last successful with 11.1.7, but can't get anything else newer to work. If I recall, one of the major differences is the boot loader isn't GRUB anymore, but the native FreeBSD boot loader. I don't know if that's part of the issue.

In updating, trying to determine the saved boot environments, can you tell me how to interpret the numbers that the loader uses?

 

[pschatz100]

When I look at boot environments on my system using System>Boot Environments, they have names that match up with FreeNAS versions.

Are you now successfully running on 11.1.7?

Are you using UEFI or legacy BIOS boot? When I updated the boot device on my X9-SCM system from flash drives to SSD, I had to install FreeNAS for a BIOS device and then change the Boot device choice in the motherboard bios from UEFI to a BIOS device. I have not done a fresh install since 11.2 so I don't know whether or not you still have to choose the boot device type when installing FreeNAS, but if you do - make certain the device type matches up with the boot device type in bios.

Since you are using such old hardware, how do you know that something isn't flakey? Power supplies and memory sticks can go bad and give intermittent problems.

As I suggested previously, I would disconnect everything except the boot device and a minimum amount of memory. Be patient and let the upgrade run to completion. If the upgrade doesn't work, then there must be an issue with the hardware. If the upgrade works, add back one thing at a time - starting with the memory. And don't overlook the possibility that a power supply can be bad - they can be very difficult to diagnose.

 

[crusader27529]

When I look at boot environments on my system using System>Boot Environments, they have names that match up with FreeNAS versions.

Are you now successfully running on 11.1.7?




Since you are using such old hardware, how do you know that something isn't flakey? Power supplies and memory sticks can go bad and give intermittent problems.

As I suggested previously, I would disconnect everything except the boot device and a minimum amount of memory. Be patient and let the upgrade run to completion. If the upgrade doesn't work, then there must be an issue with the hardware. If the upgrade works, add back one thing at a time - starting with the memory. And don't overlook the possibility that a power supply can be bad - they can be very difficult to diagnose.

When I look at boot environments on my system using System>Boot Environments, they have names that match up with FreeNAS versions.

Are you now successfully running on 11.1.7?

Are you using UEFI or legacy BIOS boot? When I updated the boot device on my X9-SCM system from flash drives to SSD, I had to install FreeNAS for a BIOS device and then change the Boot device choice in the motherboard bios from UEFI to a BIOS device. I have not done a fresh install since 11.2 so I don't know whether or not you still have to choose the boot device type when installing FreeNAS, but if you do - make certain the device type matches up with the boot device type in bios.

Since you are using such old hardware, how do you know that something isn't flakey? Power supplies and memory sticks can go bad and give intermittent problems.

As I suggested previously, I would disconnect everything except the boot device and a minimum amount of memory. Be patient and let the upgrade run to completion. If the upgrade doesn't work, then there must be an issue with the hardware. If the upgrade works, add back one thing at a time - starting with the memory. And don't overlook the possibility that a power supply can be bad - they can be very difficult to diagnose.

I'm successfully running 11.2.1, but the install ended strangely........instead of coming to the point where it showed that the install was succeesful and asking to reboot the machine and remove the install device, it just rebooted.......while it was starting the reboot, I quickly removed the memory stick, and the system booted fine.

I originally tried both UEFI and BIOS boot with no difference in rersults.

The hardware has been running 9.3 and then 9.10 for over 3 months before I tried to update to 11.2.7, so I'm sure the hardware is fine, and it's ECC memory.

I tried running with minimal memory and with minimum hardware with no real change is result. The only thing that worked was to start with 11.1.0 sequentially through 11.2.1. I plan on continuing to see if I can get to 11.2.7, but I'm not hopeful. I had also switched boot drives somewhere along the way, and the results implied that there were artifacts from previous attampts left on the system somewhere.

I've troubleshot many and varied complex electronic and software systems for over 40 years both in development and support, and my troubleshooting unfortunately didn't show any consistent direction as to where the issues were. Currently, I'm happy with where I am, but am concerned about the difficulty in getting here.

The inconsistency isn't good.........

 

[crusader27529]

I'm successfully running 11.2.1, but the install ended strangely........instead of coming to the point where it showed that the install was succeesful and asking to reboot the machine and remove the install device, it just rebooted.......while it was starting the reboot, I quickly removed the memory stick, and the system booted fine.

I originally tried both UEFI and BIOS boot with no difference in rersults.

The hardware has been running 9.3 and then 9.10 for over 3 months before I tried to update to 11.2.7, so I'm sure the hardware is fine, and it's ECC memory.

I tried running with minimal memory and with minimum hardware with no real change is result. The only thing that worked was to start with 11.1.0 sequentially through 11.2.1. I plan on continuing to see if I can get to 11.2.7, but I'm not hopeful. I had also switched boot drives somewhere along the way, and the results implied that there were artifacts from previous attampts left on the system somewhere.

I've troubleshot many and varied complex electronic and software systems for over 40 years both in development and support, and my troubleshooting unfortunately didn't show any consistent direction as to where the issues were. Currently, I'm happy with where I am, but am concerned about the difficulty in getting here.

The inconsistency isn't good.........

I've successfully updated to 11.2.8, and all installed OK for unknown reasons. I have NOT upgraded the ZFS to latest, just in case that I need to revert to something earlier. The upgrade continued behavior when the install was completed and booting by itself when it was completed.

I'll run for a few weeks before I upgrade the ZFS to a point that I can't revert.

BTW, I really dislike the new web interface in general, and I've had issues that the interface froze. I can tell that the interface has multiple bug resolved, but those were just annoying. Maybe I'll get used to/like the new interface, but in general, it's just a pretty face.

So, I'll not update to 11.3 for some time, as all I wanted 11.2 for was the new interface.

 

[SweetAndLow]

I've successfully updated to 11.2.8, and all installed OK for unknown reasons. I have NOT upgraded the ZFS to latest, just in case that I need to revert to something earlier. The upgrade continued behavior when the install was completed and booting by itself when it was completed.

I'll run for a few weeks before I upgrade the ZFS to a point that I can't revert.

BTW, I really dislike the new web interface in general, and I've had issues that the interface froze. I can tell that the interface has multiple bug resolved, but those were just annoying. Maybe I'll get used to/like the new interface, but in general, it's just a pretty face.

So, I'll not update to 11.3 for some time, as all I wanted 11.2 for was the new interface.

well 11.3 has the improved new interface that is much better than the interface in 11.2. Also 11.3-U1 is out and pretty good so far. I have not seen very many major issues.

 

[crusader27529]

I tried to update to 11.3.1 from 11.2.8 and it rebooted in the middle of the install, like it did earlier in my adventure.

I didn't try the normal sequence(for me) to try 11.3.0 first, then 11.3.1, but that shouldn't make a difference, but apparently does.

So, for now, I'll stay at 11.2.8.

Regarding to the multiple boot environment, I'm still confused as to how to determine which is which.....here's what's displayed:

default
default-20200226-171142
default-20200226-173811
Initial-Install
Wizard-2020-02-26_12-32-25
default-20200227-221745

Obviously there are entries on each line with a date/time, plus which is active, and keep status.

I have no idea how this is supposed to correlate to a version.......

 

[pschatz100]

I just finished an upgrade from 11.2-U7 to 11.3-U1. No problems, but you have to be patient and let it run to completion. Sometimes upgrades reboot during the process. Just let it finish.

After the upgrade was complete, a few pages in the new GUI did not display correctly. Reloading them in the browser fixed the problem. I would guess that the browser was confusing cached versions of those pages.

I also upgraded an iocage jail from 11.2 to 11.3. This also ran to completion, but it was slow. It actually took more time than the FreeNAS upgrade. Note... this is something you do from the Shell.

Regarding to the multiple boot environment, I'm still confused as to how to determine which is which.....here's what's displayed:

default
default-20200226-171142
default-20200226-173811
Initial-Install
Wizard-2020-02-26_12-32-25
default-20200227-221745

Obviously there are entries on each line with a date/time, plus which is active, and keep status.

I have no idea how this is supposed to correlate to a version.......

I have no idea where you are getting this information. When I look at System > Boot in the GUI, it gives me a list of FreeNAS versions, along with the date of installation and some other information.

 

[crusader27529]

I just finished an upgrade from 11.2-U7 to 11.3-U1. No problems, but you have to be patient and let it run to completion. Sometimes upgrades reboot during the process. Just let it finish.

After the upgrade was complete, a few pages in the new GUI did not display correctly. Reloading them in the browser fixed the problem. I would guess that the browser was confusing cached versions of those pages.

I also upgraded an iocage jail from 11.2 to 11.3. This also ran to completion, but it was slow. It actually took more time than the FreeNAS upgrade. Note... this is something you do from the Shell.


I have no idea where you are getting this information. When I look at System > Boot in the GUI, it gives me a list of FreeNAS versions, along with the date of installation and some other information.

I'll try the upgrade one more time, but not for a week or so, but I'm sure that I let it re-boot previously, and it didn't boot, but it won't hurt to give it another try.

I got what I showed for boot environments also from 'System > Boot in the GUI', and that's why I asked the question. It doesn't return anything that has anything associated with FreeNAS versioning.

I've no idea what that means.........