Upgraded 9.2.1.9 to 9.3.10, this is how my zpool looks like now

[Satam]

Hi,

so I installed 9.3.10 on a set of two USB sticks, and installation worked fine. I made backups of the configuration, the keys and exported the zpool on the 9.2.1.9 system. And I still kept the 9.2.1.9 USB stick.

Then I booted up the 9.3.10 system and it after uploading the database it rebooted twice and then I had no trouble logging into the web interface.

But then I wanted to import the zpool via the web interace, selected my 12 drives (2x raidz2) and decrypted them successfully, but then in the dropdown list to choose a zpool nothing came up. So I decided to use the shell and just typed "zpool import tank0" which seemd to have succeeded at first glance:



But I noticed the mount point wasn't set and I looked at the volume in the web GUI and this doesn't seem right at all:



First of all my zpool should have a configuration "tank0->storage0" and not "tank0->tank0->storage0". Where did that intermediate tank0 com from? "tank0" is the zpool and there wasn't any dataset called "tank0" in that zpool.

Secondly the Dataset storage was 18TB in size not 12TB. There's a whole raidz2-vdev missing, maybe? Although all vdevs and drives show up as being ONLINE in the "zpool status tank0" output. And 18TB can't be right either, the full zpool was at most ~14TB (4x 3TB + 4x 500GB).

It may have something to do with the firmware mismatch of my IBM ServeRAID M1015 being v16 and not v20, but the disks decrypted successfully and reading them with "dd" also succeeded. So I doubt it's an issue, although I cannot fully cancel that out.

But the problem is that before I upgrade the firmware I want to be sure it's not some other issue, because I want to be able to switch back to my 9.2.1.9 system to check the zpool.

Oh, and there's nothing under /mnt/ either. The zpool and the datasets didn't mount.

In fact that's what I'm going to do right now, to see if the zpool can be importet in 9.2.1.9 still without that error.

If anyone has any idea what might have went wrong, in the meantime please let me know.

 

[danb35]

"tank0" is the zpool and there wasn't any dataset called "tank0" in that zpool.

There was always a dataset called tank0 (a pool always contains a dataset of the same name), but it was hidden prior to 9.3. And no, there's no vdev missing--if it were, the pool couldn't be online.

 

[BigDave]

From version 9.2.1.9 go in steps...
First stop 9.3 (at this point you must bring your HBA firmware up to version P20, to match the new FreeNAS driver in 9.3.1)
Then the next is 9.3.1
Then the last is 9.10 (yes, the newest version number makes for some confusion, don't shoot the messenger)
There are already many posts on this subject here in the forum.

edit: corrected version number for P20 FW/driver change. Thanks Dan!

 

[danb35]

First stop 9.3 (at this point you must bring your HBA firmware up to version P20, to match the new FreeNAS driver in 9.3)
Then the next is 9.3.1

P20 wasn't used until 9.3.1.

 

[BigDave]

Before or after...
You can certainly do it either way, right?

Corrected previous post.

 

[Satam]

So I booted up the 9.2.1.9-stick and all works fine. Damn that was close! I almost lost my whole pool. This is how it should look like?



So the 12.9TiB was correct after all. I guess I mixed up the SI TB of the drive manufacturers with the ?th power of 1024 the TiB is referring to. So where the extra 6TiB came from the the odd 313GiB of free space just puzzles me. The missing mount point might have been due to unsuccessful importing via the shell command "zpool import tank0" I issued.

So this is actually my second attempt at upgrading to >9.3.x and it failed even worse than the first one. At my first try auto-importing via web gui at least worked and I was able to access the data in the dataset. But I reverted to 9.2.1.9 even then, because I was put off by the strange intermediary "tank0" dataset in the View Volume view of the web gui.

I'm confused. I wish the documentation would prepare me better for a 9.2.x to >9.3.x upgrade, there seems to be major changes which are just confusing.

 

[Satam]

From version 9.2.1.9 go in steps...
First stop 9.3 (at this point you must bring your HBA firmware up to version P20, to match the new FreeNAS driver in 9.3.1)
Then the next is 9.3.1
Then the last is 9.10 (yes, the newest version number makes for some confusion, don't shoot the messenger)
There are already many posts on this subject here in the forum.

edit: corrected version number for P20 FW/driver change. Thanks Dan!

Ah, makes sense. Because as I just wrote, I did an upgrade to 9.3 before and while confusing, I was at least able to access my pool data.

It had actually crossed my mind after I saw the pool not being imported in 9.10 that I maybe do an upgrade to 9.3 first before upgrading to 9.10. Well, I didn't expect going straight to 9.10 would break things in such a bad way.

Now, I still wonder... the difference between the 9.2 and the >9.3 zpool/volume structure... I mean that extra dataset in between with the name of the zpool... is that how it's supposed to be after 9.3 or does that only happen when importing a 9.2 zpool (and not happen when creating a new zpool in >9.3) and I'll have to live with it when upgrading?

Also I guess it makes sense I won't be finding any caveats concerning an upgrade from 9.2.x in the 9.10 documentation. I'll have to use the 9.3 documentation first. Still a warning regarding an upgrade from a pre-9.3 setup would have been nice in the documentation for even the latest FreeNAS version.

I'm lucky I didn't wreck my zpool with this.

 

[BigDave]

The extra dataset had actually been there, it was just hidden and was not revealed until the 9.3 update.
We have all managed to ignore/except it for what it is, get past that and you will be fine.

 

[Satam]

Ah, thanks. I almost thought as much. Now I'm relieved and more confident to try again. But tomorrow... so dunno if PROBLEM is SOLVED yet. :D

 

[BigDave]

Once you get to 9.3 (your pool should be mounted without issue during boot),
Then we can talk about the HBA firmware update that's needed to match the
new driver found in the next step at 9.3.1

Rest up, lol

word to the wise, avoid using the CLI for pool importing, without first asking for help, it sometimes goes badly...

 

[Satam]

I find the hardest part of all of this is not being able to remember what I did to the system. It had an uptime of 335 days or so. That's a long enough time for me to forget all the things I did... reflashing the HBA, setting up ssh and creating certificates, even managing the zpool. I have to document my steps myself I guess. But then I want to just do the job quick and get it over with...

This I absolutely knew would happen, me forgetting what to do. About the rest I didn't know what would happen. And of course, something weird did happen.

Regarding flashing the HBA, I think I still have a link to an excellent guide somewhere. We'll see. Thanks and good night!

 

[Satam]

Oh, I'm losing sleep over this... because I just realized one mistake I did. I saved my conf db before I exported the zpool.

So when I uploaded the conf db to the new 9.10 system, I also restored the config data of the zpool tank0 into it. The actual zpool on the disks however was in an exported state. So at first it wasn't showing up, and I didn't realize the system just didn't find the disk sets, but actually was expecting the tank0 disks to be ready.

System: tank0 was not exported, knowledge of its existence was not purged from it. Disks: tank0 was exported.
So basically re-importing the disks into a system in which they were never exported from, since I had restored the configuration with zpool info, may have caused the odd size of the zpool and the failure to mount.

Now the upgrade guide of the FreeNAS documentation doesn't mention you should export a zpool before upgrading. Well, I must admit I was consulting Oracle's ZFS Administraion Guide also, and got stuck on the idea that exporting the zpool was necessary. I believed that in my first attempt to upgrade to 9.3 almost a year ago, the double occurance of "tank0" as both zpool and dataset was a glitch and was caused by not exporting the zpool beforehand.

Something that wasn't a glitch led me to apply a supposed fix, which by itself was causing a real glitch. I guess.

Doh!

I guess I could just do an upgrade without exporting the zpool, save the conf db with the zpool info. Import disks and upload conf in a >9.3 install and all will be well. Or I could even export the zpool and—only after that—save the conf db on the old system. Then restore the db file (lacking any zpool info) on a new >9.3 install and re-import the disks, and all might as well be fine, too. Though the latter procedure is not recommended nor necessary. But most importantly the information regarding the state of the zpool must be consistent both between the installation and the data disks. Which in this 2nd attempt it definitely was not.

 

[BigDave]

You are tired and overthinking this. You are back on 9.2.1.9 right? Is everything working?
If the answer is yes, go get some sleep. Your data will be fine. Sleep... sleep... sleep...

 

[SweetAndLow]

Is your pool really 99% full? If so that is your problem, lol.

 

[Satam]

Hey, there are six 4TB drives sitting on my desk, desperate to be installed into a 9.10 system. They should fix my 99% problem I hope. :D

Anyway my plan is as follows:

• First I'll test my hypothesis, that the export/import mismatch between the disks and the system caused the weird sizes in "View Volume" and the missing mount point, by upgrading the proper way from 9.2.1.9 to a clean 9.10 (no zpool export). Because if that works, then it's most likely this mismatch and not something else that caused the glitch, the HBA firmware for example or some changes between 9.3.x and 9.10.
• But if that fails I'll upgrade to 9.3 first and then to >9.3.1 while updating the LSI 2008 firmware in between to P20.
• Then I'll update my motherboards UEFI and IPMI while at it.
• And at last I'll replace the second raidz2 which currently consists of 500GB disks with 4TB disks

Now the first parts are what are relevant to the glitch this thread is about.

 

[SweetAndLow]

The first thing you need to do is fix the 99% full problem. Your pool is not expected to work when that full.