upgrade from 11.2 u-6 to u7 - lost main pool

[fishbulbhead]

Hello All - I recently performed a standard upgrade from u6 to u7 through the UI and my main pool is not recognized. The dashboard shows my pool state as "UNKNOWN". I'm running 3 jails: plex, deluge and a generic image as a backup target. The pool is a 5 disk raid5 set that was healthy before the upgrade. I performed a FN backup as the UI walks you through.

Has anyone else had a pool drop out after this upgrade? Are there ways to recover the pool configuration (either through the FN backup or other means)? I'm searching the forums and googling to hone in on the problem/troubleshoot - but wanted to see if there was a easy/expected way to recover this pool. Any help would be greatly appreciated - thanks!

 

[Jessep]

Post console output for "zpool status" in code tags.

 

[dlavigne]

Also, when you say raid5 do you mean hardware RAID?

 

[fishbulbhead]

Thanks for the quick replies. Jessep, the pool that is missing is called "largepool" and does not show up in this list. "jailpool" is as the name suggests, a small pool to hold jail images on 128GB m.2 stick. dlavigne, this is software raid5.

The following show in dmesg:
nvd0 - NVM stick ~500GB
ada0 - ST3000DM008 3TB
ada1 - ST3000DM007 3TB
ada0 - SU800N generic m.2 stick
ada0 - ST3000DM008 3TB

crap - 2 drives are missing. I'll check cabling, etc - this is in my basement on UPS, well cooled, never touched...not sure why 2 drives would fall out. I run SMART and scrub 2x a week with no errors (drives are ~8mo old purchased new). This is an AMD ryzen3 chip, AsRock AB350 mobo with 16GB.

root@freenas[~]# zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:00:22 with 0 errors on Wed Nov 20 03:45:22 2019
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
ada2p2 ONLINE 0 0 0

errors: No known data errors

pool: jailpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
jailpool ONLINE 0 0 0
gptid/13a20c28-45a3-11e9-acdf-7085c26e4a5f ONLINE 0 0 0

errors: No known data errors
root@freenas[~]#

 

[fishbulbhead]

just re-seated all SATA cables and power to each of the 5 drives. Everything was snug, no obvious looseness. Cold boot and still unable to find 2 of the 5 drives. Could a driver have been removed from the previous kernel or some limit to the number of default devices? All 5 drives are from the onboard SATA controllers. The m.2 drive is a slot on the mobo (holds the jails) and the NVM drive is boot/root on its own SATA controller/PCI card. Any suggestions on additional debug would be greatly appreciated. I'll dig through the release notes to look for changes that could pertain to this situation. Thanks in advance for any help (sorry about not using code tags, I missed that and cannot edit that post).

 

[Jessep]

Your motherboard has (2) SATA controllers

• 4 x SATA3 6.0 Gb/s Connectors, support RAID (RAID 0, RAID 1 and RAID 10), NCQ, AHCI and Hot Plug*
• - 2 x SATA3 6.0 Gb/s Connectors by ASMedia ASM1061, support NCQ, AHCI and Hot Plug

Are the two missing drives on the ASMedia controller?

 

[fishbulbhead]

The connector layout is as follows: the 2 x SATA3 controller has two connectors that are perpendicular to the motherboard and 4 x SATA3 connectors that are parallel/on the same plane as the motherboard. (you can see them on the pictures here: https://www.asrock.com/mb/AMD/AB350 Pro4/#Specification ) I removed both data cables from the 2 x SATA controller to troubleshoot and restarted FN. Now two additional SATA3 drives have dropped out (only NVMe on its own controller, one SATA and the m.2 drive show up in dmesg/kernel). I have not changed anything on this box prior to the upgrade (it's headless, no gfx card, keyboard/mouse).

Could something in the FN release handle the 4 x SATA3 controller differently? Of course that doesn't make too much sense given that I can see at least one SATA3 drive on that controller.

 

[fishbulbhead]

ok - I've narrowed it down to the first two positions of the 4 x SATA3 controller (that would be the two plugs out of four that are closest to the 2 x SATA3 controller). Could this be a PCIe or device enumeration problem? I have read through the release notes and I don't see any mention of kernel/driver level changes. Is it possible to go back to the U6 (working) version? I don't see a way to downgrade in the UI (unless possible with an "install manual update file").

 

[Mr. Slumber]

Is it possible to go back to the U6 (working) version?

Yes, please take a look @ "System/Boot Environments". You can choose a former version there, reboot, done. :)

 

[fishbulbhead]

I guess there's a first time for everything, I replaced the SATA data cables running to one of the two drives and it's back up and running! Thanks to all of you for your help! Hardware has become so durable over the last 20+ years that I rarely look at that when other variables have changed (i.e. upgrade).

I'm sorry for wasting everyone's time when the root cause was the lack of troubleshooting on my end. Lesson learned.