SOLVED Please Help! Pool Status Offline

[bent98]

I upgraded from TrueNas 13.0-U6 to 6.1 a few days ago. System specs Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz. Running 1SSD drive to boot Truenas and then 7x8TB WD drives in one big pool which as of today, the pool BIGASSRAID is offline.

HPE ProLiant ML10 Gen9 has 5 sata and 3 are attached to an IO Crest SI-PEX40062 4 Port SATA III PCIe 2.0 x2 card.

All the disks show up in mobo bios. Also, when True Nas is booting, it detects all 8 disks.

I also had no issues with data corruption the day before as I have a cron job that runs every day and emails me SMART status.



########## SMART status report summary for all drives ##########

+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+
|Device|Serial |Temp|Power|Start|Spin |ReAlloc|Current|Offline |UDMA |Seek |High |Command|Last|
| | | |On |Stop |Retry|Sectors|Pending|Uncorrec|CRC |Errors|Fly |Timeout|Test|
| | | |Hours|Count|Count| |Sectors|Sectors |Errors| |Writes|Count |Age |
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+
|ada0 |VK1EX8HY | 34 |55360| 51| 0| 0| 0| 0| 0| N/A| N/A| N/A| 0|
|ada1 ?|VK1GHX4Y | 37 |55358| 51| 0| 1| 0| 0| 0| N/A| N/A| N/A| 0|
|ada2 |VJH5RDPX | 35 |55358| 46| 0| 0| 0| 0| 0| N/A| N/A| N/A| 0|
|ada4 |VK1GJA0Y | 32 |55359| 50| 0| 0| 0| 0| 0| N/A| N/A| N/A| 0|
|ada5 |VK1EXGGY | 34 |55359| 51| 0| 0| 0| 0| 0| N/A| N/A| N/A| 0|
|ada6 |VK1EBZTY | 36 |55358| 51| 0| 0| 0| 0| 0| N/A| N/A| N/A| 0|
|ada7 |VK1EH4BY | 36 |55359| 50| 0| 0| 0| 0| 0| N/A| N/A| N/A| 0|
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+



########## SMART status report for ada0 drive (Western Digital Red: VK1EX8HY) ##########
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p9 amd64] (local build)

SMART overall-health self-assessment test result: PASSED



Ive been using this server and hd drives since Freenas and have updated it over the years with no issues. I hope someone can help me recover as I have family coming over and had a bunch of pictures I wanted to stream to the TV from this server.

 

[bent98]

more info

 

[bent98]

 

[bent98]

 

[Davvo]

Did you try rebooting? Please use [CODE][/CODE] when pasting terminal output.

 

[bent98]

Did you try rebooting? Please use [CODE][/CODE] when pasting terminal output.

Yes it rebooted a few times

 

[Davvo]

Try running zpool import BIGASSRAID.

 

[bent98]

Try running zpool import BIGASSRAID.

Do I have to do anything after that ? I’m going to be home in a couple of hours

 

[joeschmuck]

Physically examine the server, make sure all looks good.

Do you have a backup of your configuration files? Normally during an upgrade it will ask you to make a copy. If you do, I would try to Upload them again, let the computer reboot and see if things come back to life.

Another option is you should be able to roll back to a previous version of TrueNAS 13.0x using the boot environment. This may simply restore things back working.

 

[bent98]

Physically examine the server, make sure all looks good.

Do you have a backup of your configuration files? Normally during an upgrade it will ask you to make a copy. If you do, I would try to Upload them again, let the computer reboot and see if things come back to life.

Another option is you should be able to roll back to a previous version of TrueNAS 13.0x using the boot environment. This may simply restore things back working.

I tried to revert back to earlier version from 4 days ago and rebooted but that didn’t work . I didn’t back up config files for a while .

 

[HoneyBadger]

IO Crest SI-PEX40062

Marvell 88SE9235 chipset. Not ideal, but should be supported. I also note that your SDSSDA120G (Sandisk SSD Plus) seems to have a SiliconMotion SM2246XT controller and we've seen occasional issues when TRIM was enabled with those - depending on the age of your initial install, this might be a factor, but that usually shows up as "refuses to boot" or "boot-pool logs an error" not "main data pool won't import."

zpool import -f BIGASSRAID should be your first port of call - if it successfully imports, then run a pool scrub, reboot the server, and see if TrueNAS picks it up again.

Did anything preclude this outage eg: power loss?

 

[bent98]

Marvell 88SE9235 chipset. Not ideal, but should be supported. I also note that your SDSSDA120G (Sandisk SSD Plus) seems to have a SiliconMotion SM2246XT controller and we've seen occasional issues when TRIM was enabled with those - depending on the age of your initial install, this might be a factor, but that usually shows up as "refuses to boot" or "boot-pool logs an error" not "main data pool won't import."

zpool import -f BIGASSRAID should be your first port of call - if it successfully imports, then run a pool scrub, reboot the server, and see if TrueNAS picks it up again.

Did anything preclude this outage eg: power loss?

No

 

[joeschmuck]

My IO Crest SI-PEX40062 works like a champ for both CORE and SCALE. I'm glad I purchased something that was basic and just works. I have a second one laying around, likely in another computer that I need to another project. Hopefully I find it in a few days.

I didn’t back up config files for a while .

You can use an older file if you have not changed the configuration (users, NIC, etc). Hopefully forcing the pool to mount will be the correct fix.

Whatever happens, take your time and be very intentional with your actions.

 

[bent98]

Marvell 88SE9235 chipset. Not ideal, but should be supported. I also note that your SDSSDA120G (Sandisk SSD Plus) seems to have a SiliconMotion SM2246XT controller and we've seen occasional issues when TRIM was enabled with those - depending on the age of your initial install, this might be a factor, but that usually shows up as "refuses to boot" or "boot-pool logs an error" not "main data pool won't import."

zpool import -f BIGASSRAID should be your first port of call - if it successfully imports, then run a pool scrub, reboot the server, and see if TrueNAS picks it up again.

Did anything preclude this outage eg: power loss?

How do I run a manual pool scrub ?

 

[joeschmuck]

Code:

zpool scrub BIGASSRAID

 

[bent98]

Code:

zpool scrub BIGASSRAID

Thanks . As soon as I get home I will try to run import command suggested . If that doesn’t work . What would be the next step ?

 

[Davvo]

If that doesn’t work . What would be the next step ?

Depends on the reason it refuses to import. We will likely gave you the same command with a few parameters before attempting other ways.

If zpool import BIGASSRAID doesn't work, try zpool import -f BIGASSRAID.

And personally I think the pool is just offended being called RAID instead of RAIDZ.

 

[bent98]

 

[bent98]

Depends on the reason it refuses to import. We will likely gave you the same command with a few parameters before attempting other ways.

If zpool import BIGASSRAID doesn't work, try zpool import -f BIGASSRAID.

And personally I think the pool is just offended being called RAID instead of RAIDZ.

 

[Davvo]

Try zpool import -f -F -R /mnt BIGASSRAID and if it doesn't work, zpool import -f -F -D -R /mnt BIGASSRAID.
If this doesn't work, please provide the output of smartctl -a /dev/adaX for all the disks in your pool where X is the number of the drive: if there is a bad drive you can disconnect it and hopefully your pool should be able to get imported.

This will likely not be pleasant, warm up your backup.