Freenas crash after power outage, lost vol1 and can't restore

[jwtruett]

After a power outage I lost my vol1 which is 6 WD 2GB Red drives off of a IBM1015 controller. The system boots and recognizes the SSD off the MB SATA controller. Everything else seems to function correctly.

I've replaced:
power supply
SATA cables
IBM M1015 and reflashed

I can't tell but the hard drives don't seem to power up and I cannot tell if the drive controller is functioning correctly even though the flash procedure went correctly.

Any help would be appreciated as I've exhausted all my ideas.

 

[Arwen]

Welcome to the forums.

If you can supply the output of zpool import command, that will help get a better picture. Also, could you supply the pool layout, RAID-Z1/2/3 or Mirrors.

Last, ZFS was specifically designed not to loose data on power loss. Data in flight can be lost, but then that applies every single file system out their. That said, hardware faults can cause data loss or even cause a ZFS pool not to be automatically imported.

 

[jwtruett]

Excuse my lack of knowledge. If I open a terminal window and type "zpool import" nothing is displayed. It appears the command is run, there is no error, but nothing follows the prompt. As far as the pool layout, it consists of 6 WD Red 2TB drives in a raid configuration.

This system has run for so long without issue I am unsure of specifics. If there is more information you would like, please be specific about which commands to run and how to get requested output. I am happy to send whatever information you request.

Thank you for your help.

 

[jwtruett]

Could the old M1015 going bad marked the drive not to spin up on boot? Is there a way to check and see if the new M1015 is being seen and loading correctly?


Thanks

 

[jwtruett]

I installed a new WD Red 4TB on an available channel off of the M1015 and it is not recognized. I have 2 solid green lights on the front middle of the M1015 and 1 flashing green light on the top of the card. I'm unsure how to back track to see if the M1015 is loading correctly.

 

[Arwen]

If zpool import shows nothing, then ZFS does not detect any pools in the exported state. This can be for several different reasons, like the pool was imported. Or failed disk controller.

I don't have the diagnostic knowledge for that controller. But, it does sound like it is bad.

But first, you mention "raid configuration". Please clarify.
Are you using hardware RAID?
Or ZFS RAID-Zx? (And if so, what level?)

Try the zpool status command. It is unlikely to show your pool, but I have seen times when it took a TrueNAS server 10s of minutes to boot and import the pool(s).

 

[Redcoat]

Reboot your box and you should early see the M1015 Boot Screen with the detected drive information.

You can get the manual for the original configuration software at https://www.supermicro.com/manuals/other/MegaRAID_SAS_Software_Rev_I_UG.pdf
and the SW utility at https://www.broadcom.com/support/knowledgebase/1211161499804/lsi-pre-boot-usb-tool-download

I no longer have one to test to remind myself if SAS2flsh works with the M1015.

 

[jwtruett]

Reboot your box and you should early see the M1015 Boot Screen with the detected drive information.

You can get the manual for the original configuration software at https://www.supermicro.com/manuals/other/MegaRAID_SAS_Software_Rev_I_UG.pdf
and the SW utility at https://www.broadcom.com/support/knowledgebase/1211161499804/lsi-pre-boot-usb-tool-download

I no longer have one to test to remind myself if SAS2flsh works with the M1015.

When I reboot I do not see the M1015/LSI boot screen.

 

[Redcoat]

When I reboot I do not see the M1015/LSI boot screen.

That's not a good sign. Do you have another slot you can move it to? And what is your evidence for the flash having been good, did it show up in boot and allow you drive access that is not there now?

 

[jwtruett]

That's not a good sign. Do you have another slot you can move it to? And what is your evidence for the flash - having been good, did it show up in boot and allow you drive access that is not there now?

I thought when I refreshed to IT mode I deleted the boot ROM. I have moved the card to every available slot.

 

[Redcoat]

Yes, if you deleted the boot rom that would explain not seeing it. So, I would try sas2flash -listall and see if the card shows up...

You didn't say if you ever had access to drives after the flash.

 

[jwtruett]

Yes, if you deleted the boot rom that would explain not seeing it. So, I would try sas2flash -listall and see if the card shows up...

You didn't say if you ever had access to drives after the flash.

No, I was never able to access the drives. It almost like they are not receiving a spin up command.

 

[jwtruett]

No, I was never able to access the drives. It almost like they are not receiving a spin up command.

I have another controller that I don't think I reflashed. It shows at boot but still does not see the drives.

 

[Samuel Tai]

I have another controller that I don't think I reflashed. It shows at boot but still does not see the drives.

That sounds like your backplane could also be a casualty.

 

[jwtruett]

That sounds like your backplane could also be a casualty.

Is there a way to verify?

 

[Samuel Tai]

If you have a similar system, transplant the drives along with the working controller. If the drives all appear at boot, then it's very likely your original backplane is also toast.

 

[jwtruett]

If you have a similar system, transplant the drives along with the working controller. If the drives all appear at boot, then it's very likely your original backplane is also toast.

I don't have a replacement motherboard but the Supermicro X9SCLs are available use on eBay for $30. That is more than worth it to fix this.

 

[danb35]

Just to make sure the drives aren't available to the OS, what's the output of camcontrol devlist?

 

[jwtruett]

If zpool import shows nothing, then ZFS does not detect any pools in the exported state. This can be for several different reasons, like the pool was imported. Or failed disk controller.

I don't have the diagnostic knowledge for that controller. But, it does sound like it is bad.

But first, you mention "raid configuration". Please clarify.
Are you using hardware RAID?
Or ZFS RAID-Zx? (And if so, what level?)

Try the zpool status command. It is unlikely to show your pool, but I have seen times when it took a TrueNAS server 10s of minutes to boot and import the pool(s).

It only shows the SSD on the MB SATA port. No hardware RAID, ZFS, I think 2 (don't remember).

 

[jwtruett]

Just to make sure the drives aren't available to the OS, what's the output of camcontrol devlist?

only shows the SSD SATA