Problem, TrueNas CORE is dead

[georgelza]

Hi all, not sure what went wrong where, tried to log onto flex, did not respond, went onto the NAS, saw some errors on some of the pools, but nothing seen before, that does not clear itself.

tried to restart plex, did not want to... so issued a reboot of the NAS,

it now comes up, BIOS, then kicks over to LSI card, shows 4 buses as per attached and then cycles back to bios boot up screen.
MB: Gigabyte B250M-D3H

slightly panicking, not sure where to start.

thinking, need to check/confirm the boot up volume is in tact, this is on a microsd card that plugs into a usb port at the back.
for now i've disconnected the cable coming out of LSI controller to the HDD's

get the distinct feeling something broke, as i don't think it's getting anywhere close to accessing the microsd yet.
although I eventually got into my bios, it does not want to move away from the update bios / save bios screen.
thinking of removing all HDD's (including those plugged directly into the MB), and writing a bootable image onto a usb and see if i can get it to boot.

going to need some hand holding here...

G

 

[Davvo]

That's not a BIOS screen, it's a QFLASH screen that can only be used to either update or save your BIOS.

Your boot pool could have died.

 

[Patrick M. Hausen]

Write a known good USB drive with FreeBSD 13.2, check if you can boot that.

 

[joeschmuck]

If the above recommendations do not fix the problem.
1. Power off and unplug the electrical cord. Wait 5 minutes (2 will probably be good but it's only 3 more minutes and you should locate your motherboard user manual and be prepared to switch your BIOS).
2. Plug the power cord back in and power on the computer.
3. If the same problem occurs, are you able to enter the BIOS screen (read your manual for which key to press). If you can get into the BIOS setup screen verify your boot drive is correct.
4. If the same problem occurs, since you have Dual BIOS, switch to the other BIOS.
5. If this fails, Clear the CMOS (again, read your manual), did this fix the problem.
6. If the same problem occurs, unplug your PCIE cards from your motherboard and does that fix the problem, meaning it does not stop at the BIOS Flash screen.
7. If the problem still occurs I would Flash the current firmware to your BIOS (read the user manual and make sure you grab the correct file).
8. If you have had no success and continue to end on the BIOS flash screen, disconnect everything from the motherboard that is all add-ons. Leave the RAM, CPU, Keyboard, Mouse, Monitor. Power it back on, it should not end up at the BIOS Flash screen.
9. If you still have a problem, I would suspect the motherboard is faulty. You could start moving RAM around or anything else you can possibly think of, at this point you don't have much to loose. I would not touch the CPU unless you have reason. And you might try to reseat the power supply cables to the motherboard. At this time I have no reason to believe the power supply is the culprit.

If you need to provide feedback, please be very descriptive. Remember that we are not there with you and you do not want us to make assumptions as it takes longer to isolate the problem.

Best of luck to you.

 

[Constantin]

Check all power cables. Twice. I had several import failures thanks to a bad power connection on one backplane leading to a bad import pool event, then the lovely message that the pool was dead and to rebuild it from backup.

 

[NickF]

Good news is I think your data is probably safe.
The bad news is I think you have either a bad power supply (most likely) or a bad motherboard.

 

[georgelza]

That's not a BIOS screen, it's a QFLASH screen that can only be used to either update or save your BIOS.

Your boot pool could have died.

please explain this a bit more, i know about bios on mb and i know fw on expander cards like a LSI.
I like to understand what I'm told so excuse the 101 Questions.

G

 

[georgelza]

Write a known good USB drive with FreeBSD 13.2, check if you can boot that.

Will do, was a thinking by me also to confirm the hw excluding the TrueNAS stack. but sadly as per others, what I'm seeing looks like well before the HW is reaching out for OS to boot it.
thank BSD link is stuck at searching, will do search.
G

 

[georgelza]

If the above recommendations do not fix the problem.
1. Power off and unplug the electrical cord. Wait 5 minutes (2 will probably be good but it's only 3 more minutes and you should locate your motherboard user manual and be prepared to switch your BIOS).
2. Plug the power cord back in and power on the computer.
3. If the same problem occurs, are you able to enter the BIOS screen (read your manual for which key to press). If you can get into the BIOS setup screen verify your boot drive is correct.
4. If the same problem occurs, since you have Dual BIOS, switch to the other BIOS.
5. If this fails, Clear the CMOS (again, read your manual), did this fix the problem.
6. If the same problem occurs, unplug your PCIE cards from your motherboard and does that fix the problem, meaning it does not stop at the BIOS Flash screen.
7. If the problem still occurs I would Flash the current firmware to your BIOS (read the user manual and make sure you grab the correct file).
8. If you have had no success and continue to end on the BIOS flash screen, disconnect everything from the motherboard that is all add-ons. Leave the RAM, CPU, Keyboard, Mouse, Monitor. Power it back on, it should not end up at the BIOS Flash screen.
9. If you still have a problem, I would suspect the motherboard is faulty. You could start moving RAM around or anything else you can possibly think of, at this point you don't have much to loose. I would not touch the CPU unless you have reason. And you might try to reseat the power supply cables to the motherboard. At this time I have no reason to believe the power supply is the culprit.

If you need to provide feedback, please be very descriptive. Remember that we are not there with you and you do not want us to make assumptions as it takes longer to isolate the problem.

Best of luck to you.

hmmm, manual.. good points...
some questions.
as you mentioned, dual bios...
then why did it (2nd bios) not kick in and bring the system up, (will need to figure out how to instruct it to switch bios's)
I know F12 and F9 kicks me into various interfaces, if i recall from last night (I'm GMT+2) did not get much response out of the KB...
main power cable have been changed, to work on machine I needed to move it which then introduced a different, longer power cable as required where it is stamina vs where it was in my rack.
also thinking #6... if you see one of the photos shoes where it is showing scanning by the LSI card and then output from that, making me think we've gone pass ignition bios startup.

G

 

[georgelza]

Good news is I think your data is probably safe.
The bad news is I think you have either a bad power supply (most likely) or a bad motherboard.

like the Good news bit, i'm focusing on that part...
G

 

[Davvo]

please explain this a bit more

You entered the motherboard's utility to flash its own BIOS, not the BIOS interface.

 

[georgelza]

so... a little update

I first just unplugged the cable with the HDD that goes into my LSI controller... no change in behaviour. still does LSI scan, showing the devices plugged into it, then cycles back to bios boot screen,

I then removed the LSI controller which now (much faster) took me to a line that said
"This is a FreeNAS data disk and can not boot system. System Halted"

Now i restarted system, went to bios, reselected my boot device => external USB
(I plugged in a Ubuntu bootable USB.) and let it boot onto this (I want to see if the Ubuntu installer see 3 drives (plugged directly into MB) or all 7, 6 x 4TB HDD + 1 x 128GB M.2 SSD, excluding the boot usb), and surprise... it saw 6 HDD's, no mention of the M.2 SSD,
Guess this is good news, it means my LSI controller works.
What is/was/suppose to be on the M.2 it is a App boot pool, so things like Plex, qBittorrent, Unify controller and a FreeBSD Jail running some random apps like network speed tested + influxdb.
so bit perplexed, if that's whats on the M.2 why is it not booting... it should just have a problem with the pool.
some more fault testing.
G

 

[georgelza]

m

You entered the motherboard's utility to flash its own BIOS, not the BIOS interface.

right have hit think end or del which does that... while it was all dead, trying to see if there is life anywhere... if the KB batteries are good... it's a BTLE KB... and was worried some of my it's not listening to me might be the KB, since just switched over to a cabled KB to make sure i'm not fighting myself.
ok so all indications are the MB is ok,
G

 

[georgelza]

interesting, after removing the Ubuntu usb, and plugging the TrueNas boot USB back, booting and going back into bios to select boot device.
(it's seeing all 6 HDD's + the M.2 and the external TrueNas boot device).

Notice the list of possible Boot devices having change from early... to later. only thing i did was removed TrueNAS usb put in Ubuntu check it booted and then went back to TrueNAS boot device.

Can't recall if the boot device was tagged UEFI or not, having to try both. first try still took me to "This is a FreeNAS data disk and can not boot system.... "

G

 

[georgelza]

It's Booting.... :)

G

 

[georgelza]

wondering if the TrueNAS USB boot device is on the way out... - will look at replacing this. to be sure,
see attached, do believe my M.2 is gone that house App pool. - have some brand new 500GB SSD's I can put in. have 2, so can do a mirror pair.

options ?
happy to consider switching to scale, while we at it.

to consider :

1. Plex install - MOST Important , the data is spread over 2 pools located on the 6 x 4TB HDD's, app is on the M.2 SSD
2. Unify controller [have a export/backup],
3. qBittorent - no sweat, simple reinstall
4. BSD Jail, can rebuild

G

 

[georgelza]

No that it's been up once... lets check stability... going to do a total reboot.
See if we end with what we have currently.
G

 

[Davvo]

Not really a need to mirror the boot pool.

 

[georgelza]

so a boot took me back to this is a FreeNAS data disk, and a halt of system.
doing another power cycle
reselecting the boot device.
need boot stability before anything else.
G

 

[georgelza]

Not really a need to mirror the boot pool.

the device that's gone is not a boot pool, it was my app pool, where apps was installed.
think i happen to run into multiple things here.
the app pool failed, and at the same time it seems to be having a issue sticking to the correct boot device=> being a smallish USB
G