BTX Halted - Help Newb

[Stratis]

Let me start off by saying I'm out of my depth on running a NAS. A couple of years ago I got interested and set up a TrueNAS Core installation - even upgraded from a FreeNAS, but really never had the time to dig deeper into this tech. (BIG MISTAKE!)

So now that's it's been a up and running a while I looked at the TrueNAS dashboard and figured it was time to install upgrades. I was running TrueNAS-12.--U8.1 and I thought I should upgrade to 13. That's when my problems started.

I don't want to do anything further and loose my data so I'm posting to this forum for some expert help.

After what seemed like a good download and upgrade progress bar movement, it seemed the machine tried to reboot itself - I think it even said it was done with the upgrade. When I didn't see it come back online I went to check the console and got this screen:

After letting it sit like that for a long time (hour+), I rebooted and got this screen:

As part up the upgrade process, TrueNAS asked if I wanted to download a config/setup (?) which I did. I'm hoping that contains all the needed data to get this working again. Otherwise I don't think I have any stored passwords anywhere.

I also tried to launch the BIOS boot launcher to try some of the other drives (someone mentioned that on another post). The only one that did anything was the Zozt G3000 120 drive. I'm assuming that's the new TrueNAS install boot drive.

From what I have been able to understand from reading this forum it seems I might need to roll-back to 12.0-U8.1 but I'm not sure if that's the best thing to do, or even how to do it.

Some background:

The system is based on a SuperMicro MBD-X9SRA-0 with a bunch of small internal drives (it's an old rack mount surplus machine). I don't remember how many drives I installed (maybe 7), and I think I have a decent amount of memory. I can't get past the boot screens to check, but if it's important I will open up the box and take inventory.

Back when I did the original trueNAS install I must have had some issue, because I found a note in a folder I keep next to the computer that said "Had to install TrueNAS using legacy BIOS - UEFI just didn't seem to work - also couldn't get it to boot to an SSD. " Not sure if any of that is important but thought it might be a hint for someone.

I going to thank you for your patience in advance since I'm sure I'm going to need so extra hand holding.

Thanks!!

smk

 

[Stratis]

I'm still stuck on this - I'm hoping this "bump" will put my question back in front of some much smarter than me people.

Please HELP! I don't want to loose all my data because I did something else even dumber!!!

Thanks,

smk

 

[joeschmuck]

The only one that did anything was the Zozt G3000 120 drive. I'm assuming that's the new TrueNAS install boot drive.

You don't know which is your boot drive? That is a little problematic.

Please give us some system specs, it can help.
DO NOT UPGRADE THE ZFS POOLs, Disregard the Alert !!! If you accept it then you will not be able to roll back to version 12 should you need to.

This is not going to be easy if you really do not know much about your system. We will start from the basics.

First of all, do you know which drives are physically for the pool/VDEV? If you know exactly which drives are you data pool(s), with the power OFF, disconnect those drives. This should leave just the boot drive. How is this drive connected to your computer?

Power on the computer, you should have one drive left, select it to boot from.

If you can boot up the boot drive and get to the GUI, then make note of which drive it is. This is the best situation you could hope for.

Answer the questions here and report the results.

 

[Stratis]

Thanks for jumping in and helping out!

I took the machine offline and set it up on my bench. Then I removed all the drives except what I'm pretty sure is/was the boot drive, which is connected via a SATA cable. The drive is a small SSD:

ZOZT - 120 GB SATA G3000 (#SSDSG30120G01)

Additionally there were 6 other drives in the machine, which I think were the data storage drives.

Western Digital WD10EADS - 1TB (qty: 1)
Western Digital WD2003F2EX - 2TB (qty: 4)
Seagate Constellation ES.2 ST32000645NS - 2TB(qty: 1)

I then powered up the unit and this time it booted all the way to the TrueNAS boot screen:

Then it autobooted to the console screen:

I'm not sure how it booted all the way to the Console Setup screen this time when before all I got to was the BTX Halted message. What's different about not having the other drives installed?

Hopefully this helps you guide me to the next step.

Thanks,

Stratis

 

[joeschmuck]

So this is a good thing and the best you could expect.

Please read everything before making any moves, and read it again. Try to comprehend what my game is. If this confuses you, do what you can and report back.

What we know now is part of it is working. Do you have a network cable connected? Having the GUI would be the next step. Once you get the GUI running then make sure it's the correct version of TrueNAS. Next make a backup of the TrueNAS configuration file and save it on your local computer (that you are running the GUI on). This is just in case you need it later. A TrueNAS 13 backup file cannot be used on version 12 as far as I'm aware, but version 12 can be used on version 13.

If all is good so far and the GUI looks fine, STOP!
Do you want to roll back to version 12.x ? If yes, Go into the GUI and change the ACTIVE Boot Environment, then reboot. Make sure the system bootstraps correctly. If it does the continue.

Power off and reconnect your hard drives.
Power on and go into the BIOS to select the Boot Drive, make sure it is correct and let the system boot up. Pay attention to everything noting what may not look normal.

If all is good then your system will boot up normally. If it fails with the same original failure then How did you install TrueNAS 13? Did you do an online upgrade via the GUI or did you download the ISO image and do a local upgrade or clean install? The devil is in the details. Here you probably cannot provide too much information.

If you did a local manual upgrade or internet upgrade the system "should" be recoverable. If you did a clean install and had to pick the drive to install to, you may have messed up a data drive. What I'm saying is not that you did that, just that its possible.

At this point if things are not working, power off, unplug the boot drive. Power On. See if you can boot from another drive and get the system crashes. You could do this one hard drive at a time if you like. Use your imagination. The goal it to find out if you installed TrueNAS to a data drive by accident.

If nothing comes of this, post all your results and observations. Our goal here is to ensure your data remains safe and recoverable.

 

[Stratis]

I reconnected the server back to my network and was able to pull up the GUI:

You asked that I make a backup of the TrueNAS configuration file. Is that the option located under the System/General menu?

You also asked how I installed the upgrade. I used the GUI to do it remotely. I didn't download anything to a USB or anything else at the physical server. I just followed the GUI instructions.

Since this is a learning opportunity for me, can you explain why the system seems to boot up now just fine when all the other drives have been removed? I know at some point I tried to boot the server from each of my drives via the BIOS screen. Why wouldn't the ZOZT SSD drive boot up with the other drives connected?

I did notice that under the System/Boot menu it seems that I have multiple Boot options. I assume that means I had upgraded previously without issues.

As for your last question about which environment I want to keep 12 or 13, I really don't know enough to even choose. I'll be happy to follow your recommendations. I would like to get more involved with this server and expect to keep it updated into the future. That is after I create another backup of all my data first at another location!!!!


Thanks again - I'll wait for you answers before I proceed.

Stratis

 

[joeschmuck]

You asked that I make a backup of the TrueNAS configuration file. Is that the option located under the System/General menu?

Yes, Save Config.

Since this is a learning opportunity for me, can you explain why the system seems to boot up now just fine when all the other drives have been removed?

My theory was that you may have installed TrueNAS to a different drive if you did a manual update (in-person). But that was not the case, now I suspect there is something wrong with your pool, "maybe". Hell, it may work perfectly fine now, let's hope so.

I did notice that under the System/Boot menu it seems that I have multiple Boot options. I assume that means I had upgraded previously without issues.

Yes, and you are able to easily roll back if you desire and then delete the 13.0-U6.1 if we go that route.

As for your last question about which environment I want to keep 12 or 13, I really don't know enough to even choose. I'll be happy to follow your recommendations.

Since the previous version worked fine for you, you can mark it as KEEP. This way it will not be accidentally deleted when the boot drive becomes full.

Now lets keep working to fix the problem you are having.
1) In the Boot Environments, "Clone" 13.0-U6.1 and once the clone is done, Make it Active.
Why do this? this will preserve the original as we are going to try a few things. If we screw it up, you just make the 13.0-U6.1 active and reboot, then you can delete the clone and it is like nothing ever happened.

2) Reboot, make sure the clone works. If it boots up, then shut it down.
3) Reconnect your hard drives and power up.
4) Did TrueNAS boot up all the way? I'm hoping so but it may not. If it boots up, check your pool status, make sure all looks good, No errors. If you are satisfied then run a SCRUB from the GUI.
5) If the system crashed again, now time to make some changes. Make sure you know where your backup configuration file is located at.
6) Disconnect your drives and bootup back to the GUI.
7) Reset to Factory Settings. You can use the GUI or the Console. Realize that your IP address may change unless you are using DHCP.
8) If it boots up, power off, reconnect your drives, power on. Does it crash?
9) If it crashes then you may have a severe problem or TrueNAS 13 is just not compatible.
10) Power Off, disconnect your drives, reboot to the GUI. Go to the Boot Environment and make 12.0-U8.1 Active, reboot.
11) If it comes up, power off and reconnect your drives. Power On, Cross Your Fingers. This "should" put you in the same situation you were in before the upgrade.

REMEMBER --- Do not upgrade your pools to a newer ZFS version, you do not need it and it will block you from rolling back to an earlier version.

That is a lot of steps but the goal is to get the system running with the pool drives installed.

I'm not familiar with your motherboard however I'm thinking it MIGHT be too old for the SAS connections (SATA 0-3). Just a thought. You may be stuck with TrueNAS 12.0-U8.1.

I await your feedback.

 

[Stratis]

Yes, Save Config.

I saved the Config File

Since the previous version worked fine for you, you can mark it as KEEP. This way it will not be accidentally deleted when the boot drive becomes full.

I marked the 12.0-U8.1 environment as KEEP.

Now lets keep working to fix the problem you are having.
1) In the Boot Environments, "Clone" 13.0-U6.1 and once the clone is done, Make it Active.
Why do this? this will preserve the original as we are going to try a few things. If we screw it up, you just make the 13.0-U6.1 active and reboot, then you can delete the clone and it is like nothing ever happened.

I made the clone but haven't made it ACTIVE yet - before I go further should I also mark both the clone and original as KEEP?

Sorry I'm taking this slow, but I don't want to screw things up.

 

[joeschmuck]

I made the clone but haven't made it ACTIVE yet - before I go further should I also mark both the clone and original as KEEP?

You do not have to. The oldest environment will be deleted first if needed, but the KEEP was for before you made the clone, just as a precaution.

Sorry I'm taking this slow, but I don't want to screw things up.

Nothing wrong with that.

 

[Stratis]

2) Reboot, make sure the clone works. If it boots up, then shut it down.
3) Reconnect your hard drives and power up.
4) Did TrueNAS boot up all the way? I'm hoping so but it may not. If it boots up, check your pool status, make sure all looks good, No errors. If you are satisfied then run a SCRUB from the GUI.

Sadly this didn't work. It does seem that the hard disks were recognized by the system though - this was before it crashed with the BTX message::

5) If the system crashed again, now time to make some changes. Make sure you know where your backup configuration file is located at.

I'm assuming this is the config file you had me download when it booted to TrueNAS 13 - I think I also might have one I saved before I performed the upgrade under TrueNAS 12.

I see your next steps are to get the system up and running with TrueNAS 13. I'm just curious, would it be better to boot under 12 and burn a full copy of my data (assuming it will boot) and then try to get 13 up and running? At least then I would have saved my data and can tweak things without worrying.

Thanks

 

[joeschmuck]

10) Power Off, disconnect your drives, reboot to the GUI. Go to the Boot Environment and make 12.0-U8.1 Active, reboot.
11) If it comes up, power off and reconnect your drives. Power On, Cross Your Fingers. This "should" put you in the same situation you were in before the upgrade.

See if this gets you back online.

 

[Stratis]

SUCCESS!!!!

YOU ARE A HERO!!!!

I'm back to where I started...now I know that I can roll back to an earlier configuration. This should really help me further troubleshoot why TrueNAS 13 won't boot up for me.

First I'm going to make a massive offline copy of all my files and then see why 13 won't work.

Given that 13 will boot up on my SSD, I wonder if one of my data drives is the issue. They are of different sizes and manufactures so maybe one or more are just too old and not compatible with 13. Could I try only connecting one drive at a time looking to see if I can get past the BTX Halted message? Also what is that message actually telling me?

Could it be the machine's BIOS? I'm running ver 2.15.1227 and when I checked Supermicro's website they are at BIOS ver 3.2.

I can't thank you enough for walking me through this process!!! I've definitely learned a lot in the process. You've given me the help to understand how to work with TrueNAS and be able to make changes and have a way to roll-back to a known good configuration.

If you have any additional thoughts of what I should troubleshoot first (after my offline backup), I should have some time this weekend to give some things a try. Also, is there something I should be doing to verify that my data's integrity? You mentioned something about a scrub?

Thanks again for your help!

smk

 

[joeschmuck]

Given that 13 will boot up on my SSD, I wonder if one of my data drives is the issue.

While I guess it could be possible, I really doubt it is a drive causing the issue. Not the drive physically. Maybe the drive format, that could be it.

I again, would not upgrade the pools to the new ZFS version, you will not be able to roll back. And the newer version is not required, or I have NEVER seen it be required.

Could it be the machine's BIOS? I'm running ver 2.15.1227 and when I checked Supermicro's website they are at BIOS ver 3.2.

Yes, it could be. Let's talk a bit here about your old motherboard.
Your motherboard has six SATA connectors near the front of the board. These are TRUE SATA controller connections. Then you have four SAS (same connector as SATA) connections. SAS is a different controller. Your problem could be coming from TrueNAS 13 does not support your SAS or possibly SATA controller. A firmware upgrade MIGHT make it work. That is a big might.

Here is the link to a connector layout, items 19, 20, 21, and 22 are the SAS ports.

https://www.supermicro.com/QuickRefs/motherboard/C600/QRG-1281.pdf

In my opinion, your Boot drive should be plugged into one of the four SAS ports. If it already is, then maybe it is the SATA controller.

Just be prepared to stay on TrueNAS 12 is all I'm saying, if we determine this. Or you could purchase an Add-On card for SATA ports that are supported, or just buy a new system. Your problem is not an easy one to isolate and identify to a specific item.

Another thing you could try, install TrueNAS 13 on a USB flash drive which means you do not have the SAS ports connected at all. If that works then you know it's the SAS ports.

Making a backup is very smart, but it will suck to have to restore it if you need to later, so try to keep your data in-tact.

My recommendation at this point, because I'm not certain you have done this is, if you want to migrate to version 13...

1) In the GUI, delete the TrueNAS 13 boot environment. make sure the current boot environment still is tagged as KEEP.
2) Delete any other boot environments you desire but you don't need to do that.
3) Make a backup of your configuration data again, make sure it is saved on your local computer, not the NAS.
4) Download the ISO of the TrueNAS software.
5) At this point, if you have another drive you can use, power OFF, replace the Boot Drive with a new clean drive.
6) Disconnect your data drives. This is to prevent you from accidentally installing TrueNAS on one of your data drive. Yes, it is possible.
7) Install TrueNAS 13.0-U6.x to the only drive in the system. Configure the bare minimum, Root password.
8) Power off and wait a few seconds, power back on. Does it boot okay, no issues?
9) If yes, now power off again, reconnect your data drives. Power On.
10) Does it boot up okay? If it does then continue, if not then skip to the section below.
11) Upload your configuration file, the system should reboot, if it does not, make it happen.
12) Your GUI should be on the same IP address as it was originally, log in and see if all is working. The best outcome is Yes.
13) Have I said this enough? Do not upgrade your Pool, dismiss the alerts.
14) All should be good. Why would this have fixed it? Often I have seen an online upgrade just fail.

If step 10 fails:
1) You need to do some troubleshooting. It is very repetitive.
2) Power off, Disconnect all but one data drive, Power On.
3) Does the system boot? This will not cause any issues with the pool, the pool is not recognized by TrueNAS yet since the system is not configured.
4) If the system boots without any issues, Power Off, connect the next drive, leaving the previous drive connected. Repeat this until either all the drives are reconnected and the failure does not happen (highly unlikely since we are here in the first place), or until one drive causes an error.
5) If the error only occurs when the last drive is attached, then disconnect a previous drive so you still have one drive out, Power On.
6) The best case is you have another failure and the problem is likely the last drive you attached. The worst case, the problem goes away indicating that only when all data drives are connected, there is a problem.
7) HOWEVER, we do have one other thing to try, maybe the issue is a SATA controller or data cable. Move that drive data cable connection from the back of the suspect drive, to another drive. The goal is to reconfigure which SATA controller port the drive is connected to. And it's important it is the back of the drive connection.
8) If relocating the SATA cable makes the suspect drive move, then replace the SATA cable. If that fixes it, good, if not then the SATA controller is suspect.

Just to be clear... The drives are all connected directly to the Motherboard SATA ports, not a

My way of troubleshooting is just how I would likely do it. If the system were in front of me, I'm sure I'd figure it out in under 1 hour and I would not follow the instructions I listed above because an indication may happen that would guide me down another path, be the idea it still the same and it is very difficult to guide someone remotely to do these things.

IF you are able to get the system working (not crashing on boot), take it a step back to induce the fault again to confirm what you just did was the issue. Always confirm.

This is a lot and it's up to you if you want to take this on. If your system is working and doing what you need, I would honestly leave it alone. But if there was something you needed that TrueNAS 13 offered then I understand. Also, while I am not promoting SCALE yet, it might be worth looking at. It is a little different but the GUI will feel very similar. The base is built off of Debian (linux) vice where CORE is FreeBSD OS. I would only try that if TrueNAS 13 was not working for you. SCALE is the future, but I'm not moving over to is for a while as it becomes a more stable product over time. Some please love it. Just food for thought.

Let me know what you end up doing.

 

[Stratis]

I took the time and created a complete back-up off all my data and now have it sitting offline - THANKS for getting me there!!!

I was started to follow your very detailed (and much appreciated) instructions but I stopped at the SATA SAS part. I see that one of my drives is plugged into SAS port 20, while all the other drives are plugged into SATA ports 29, 30, and 32-35. Could this be my issue - i.e. just using the one SAS port?

It looks like I needed the extra port to reach my file storage goal. If this is the issue then I could look to either decrease my overall pool size or just buy some larger drives.

I'll pause here and see what you think before I go down the next rabbit hole.

smk

 

[joeschmuck]

Where is your boot drive plugged into? What connector? If it is not connector 20 then swap the connectors so your boot drive is in connector 20 and the drive that was in connector 20 is moved to one of the other connectors.

The problem appears to be the driver, if all your data drives can be placed on the connectors as I indicated above and you boor drive can be on connectors 19-22, then you just need to setup the BIOS to boot from the boot drive. If you like, unplug all the data drives, leave ONLY the boot drive in connector 20, setup your system to boot from it. Once done, power off and reconnect the data drives. Power On and see if it boots, if not, go into the BIOS and set the boot drive. Should be easy but if you are unable to boot form the SAS connector then you may need to re-evaluate your situation. You could also boot from a USB drive, although I would not typically recommend it, but there are folks who do this. Get a SSD to USB adapter and go that route. But first try what I have said, hopefully you can boot from the SAS connector.

 

[Stratis]

OK, I moved my boot drive to connector 20 and the system had no issue booting up under both TrueNAS 12 and 13. So I then tried to boot up under 13 with the data drives and I got the BTX Halted message again. I then tried just booting up with a single data drive and that still caused the error. So it seems that the issue is with the SATA controllers.

I suppose the next step would be to try and update the BIOS and see if that makes a difference. Or as you said I could stay at 12 and look to upgrade the SATA controller or just get a new machine.

I'd hate to trash this machine - it's a beefy box with a Xeon CPU and 16 GB of RAM and up until now it's been working beautifully.

 

[Stratis]

Just saw this message on the Dashboard - could it have any connection to my current issue?

 

[joeschmuck]

The expired certificate is a problem. It will not stop TrueNAS from booting.

You never stated that you updated the BIOS to version 3.2, did you? If not, do it and try again to boot the system.

So I then tried to boot up under 13 with the data drives and I got the BTX Halted message again. I then tried just booting up with a single data drive and that still caused the error.

"Exactly" what was the message. The same BTX Halted after what looks like a register dump? Or something like this is a data disk? A screen capture would be handy if it's something different.

OK, I moved my boot drive to connector 20 and the system had no issue booting up under both TrueNAS 12 and 13. So I then tried to boot up under 13 with the data drives and I got the BTX Halted message again. I then tried just booting up with a single data drive and that still caused the error. So it seems that the issue is with the SATA controllers.

This is very strange, very strange. If the boot drive is plugged into the SAS port and no other drives are connected, and the system fully bootstraps that is a good sign of the SAS controller can support booting, which is what I expected.

But why does it crash when a data drive is connected as well. Makes no sense.

Just to recap: The boot drive is plugged into connector 20 and there is no other cables on connectors 19, 21, or 22? And TrueNAS 13 boots fine, minus the certificate error?

And when you power back down and connect a data drive to connectors 29, 30, 32, 33, 34, or 35 then the system fails to boot?

If all of that is true then I have one last thing to try.
1) Create a Bootable USB Flash Drive with TrueNAS 13 on it (the most current version as it will save you from a time consuming update).
2) Connect all your data drives to connectors 29, 30, 32, 33, 34, and 35, DO NOT use the SAS connectors for anything.
3) Bootup from the Flash drive.
4) Does the system boot or crash?
5) If it crashes, I'm out of ideas at the moment. If it boots go to step 6.
6) Restore your configuration file. Make sure all works. Fix that certificate (there is a lot of mention about that in the forums).
7) Buy a USB to SATA adapter and plug your SSD Boot drive into it. There are many adapters that will power the drive from the USB port.
8) Does it boot? If yes then the problem is "fixed" for this system.

I have not idea if you have any spare/unused hard drives laying around, one that has not been formatted for NAS use, but if you do, lets go back to the beginning and just plug in your boot drive to conector 20 and connect this completely different drive to port 32. Does it boot?

Here is what I'm thinking and it is grasping for straws because your problem is so unique, but maybe there is something odd about your data drives formatting.

That is all I have right now. Oh, I don't think I ever asked you about the amount of RAM you have?

The absolute last think I have is to possibly run a CPU Stress Test and MemTest86 to verify all is solid.

 

[Stratis]

SUCCESS!!!!!

Updating the BIOS seems to have fixed the major issue! The system is now booting up in TrueNAS 13 with all the data disks connected. Given all the issues I've had, is there something I should do now to verify my data's integrity? Some test I should run?

Also, I still see that certificate message. I suppose I need to address that too.

Thanks again for all your help - you've been amazing!

 

[joeschmuck]

I should do now to verify my data's integrity? Some test I should run?

Run a Scrub on your pool. You can do this from the GUI. Odds are you will have no issues but that is the test. Additionally a Scrub by default is scheduled to run once a month. You can check the status of the Scrub from the GUI as well.

Other testing I recommend if you have not already done so:
1) Daily SMART Short tests on all your drives. This test takes approx 2 minutes and you can setup the test to perform on all drives at the same time. I typically recommend as an example to run the Short tests at 2AM each day.

2) Weekly SMART Long tests on all your drives. This test can take some time for a drive to complete the test depending on it's capacity. Since the largest drive you have is 4TB, I would expect the testing to take about 4 hours. You can look at the polling time for Long/Extended and the value is in minutes by running smartctl -a /dev/ada0 as an example and near the top of the output you should find something that talks about polling or completion time for extended test (it will be the largest time). For argument sake call it 260 minutes or 4 hours and 20 minutes. You have a choice here, since you do not have many drives then your drives should not get very hot while doing the Long test all at the same time but you could setup the schedule to run one drive a day (Mon/Tue/Wed) for example at 2:15AM. Why this time? It is 13 minutes after the daily Short test. You could make it at 2:05AM as well but "IF" there is a lot of data work happening, the SMART tests will give priority to the data requests and that slows the speed of the SMART test down. We want to ensure there is no overlapping tests as the Long test will not be allowed to run if another test on that drive is in progress. I'm extra cautious so 2:15AM feels good to me. Again, just an example.

With these added tests you can be certain you are doing as much testing as you should do to ensure you are warned before a typical failure. This does not help for a catastrophic failure such as the drive motor electronics failing all of a sudden. SMART was designed to give you "up to" 24 hour notification that a drive failure may happen, but it's not perfect, it is just what we have today.

I'm glad you got it all working and if you need anything else, just ask. Don't be surprised that if I feel the answer is a easy find, I will recommend a Google search for "truenas " and appropriate words about the issue or question. There is a lot of data out there, some people just don't know how to look for it. Just be careful because if it sounds too good to be true, it probably is not correct.

I provided a lot of information above in the spirit of hoping you would understand more on how this works, not trying to confuse you. So in summary: Everything can be done in the GUI. Run a Scrub, let it finish. If there is an error, TrueNAS will let you know. Next ensure you are running routine SMART tests.

Best of luck to you.