boot up stuck at importing pool

[nkrapf]

today our freenas stopped responding.
once we got a chance to look at it we noticed it looked like it was rebooted and sitting at

Importing Pool
txg 2796838 open pool version 5000; software version 5000/5; uts 9.3-RELEASE-p1
tsg 2796842 import pool version 5000; software version 5000/5; uts 9.3-RELEASE-p16 903000 amd64

its been at this state for over 3-4hrs

any idea if i should leave it sit here or what?


Supermicro server with 16 2TB drives
48G ram
dual cpu
mirror 240 SSD
single 320 SSD
LSI 9211
boots from usb stick
latest update release of 9.3 9.3-stable 201506292332

single pool
single vol
presented to a windows server via iscsi

attached is screen we are stuck on and storage config (not exact but very close, we have 2 of these set up)

 

[joeschmuck]

Please post your system hardware specs/configuration and FreeNAS software version if you are expecting some assistance.

 

[nkrapf]

updated, thanks

 

[joeschmuck]

Sounds like your boot USB drive is corrupt. I'd grab a new/different USB drive and install FreeNAS again, then restore your backup configuration file and cross your fingers. Report back how that works.

 

[SweetAndLow]

Nvm

 

[nkrapf]

i have not replaced usb yet, but i was able to boot to the original config profile.
there i ran zpool import and zpool status and it all looked ok.
from gui i did import vol, selected the vol and it went to step 2. how long would the second step usually take ?

 

[SweetAndLow]

If you imported the pool from the cli you shouldn't import it from the GUI also. You should unmount it and just use the GUI.

 

[nkrapf]

I let it import the vol over night and it was finished in this morning, i rebooted and booted in the last profile made when we updated last and it seems to be ok now.
any idea what the issue could have been?

 

[cyberjock]

Yes.. sync destroys of 2TB of data like I said above. ;)

 

[nkrapf]

cyberjock, i dont see any previous posts on this topic from you, not sure what your refering to.

 

[cyberjock]

ROFL... different thread but similar question.. https://forums.freenas.org/index.php?threads/mounting-local-file-system-hang.35339/#post-214309

 

[nkrapf]

ah good info. can you point me in the right direction as to where i would find logs to try to track down why it originally went offline?

 

[cyberjock]

Can you post a debug file? System -> Advanced -> Save Debug.

 

[nkrapf]

here you go, issue started about 24hrs ago
and yes we are using raidz1-0, which i see you recommend against. If we had space to migrate data off and rebuild i would.
this is a backup target for our daily backups, these backups then are replicated to a second off site freenas server. then the servers are replicated off site as well.

 

[cyberjock]

Do you guys start your servers every morning?

I'm a but puzzled by the log. I see on Jul 3 at 0000 a log entry was made due to midnight rollover. The next entry was at 6:04:03 and was an entry that the server was booting up. So either it paniced, someone powered the box off, or someone hit a reset button.

 

[nkrapf]

no we do not reboot these boxes unless its required and i dont recall rebooting it on that day.

 

[cyberjock]

Well, there isn't a smoking gun for why the box would reboot. It looks like it's rebooted multiple times lately based on the logs. But no reason is given. You also don't have any crash data in the debug file, so the box doesn't appear to be panicing and rebooting.

So it seems logical to conclude that the box is just spontaneously rebooting. 99% of the time, if the machine is rebooting and not crashing, the hardware is to blame. Unfortunately that's all I can tell you about what is going on. You might want to look in the IPMI logs for the system (if it has IPMI) and see if any entries exist. If not, you're kind of stuck with trying to find the proverbial needle in a haystack.

You might want to try the common stuff:

- Swap the power supply if you have a spare (this is often an easy thing to do which is why I mentioned it)
- Update your BIOS, IPMI, etc.
- Make sure the system is attached to a good UPS that has surge protection.

 

[nkrapf]

Thank you for taking the time to look over the logs.
the several reboots were from trying to get it back up and running after the self reboot.

After starting to write this i went to check and it appears the server must have rebooted again.
Its stuck at the attached screen shot screen. So we should let it sit here till its done importing?

 

[joeschmuck]

Did you use a NEW USB Flash boot device? I don't recall seeing you ever state that. These devices do go bad in a FreeNAS environment.

I agree with Cyberjock, a hardware issue could be causing reboots. First ensure you have replaced the USB flash drive with a good new one. If that don't fix it, grab any computer with at least 8GB RAM (ECC preferred) and move your drives and USB stick over to it. See if things work fine. This would rule out your hardware other than the USB and hard drives.

 

[nkrapf]

after system booted we took a new backup of configs,
had issues with the download image hash not matching so downloaded 1 release prior to main download
had issues with usb sticks we bought (sandisk cruze 36G), kept giving errors after install reboot
tried another usb stick (generic) and would not even install on those
ran basic mem test to rule that out
tried a third set of usb sticks (PNY) which freenas seemed to like and was able to get everything back up and running on new usb drives.
tested everything and tested data on pool and everything looks good so far
started boot drive mirror process
it did not reboot last night so we will monitor it and give update in a couple of days.

Thank you everyone for your support and feed back.