Dead boot device... some guidance would be appreciated.

[Robert Thomspon]

Hey guys... It seems that my SD card boot device decided it didnt want to be an SD card anymore...

Anyway, was wondering, what are the procedures to get a working system again (i had backups, but i believe they were on the SD card that took a shit... i know, not good practice)... I dont have anything super important or not otherwise backed up from this NAS, but would prefer not to have to copy all of it back over....

Is it as easy as re-creating a USB stick/SD card to boot freenas from? Or are there a bunch of extra steps after that to get it back up to where it was before it died.

(I believe it's SD card because of the errors its throwing when trying to boot... POSTs fine, begins boot to FreeNAS (if it can find the boot file... sometime it locks up looking for it)... anyway, I was having issues SSHing into the machine and accessing it by web interface.Went to the physical machine to try to reboot, it locked up after asking if i want to reboot Y/N (stalled there for about an hour til i hard reset it... and now i cant get past it trying to create the ZFS pool....

 

[Jailer]

You don't need to recreate your pool. Full hardware specs would be helpful to help diagnose your problem since we know nothing about your system.

 

[Robert Thomspon]

You don't need to recreate your pool. Full hardware specs would be helpful to help diagnose your problem since we know nothing about your system.

Not sure how much help the specs will be... but,
HP Proliant ML350 Gen8, 2x2690 V2, 96GB Ram, 4x8TB Seagate Barracuda HDDs, FreeNAS 11.2 (Im assuming thats the most important part of the system specs... ) All boots off of the internal SD card (i had a hell of a time trying to get the internal USB to boot sucessfuly for some reason... )

System ran fantastic for the last 2 months or so, started having weird issues when Kodi would end playing a show... id get a 20-30 second lag til it dropped out to a menu (all of the kodis in the network pull from a MySQL database running in a jail on the FreeNAS)... Then I went to reboot it today and i couldnt get on it on the web interface and SSHing into it would either tell me my "home" folder was missing, or it would just sort of freeze as soon as I SSHed into it...

I DID get it to boot sucessfully and made a backup... but its freezing up again...

(So... I wrote the above like 2 hours ago... It didn't post for some reason and when I logged on with my phone, it was sitting there waiting for me to finish ;) )... That being said, I am not getting the error message (when it tries to boot from 'USB Let's (SD card), I get: No ZFS pools located, can not boot.

 

[Robert Thomspon]

OK... I think... Maybe I need help on identifying if the issue is the boot media (SD card) or a failing/failed hard drive....
I feel like a failed hard drive still boots free as the whole way... But without storage or crippled storage, and a failed boot media would cause it not to boot...

I ask, because, please see attached screen shots (phone pics taken of the console connected to free as...)

 

[Heracles]

Hey Robert,

As long as the issue is limited to the boot device, the pool should remain intact. You need to create yourself a new boot device and boot up your server with it. You will need to re-configure the network interface first.

Once you can reach the WebUI, you will be able to import your existing pool. You do it from the Storage menu ; Pool.

The thing is that all your configs are lost : your shares, users, etc. If you put your system dataset in your main pool (good practice and by default), you should be able to recover a system config backup from that dataset. Once you got it, you can restore it.

After that, you should be back in business : your pool of data and your system config as they were.

Good luck,

 

[Robert Thomspon]

OK, well, THAT was a nightmare and a half...
I spent the better part of yesterday trying to get this rig to take an install of FreeNAS, only to get 9 hours into it and finally get it to install completely on an SSD drive and then to have it give me a weird white screen error... (right 1/3 of the screen was just real close white lines, left side was blank)
Most of the fight had to deal with FreeNAS losing its shit every once in a while for me having a USB keyboard attached (good luck installing via telepathy :) )

Anyway, FINALLY got it running, all looked good save for an offline error on one of my drives (being replaced today)... so, I flashed the old settings back on to the thing, reboot... BAM, same old 1/3 white screen... WTF!!! So, I went to bed, only to wake up and find the rig just happily humming away, being a good little NAS.

Im not 100% sure what went wrong other than the boot device (SD card) got corrupted or damaged... follow that with the keyboard lockup issues (KDB: enter: panic), and trying to use USB3 keys on this rig (it gets REAL tempermental with USB stuff sometimes... though, that may be freeNAS).

Anyway, wanted to report that, as of now, I am up and running... Ill replace the failing drive tonight and let it resliver overnight... and report back in the morning... Ill mark this thread as solved if i can go a couple days without it shitting the bed again...

 

[Heracles]

Hi again Robert,

That sounds like a defective USB port / USB hub. For that many USB devices to go crazy, it really points to hardware problem. FreeNAS has no problem with USB keyboards, as long as they work as they should. If the keyboard, port or internal hub is defective, it can do that. FreeNAS does rely on the hardware to behave correctly and if it does not, it can create that kind of problems.

Any chance you can try with different port ? Ports from the back / from the front as they may be linked to different hubs ? Have a chance for another keyboard than this one ?

Good luck troubleshooting that one,

 

[Robert Thomspon]

Good luck troubleshooting that one,

I KNOW, RIGHT!!!! :)


Actually, on a different part of this...
The hard drive that is failing is actually a 6TB drive for some reason, not an 8...
I picked up an 8 to replace it with, will this cause issues on the resilvering? or will it effectively function as a 6TB drive rather than an 8?
(Im not putting the new drive in until tonight so it can resilver overnight... last time i needed to do this, i t took like 14 hours or something like that :)

 

[Jailer]

I picked up an 8 to replace it with, will this cause issues on the resilvering?

Nope.

or will it effectively function as a 6TB drive rather than an 8?

It will show up as the same size as the smallest drive in the same vedev.

 

[HoneyBadger]

Hold up, are you running 4x8TB or 4x6TB? The post back here says:

HP Proliant ML350 Gen8, 2x2690 V2, 96GB Ram, 4x8TB Seagate Barracuda HDDs, FreeNAS 11.2 (Im assuming thats the most important part of the system specs... )

So were they all 6TB, all 8TB, or a mix?

Also, "8TB Seagate" is my danger phrase these days. Can you post an exact model number so that we can be sure you don't have Archive/Shingled drives?

One more crucial piece of information about your system as well ... are you using onboard SATA, a built-in SAS controller (potentially RAID?) or an add-on card to connect the drives?

 

[Robert Thomspon]

Hold up, are you running 4x8TB or 4x6TB? The post back here says:



So were they all 6TB, all 8TB, or a mix?

Also, "8TB Seagate" is my danger phrase these days. Can you post an exact model number so that we can be sure you don't have Archive/Shingled drives?

One more crucial piece of information about your system as well ... are you using onboard SATA, a built-in SAS controller (potentially RAID?) or an add-on card to connect the drives?

They were supposed to be 4x8TB... But for some reason, there's a 6TB drive thrown in the mix... So, 3x8TB+1x6TB... The 6TB is the one throwing errors... I actually assume I've been running with 18TB of total storage (6TB lost to the redundant drive) and will see an actual gain of 6TB by removing the 6TB drive with the 8TB (bringing it to 32TB of hard drives, with a loss of 8TB for the redundancy, bringing my storage space to 24TB...)

That being said, I have a separate card flashed to HBA to run the drives, it connects to a passive backplane and all are run as SATA (I've never effectively gotten SAS to work correctly with FreeNAS)

And the 8TB seagates are pulled from USB enclosures (super cheap at Costco... I paid $120 for this one)... And they tend to be fairly reasonable if they fail (I've gone from 4x2TB to 4x5TB to 4x6TB to 4x8TB, only ever had 1 of the drives fail (5TB)... And I don't store anything critical on this beast, just mostly movies and TV shows... It also runs a couple jails/virtual machines, but all of that data is kept on a separate drive backed up once a week (that is sort of critical data... But only insofar as needing it to be kept off the machine once every week...)

Sorry for the long response...felt it appropriate to explain ;)

 

[Robert Thomspon]

Whoops... Missed the model number...

Seagate barracuda ST8000DM004

 

[HoneyBadger]

Yep, those are SMR. For long term archival storage this will be okay, and as long as you never overrun the cache on each drive it might be okay; although the resilver might shake a few gremlins loose. Look up threads from the poster @deafen though as he's catalogued his experience with SMR drives over a couple posts.

 

[Robert Thomspon]

Yep, those are SMR. For long term archival storage this will be okay, and as long as you never overrun the cache on each drive it might be okay; although the resilver might shake a few gremlins loose. Look up threads from the poster @deafen though as he's catalogued his experience with SMR drives over a couple posts.

That's interesting... I actually had similar issues with my SMRs but only when one is getting ready to fail...
Initial indications (even before offline/unreadable sector errors would be huge latencies across all drive) and consume huge quantities of resources in machines (I've seen them gobble up and run my CPUs at 100% and peg out ram usage of 192GB of ram). The worst part is trying to track down the troubled drive when all indicators show everything as healthy... IIRC, it's actually why I moved from the 5TB drives to the 6 (they were same price at the time and I was trying to eliminate the cause of the slow down... Replacing 1, resilvering, check performance, replace the next... Til they were all replaced and the system ran fine...

Don't get me wrong, they CAN be fantastic drives, and generally speaking perform just as well, but at half the price... But man, when you hit a snag with one, it's like unlocking all the cages for all the Gremlins everywhere at once...