Advice for NAS critical failure victim

[AlexF]

Hi Experts,
I've been running FreeNAS 12.x on my two HP Microservers, N36L and N40L for a long time (one backup for the other, both on UPS) - only powered-up on per-need basis - typically once a month, last time accessed, probably a month ago, all without fail. Each has 4x 3TB WD Red 2x USB Boot (ZFS Pool) drives, storing valuable domestic records, personal information, photos, music, etc.. I haven't back-up to standalone external USB dsk for a long time, so, it may/no be useful. Sigh...

Today, both servers failed to boot-up - (front) "hp" logo lights being red on both! (Normally, blue.) Attached VGA monitor shows no signal at power-up, bit built-in and PCI card Ethernet lights are blinking as normal, so I suspect not-FreeNAS related. Google says it may be motherboard or hardware-related fail, but, why concurrently - coincidence, really? I'd be most interested in experience/advice to let me boot at least once...

Otherwise, (sigh...) if I purchase a new computer, install the 4x disks and 2x USB Boot drives, what are my chances of FreeNAS recovery?
R's, AlexF

 

[Heracles]

Hi,

lets re-assure you first about this :

what are my chances of FreeNAS recovery?

They are excellent if your hardware was adequate for TrueNAS and properly configured.

What you describe indeed looks like catastrophic electronic failure, but failure way before the hard drives even come in to action. Motherboard, RAM, CPU, ... no matter which one of these failed, probability for both of your pools ending up corrupted is close to zero. Actually, they are probably both fine as of now.

So first thing will be to stop touching one of these two guys and focus only on the second. Do not even try to boot up the second. Its hard drives are stopped and cold now, so lets keep them as they are. To spin them up and down and up and down will only stress them and increase the risk of a failure of a different kind.

Because it is the hardware that is main suspect, I would disconnect all the data drives from the system you will try to recover. Then, start to boot it either with TrueNAS or anything else. If nothing can boot it, you will probably have your answer about this guy.

If something can boot it, try to test the hardware with tools like memtest+. See if you can detect anything wrong with the hardware.

Should you give up on booting the first guy, still do not touch the second and try to build yourself a new TrueNAS server. Once done, re-plug the data drives in that new guy and import your pool. Once you have access to your data, you can try to do another backup to another functional system or start working out the second system.

For more precise advice, it would be good for you to post the complete hardware inventory and configuration used on both servers.

Last note is that if both of your servers were built without proper consideration for TrueNAS requirements (like not using hardware raid but you chose to use such hardware raid card in both of them), then unfortunately simultaneous failure of both is much more probable...

 

[AlexF]

Thank you for great advice! I know this isn’t right thread, but is Microserver Gen 10+ natural source for off-the-shelf host? Any better advice?

 

[Heracles]

I have no experience with HP servers. Using Dell over here.

The most important thing to confirm is how drives are handled and presented to TrueNAS. TrueNAS must have direct physical access to all its drives. Hardware raid cards are very problematic even in JBOD mode.

So how your disks are connected to the server and how are they presented to TrueNAS ?

You can read this about hardware raid and the problems they represent.

 

[AlexF]

Hello!
Thank you Heracles, for your time and expertise.
Good news (for me) is that simply replacing system/coin battery ("CR2032") worked - both servers now boot-up - (front) "hp" logo lights are blue; and attached VGA screen shows normal boot.
Now that cause is known, ... a host failing to show anything on attached monitor (because of a battery) is simply inexcusable.
And, there's a powerful and expensive lesson for me - I'll now have to pay for offsite (probably, cloud) storage to prevent/preempt recurrence.
R's, AlexF

 

[danb35]

is Microserver Gen 10+ natural source for off-the-shelf host?

I have one and like it, though it seems prices have skyrocketed since I bought mine and I'm not using it as a NAS. The fact that it has only four drive bays is a drawback as a NAS, though it's probably the smallest hardware that's a proper server. I'd use a USB-NVMe adapter, put that on the internal USB port, and use it for the boot device.