Please Help!! FreeNAS 11 not booting

[zohan]

Hi All,

Please need your advice, help and opinion,

I have a FreeNAS server running on ESXi Vmware with 3 storage pools.
One of the pools failed and the FreeNas is not booting and hangs on GRUB loading .....

I read a few posts here but I didn't find a solvency solution except one to install a fresh VM with same FreeNAS 11 version and then to attach the disks and after to import them.

Any ideas, please? Are reattaching and importing will make any damage?

Thank you all.

 

[jgreco]

Hi All,

Please need your advice, help and opinion,

I have a FreeNAS server running on ESXi Vmware with 3 storage pools.
One of the pools failed and the FreeNas is not booting and hangs on GRUB loading .....

I read a few posts here but I didn't find a solvency solution except one to install a fresh VM with same FreeNAS 11 version and then to attach the disks and after to import them.

Any ideas, please? Are reattaching and importing will make any damage?

Thank you all.

Welcome to the forums.

Sorry to hear you're having trouble. Please take a few moments to review the Forum Rules, conveniently linked at the top of every page in red, and pay particular attention to the section on how to formulate a useful problem report, especially including a detailed description of your physical virtualization hardware, along with a description of your virtual hardware, what your PCIe passthru configuration is, and what sort of controller you've used for passthru.

You've basically given no one anything to work with, so the responses will tend to be random guesses rather than anything useful.

 

[zohan]

Sure. Hope these details provide the full picture

DELL PowerEdge R730XD
CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
MEMORY: 127.91 GB

WITH PERC H730 MiN with controller policy of Adaptive Read Ahead and write back
at the enclosure

VD No 1:
2 X 128G SAS disks in raid 1 for the system, the FreeNAS is running here

VS No 2:

12 X 8T nearline SAS in RAID 6

ON PERC H800 with controller policy of Adaptive Read Ahead and write back

attache 2 DELL MD1200

MD 1200 No 1 configured as VD No 3

12 X 8T nearline SAS in RAID 6

MD 1200 No 1 configured as VD No 4

12 X 10T in RAID 5 this is the faulted drive

The system is running DellEMC-ESXi-6.5.0-5310538-A05 (Dell)


The system is running and working well for 5 years until the VD crash

 

[jgreco]

You should not be using a PERC H730, a PERC H800, or any RAID 5 or RAID 6.

The system is running and working well for 5 years until the VD crash

Yeah, I imagine so. Once a failure occurs and ZFS has nowhere to pull redundancy from, you're very toasty and probably done for the day.

Please review

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[zohan]

I wish the mentioned above KB existed in 2017 when this NAS was installed:-(
what about the 2 others pools that are OK?
if I will install a fresh FreeNAS and try to import the 2 Healthy Vd's it could decrease the chance to operate the NAS again?

any other solutions or ideas?

 

[jgreco]

I wish the mentioned above KB existed in 2017 when this NAS was installed:-(

There's been a warning about this in the manual for quite some time, though I am happy to concede it came as a late entry to the show, possibly in response to my somewhat frequent criticisms of the Hubbard era "they should be able to use whatever they like" attitude, which, while commendable in its desire to make it possible to use whatever you like, flies in the face of practical reality.

what about the 2 others pools that are OK?

So by "VD" you mean virtual disk? It sounded like it, but it was unclear, as there is a ZFS construct referred to as a "vdev". As long as there has been no data loss or corruption on your virtual disks, they are likely fine. However, ZFS cannot offer them data protection services. The hardware RAID controller hides the parity disk(s) from ZFS, making that data source unavailable to the ZFS checksum and correction algorithms. ZFS is probably okay with working on your virtual disks until something corruption-worthy happens.

I should point out that under ESXi that it is possible to be running these virtual disks as big vmdk files on top of VMFS, which is generally a bad idea. The hypervisor's device driver layer is considered to be reliable and relatively safe, but the problem is that if your LSI VD blows up or starts on fire, you may risk losing the entire thing.

Passing the entire controller through to TrueNAS and giving TrueNAS an LSI VD to manage isn't much better. There are some driver issues that might cause problems, but the biggest terror is that your VD fails and there's no way for ZFS to recover.

In the future, when you rebuild these pools, you need to use an LSI HBA and allow ZFS to manage the RAID aspect for you. The LSI HBA needs to be passed thru to the TrueNAS VM with PCIe passthru. See the following article for more details.

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

if I will install a fresh FreeNAS and try to import the 2 Healthy Vd's it could decrease the chance to operate the NAS again?

Should be fine.

 

[zenon1823]

I setup my first FreeNAS in 2018 and there was ample if not copious amounts of warnings at that time about the risks of RAID cards and esxi virtual disks as pool members. I followed the path jgreco described of using pcie passthrough of an ITmode flashed HBA. At the time i was getting warned against that by some but felt i was sufficiently comfortable with my destiny. Its been rock solid and I thought I'd chime in to support that in my experience this is stable and safe WHEN done properly. But you wont or didnt find as much community support since I believe its a minority of users who have done it. Its truly unfortunate you came on board before it was so well warned and documented about :( - I feel for you.

With any luck you will be able to import the healthy VD's and salvage the remaining data. As a side note if you want to use those MD1200 shelves with FreeNAS safely - toss the H800 and get an H200E or similar HBA - reflash it with IT firmware and connect the MD1200's to it and pass the h200e card through to the esxi vm that is the freenas OS and build your pools. (NOTE you will not be able to connect the MD1200's to the IT-mode H200E and import the existing pools. You will need to move your data elsewhere and build new pools with the drives in the MD1200's)

As a side note you will also see better pool performance once you replace that h800 with a simple HBA. You will also need an internal HBA like an h200 to pass through your former VS2 drives.

Moral of the story - H800 H730 etc should only be used for drives for your esxi data stores for VM's. Not for zfs datastores as JBOD or VD's.

How do i know the details of what your doing so well? I'm also running the MD1200 exactly how ive described. I have some build documentation on the flashing process etc if you need help I can pass along what i have. But there is lots of posts and guides talking about this you got a fair bit of reading to do i know i spent alot of time figuring this out.

 

[zohan]

Thank you jgreco and zenon1823 for your important replies. I hope this post will succeed in helping others in the same or similar situation and prevent others to get into this situation, after all, that is what this forum is for, helping others

In the small update, I copied the vmdk file that runs the system of FreeNAS without attaching the virtual disk with the RAW disks that hold the storage pool and the system came up. so the problem with the "GRUB loading ..... " is because of the damaged pool.

 

[zenon1823]

Thank you jgreco and zenon1823 for your important replies. I hope this post will succeed in helping others in the same or similar situation and prevent others to get into this situation, after all, that is what this forum is for, helping others

For sure we all learn, live and fail together :)

In the small update, I copied the vmdk file that runs the system of FreeNAS without attaching the virtual disk with the RAW disks that hold the storage pool and the system came up. so the problem with the "GRUB loading ..... " is because of the damaged pool.

By any curious chance did that pool have your system dataset?

 

[zohan]

Yes this pool had the system database

 

[zohan]

UPDATE: I removed the damaged vdisk/vmdk raw disk from the FreeNAS guest and the FreeNAS booted successfully and I can see the 2 other storage pools.