8.3.0 -> 8.3.1 GUI Update Fails: "run_interrupt_driven_hooks: still waiting"

[mattlach]

Hey all,

I had a 64bit 8.3.0 install running in VMWeare ESXi 5.1. I decided to upgrade it to 8.3.1 today using the GUI Method.

System went down for reboot.

I was watching the console during the operation. Now it is stuck here, with constant 75% CPU load:

Any idea what may have gone wrong?

I've seen previous posts that this is a 1394 problem and to disable it in bios, but the previous version worked just fine, so I don't think it's related.

Any thoughts? I've been just leaving it on, just in case it figures itself out and stops waiting, but I would appreciate any thoughts / suggestions on what to do next!

Thanks,
Matt

 

[mattlach]

So I tried to boot from the 8.3.1 cd image to fix this.

Same error as above. So it looks like something new in 8.3.1 that wasnt there in 8.3.0 really doesn't like my system...

 

[mattlach]

So, now I am REALLY scratching my head.

Just to make sure it wasn't a 1394 problem as suggested in another thread, I shut down all my VM's and rebooted ESXi, and entered the server BIOS.

I found what I expected. I always disable any integrated system component I won't use in order to minimize compatibility problems and to avoid wasting system resources. 1394 was already disabled.

I booted back up again. The FreeNAS VM started on boot (like I have it configured to do) and to my surprise, this time it went further. It displayed a message about successfully updating databases, and then rebooted again, and this time booted back up like it used to, and everything works...


Except, it still lists the build as: FreeNAS-8.3.0-RELEASE-x64 (r12701M)

So somehow it must have reverted to 8.3.0 after failing with 8.3.1, but I don't think I did anything to make it do so.

Does anyone have any insight here?

Thanks,
Matt

 

[cyberjock]

I don't have any recommendations. There's 2 partitions on the boot disk for FreeNAS. Typically the current and previous version. Since you are still booting up and seeing what should be the previous version I'd wager the update didn't finish.

Secondly, there are plenty of threads, including http://forums.freenas.org/showthrea...oduction-as-a-Virtual-Machine!&highlight=esxi that explain why running FreeNAS as a virtual machine is just asking for trouble. Everything works great...until it doesn't. And when it doesn't, data is often lost even with redundancy. I've only seen that error in ESXi when I did some experimenting with it and I had no fix for it. It seems to be somehow related to ESXi because as soon as I pulled ESXi and booted from the same hardware without ESXi the problems went away.

 

[mattlach]

I don't have any recommendations. There's 2 partitions on the boot disk for FreeNAS. Typically the current and previous version. Since you are still booting up and seeing what should be the previous version I'd wager the update didn't finish.

Secondly, there are plenty of threads, including http://forums.freenas.org/showthrea...oduction-as-a-Virtual-Machine!&highlight=esxi that explain why running FreeNAS as a virtual machine is just asking for trouble. Everything works great...until it doesn't. And when it doesn't, data is often lost even with redundancy. I've only seen that error in ESXi when I did some experimenting with it and I had no fix for it. It seems to be somehow related to ESXi because as soon as I pulled ESXi and booted from the same hardware without ESXi the problems went away.

The guy in the linked post makes some good points, but...

1. through 3.:
Yes, anyone trying to run ZFS on virtual file systems other than for testing and playing around is an idiot. This is almost guaranteed to end badly.

4.
His concern with direct I/O passthrough seems unwarranted. I pass through a storage controller known to work beautifully passed through in the ESXi Whitebox community (IBM M1015 SAS controller with IT firmware), on a very popular motherboard in the ESXi whitebox community (Gigabyte GA990FX-UD3, presumably due to its low bundle price at Microcenter), so I am not worried here.

5.
This is only a problem if you are an idiot and disregard the resourcing guide in the FreeNAS wiki. My ESXi server has 32GB of ram, 22GB of which is dedicated to FreeNAS.

6.
I agree with him here. Relying on storage for your VM host that is hosted on a guest inside that same host sounds like a terrible idea. But if you aren't a moron, you won't be doing this, and as such it isn't a problem :p

The whole point of view that "someone might do something stupid if they run it in a VM, because of this no one should run it in a VM" is really rather silly.

To me the biggest issue of running FreeNAS as a VM, when everything is configured like someone who knows what they are doing might do, is that there are fewer of us than there are bare metal folks, and as such ESXi systems don't get as much exposure during the betas and more incompatibilities might pop up as a result. this is my theory of what is going on here, but you never know.

From a power and server cost perspective, I simply can't justify running a separate server for every OS I need to run, so VM's are a necessity. Lots of people run ESXi and FreeNAS in the configuration I do, and I was hoping that some of them who do, might chime in to this thread if they are having the same issue.

 

[mattlach]

Actually, some further searching using keywords I didn't try the first time may have led me to the solution:

http://forums.freenas.org/archive/index.php/t-6166.html

I am going to try this when I get home, and post back here for posterity.

EDIT:

Actually, come to think of it, I vaguely remember having to do something like this when I first installed 8.3.0 as well. It's all coming back to me now.

If I boot without the controller forwarded, then load my stored config, and reboot with the controller forwarded again, it would probably work.

It's too bad FreeNAS doesn't remember loader.conf during a gui upgrade.

 

[paleoN]

Ticket #2056

 

[cyberjock]

Not to dismiss your comments mattlach, but he has a boatload of experience both in and out of this forum with FreeNAS and VMs. He and I have seen so many configurations go horribly wrong with ESXi it isn't funny. Not to say its not possible, but the people that made ZFS recommended against virtualizing, and who are we to think we know better. Is it doable? Absolutely. Is it reliable? Depends on alot of factors.

I think jgreco has it right when he makes comments like the ones you are disagreeing with. Unless you have a heavy background in VMs and whatnot, you will probably frequent the forums. His statement is meant to scare off people(and save their data too) that actually need a forum for support. For those that can download FreeNAS and get it working with no forum support they may have the necessary experience. But I will tell you PLENTY of people have lost their data because they chose to not follow jgrecos recommendations. 99% of his recommendations come from blood, sweat, and tears from forum users that have had to learn ESXi lessons in the hardest ways.

I have seen people lose data in about every conceivable way because of virtualization, but I have yet to see someone lose data that used pci passthrough.