Virtualbox + Windows 7 = Multiple hangs / resets

[andyclimb]

DISCLAIMER: I posted this thread on the virtual box forums and basically got told to take a hike. So I hope its ok that I'm putting it here. I've made comments about this before but now I have a bit more info.

Hi,

I'm new to virtual box, and this is my first post!

Here is the background:
I'm running freenas, the latest beta 9.2.1.6 on both an HP n40L and a new build xeon E3 1220 V3 with a super micro X10SL7-F mobo, 16gb ECC ram. Freenas now comes with a virtual box template version 4.3.10_OSE which I have been experimenting with. My goal is to run windows 7 as a VM with iTunes to serve music to a bunch of airport expresses that i have in the house. My n40L works flawlessly, in this set up and I had windows running absolutely fine as a VM in a virtual box jail and playing music. The problem is that 1) this machine is very slow, it works but it is slow 2) it is a test-bed system so I don't want to depend on it... For this reasons I want to run my VMs on my new xeon machine!

The problem...

Windows 7 as a VM in Virtualbox seems to be highly unstable, especially when doing anything with iTunes. Basically at some point, often/always when using iTunes the Windows guest crashes... sometimes this goes to an aborted state in virtual box GUI but not always. If i try ACPI shutdown or Reset then the machines starts to shutdown but the progress indicator gets stuck at 28% in the virtual box GUI. At this point my only options are to disable the virtual box jail and re-enable it in the freenas GUI. The slightly worrying thing is that at any point - after windows has hung my entire freenas machine can just reboot! most often it occurs when disabling the jail. Someone has said that this could be due to an incomplete shutdown of a VIMAGE stack!

Looking through my vbox-logs.... all 3 have this error that causes an aborted machine....

08:40:37.934868 AIOMgr: I/O manager 0x00000808ff9920 encountered a critical error (rc=VERR_FILE_AIO_NO_REQUEST) during operation. Falling back to failsafe mode. Expect reduced performance
08:40:37.935169 AIOMgr: Error happened in /wrkdirs/usr/ports/emulators/virtualbox-ose/work/VirtualBox-4.3.10/src/VBox/VMM/VMMR3/PDMAsyncCompletionFileNormal.cpp:(1664){int pdmacFileAioMgrNormal(RTTHREADINT*, void*)}
08:40:37.935181 AIOMgr: Please contact the product vendor
08:41:38.191679 AHCI#0P0: Cancelled task 3
08:41:38.191957 AHCI#0P0: Cancelled task 6
08:41:38.191964 AHCI#0P0: Cancelled task 8
08:41:38.191980 AHCI#0P0: Cancelled task 10
08:41:38.191986 AHCI#0P0: Cancelled task 11 etc etc etc......



I have tried the following things and nothing seems to work.

1) new windows installs - i've tried about 6 including 32bit and 64bit
2) enabling and disabling... ALL hardware virtualisation, including VT-x/AMD-V, Nested Paging, Large Pages, VT-x unrestricted execution and VT-x VPID (Intel only)
new install of freenas
3) The vdi file location: 1) on SSD formatted to ZFS and mounted in the jail 2) within the jail itself which is on a ZFS pool consisting of a SSD mirror.. on my n40L the jails were located on a single spinning disk formatted with ZFS.

As I'm trying to get windows to run iTunes... that has been my main objective so I'm not sure how often windows does/would/will crash if I'm not just using iTunes... as it is quite an important server and i've not tried installing word. My Virtualbox is also running pfsense also using VIMAGE stack and seems to be stable and working without problems.


Is anyone able to shed some light on this?

Any ideas or help would be appreciated.

Thanks

Andrew

 

[cyberjock]

Well, if you read my thread on VBox, you need 8GB of RAM plus whatever RAM you use for the VM. So using 8GB of RAM with any other VMs is a major "fail" in my book.

If you aren't doing mirrored RAIDs or running off of SSDs you can expect poor performance. In fact, some programs may crash because the disk latency may be so high it starts misbehaving. Which leads me into my next comment....

Why on God's green earth are you doing 2x120GB SSDs on your E3 1220 system, but have only 16GB of RAM! If they are your boot drives that's a sad disgrace for those disks as that's total overkill for the cost of a USB stick. If they are your ZIL or L2ARC you are also failing as you should have MUCH more RAM before considering those. But, if you repurpose them as a zpool for a jail running Windows you might see some problems go away....

Other than those few ideas I don't have much more to offer. I've run Windows 7 on my mini and it works fine for me. That's all I can say. :(

 

[andyclimb]

1) I've got 16gb ecc ram so point 1 doesn't really apply. My test system has 8gb but I've not got any problems whatsoever with that one. Runs windows stable as a rock!

2) the SSDs are only there to run the jails.

Windows is super fast so that's not the issue. But somewhere there something that is not happy. Ironically on the n40l with its 8gb ram I had it running of a single 250gb hdd and it was fine!

I have done extensive reading on this forum and basically followed your advice!

Thanks for chipping in though.


Sent from my iPhone using Tapatalk

 

[cyberjock]

Holy! I think I just wet my pants! You mean you did everything right! Someone give this guy a medal! You know how rare it is that people did stuff right! You have impressed me sir!

So, with that out of the way...

I don't have any immediate guesses since you threw my first post back at me. ;)

Did you try searching for the error message in Google? I found things like https://www.virtualbox.org/ticket/12648 that make me wonder what is going on with your setup. It cold be that the N40L's controller or BIOS just doesn't quite work right. Only a guess though.

Are your SATA controllers set to AHCI mode in your BIOS? Do you even have the option of setting AHCI mode?

 

[andyclimb]

Well thank you! I was nervous that you were going to destroy my set up there.. even though I've followed your advice as far as I can tell. I've even got my freenas db backing up daily to my spool (thanks for that script), scrubs scheduled monthly, and snapshots set up with appropriate expiry dates.. so hopefully I'm safe.. I'm running RAIDZ2 as well...

so on to business...

I made a post earlier with the CPU PARITY errors... which only occur when I'm virtualising... and as far as i can tell only when I'm running iTunes in my VM.. So far today I'm running windows 7, ubuntu and pfsense in virtual box and i've not had a single problem BUT i've not opened iTunes... Somehow this all comes back to iTunes being the culprit... I'm nervous to test it too much as I'm not home and won't be till the end of july... and these hard reboots have a tendency to corrupt the usb stick and i need to re-image and restore the config.. so if that happens I'm totally screwed....anyway I digress..

I've done extensive googling... to the point where the only posts I get back are my own ones on this forum and the virtual box one... I came across the ticket you mentioned. and yes it it the same message, but I only have the one virtual disk. The symptoms are the same, guest hangs... virtual box process needs reseting.. The more complicated thing is the hard reboots that then occur....someone else has mentioned to me that an inappropriately terminated / crashed VIMAGE networking stack with a bridge interface causes / can cause a kernel panic in freebsd and then a hard reboot... which is what seems to happen to me. And so far this only happens when I'm using iTunes... i have yet to have it happen just using windows... although that is hard as the only reason for me having the VM is for iTunes. I'm totally stumped and rather annoyed! I've put a 100MB/s cap on the speed of the virtual drive to see if that makes any difference...but like I said it going to be a while before i load up itunes and test it as I'm not here... and don't want the whole machine to fail a boot...

when I'm back I'll enable the kernel debug setting in freenas, and give iTunes a test and see what happens..

I'd be interested if anyone else can recreate this / has similar issues....

i guess my ultimate test is to install hypervisor and run a windows VM in there and see if i get the crashes... at least that means it is not my hardware / or is my hardware...

all of these problems do not happen on the N40L... that is fine... it is my new fancy xeon E3 1220 set up that is plagued by these crashes.. despite the ECC, despite lots of RAM, despite SSDs...

When you ask about my SATA controllers do you mean in virtual box or on my actual motherboard.. in virtual box they are AHCI, on my mobo the SSDs are on the mobo sata controllers which will be AHCI... my pool is on a flashed LSI SAS controller that comes with the X10SL7-F... its in the direct pass through mode... I followed all the advice on the forum on that one too.. That all seems to be working fine.

 

[andyclimb]

Found this rather interesting thread which seems to correlate to my behaviour!

https://forums.virtualbox.org/viewtopic.php?f=7&t=57842

although the devs say its fixed... there are still people getting the same sorts of crashes!

How difficult would it be to get virtual box.. 4.2.18 on freenas?

 

[cyberjock]

What I think would be the ultimate test would be to take that virtualbox VM and move it to your desktop and run it in virtualbox on your desktop and see if it crashes. ;)

I really don't have any recommendation. I'm pretty stumped. You've done everything right to and there's nothing unreasonable about what you are expecting. I just don't know what to say.

 

[andyclimb]

I'm stumped too... Right now I'm just running it without iTunes running and touch wood it is stable as a rock and running just fine... 3 days now.. although as windows does it has constantly installed updates... if iTunes is the culprit it begs the question what the hell is it doing that causes it to hang windows, hang virtual box and reboot my server!

 

[cyberjock]

I'm stumped too... Right now I'm just running it without iTunes running and touch wood it is stable as a rock and running just fine... 3 days now.. although as windows does it has constantly installed updates... if iTunes is the culprit it begs the question what the hell is it doing that causes it to hang windows, hang virtual box and reboot my server!

I don't know. I know that iTunes is a horribly written application and needs to die in a fire.

 

[Ericloewe]

I don't know. I know that iTunes is a horribly written application and needs to die in a fire.

Same could be said of everything Apple writes for Windows. More fuel for the "OS X is more stable than Windows" FUD fire...

I believe iTunes installs all sorts of crap related to networking and USB drivers for iDevices. I imagine either could be conflicting with something, like the VirtualBox guest additions (particularly the USB stuff). Of course, this doesn't explain why some have been successful...

 

[andyclimb]

That's given me an idea though. I guess I could go the mac virtualisation route! After all. I'm only after iTunes airplay!


Sent from my iPhone using Tapatalk

 

[cyberjock]

No, iTunes being horribly written isn't FUD at all. Even Apple has admitted it and is actually trying to kill iTunes. Last I read Apple is experimenting with an "always online" type situation where everything is pretty much handled on the web like what Android does. You go log into a webpage and do all of your manipulations on a webpage and then the changes sync to your iWhatever over the internet. iTunes started out as an amazing software package and has just gotten worse over time.

History lesson:

Apple didn't actually create iTunes. Apple bought iTunes and some other MP3 player related patents from another company 10+ years ago. Apple has hated it and basically been taking the proverbial "car" and just kept adding accessories that were never conceived of in the original code. Unfortunately the original code wasn't well documented and was a nightmare to "buy" since everything tied in to everything else to keep the install package small. Apple didn't try to mess with it and left as much of the original code in there as possible(which is why the install is like 100MB or something now).

Anyway, some of the dll's are still stamped with the date, time and original company name because they haven't been recompiled with Apple's trademarks since then. ;)

So it's definitely not FUD when I say iTunes is a POS software. It is a fact and Apple knows it. They've admitted it and they have been working on getting rid of iTunes for more than 2 years. The real questions have revolved around how to continue to provide support for iWhatevers that don't have wifi. Apple has taken pride in letting iTunes be backwards compatible quite a ways and allowing syncing via the wire.

 

[Ericloewe]

No, iTunes being horribly written isn't FUD at all. Even Apple has admitted it and is actually trying to kill iTunes. Last I read Apple is experimenting with an "always online" type situation where everything is pretty much handled on the web like what Android does. You go log into a webpage and do all of your manipulations on a webpage and then the changes sync to your iWhatever over the internet. iTunes started out as an amazing software package and has just gotten worse over time.

History lesson:

Apple didn't actually create iTunes. Apple bought iTunes and some other MP3 player related patents from another company 10+ years ago. Apple has hated it and basically been taking the proverbial "car" and just kept adding accessories that were never conceived of in the original code. Unfortunately the original code wasn't well documented and was a nightmare to "buy" since everything tied in to everything else to keep the install package small. Apple didn't try to mess with it and left as much of the original code in there as possible(which is why the install is like 100MB or something now).

Anyway, some of the dll's are still stamped with the date, time and original company name because they haven't been recompiled with Apple's trademarks since then. ;)

So it's definitely not FUD when I say iTunes is a POS software. It is a fact and Apple knows it. They've admitted it and they have been working on getting rid of iTunes for more than 2 years. The real questions have revolved around how to continue to provide support for iWhatevers that don't have wifi. Apple has taken pride in letting iTunes be backwards compatible quite a ways and allowing syncing via the wire.

No, no, you got me wrong: iTunes is the very reason I never bought an iPod. The mere thought of installing that piece of crap kept me away. I'd rather install Norton and McAfee simultaneously than mess with iTunes.
I meant they're probably using iTunes' nasty behavior as FUD against Windows.

 

[cyberjock]

Ah! Ok. My mistake.

 

[andyclimb]

ha... well my sneaky plan also failed.... got the same error when using OSX...

Code:

02:16:54.081160 Changing the VM state from 'SUSPENDED' to 'RESUMING'.
02:16:54.081251 Changing the VM state from 'RESUMING' to 'RUNNING'.
02:21:31.362585 AIOMgr: I/O manager 0x00000808e33b20 encountered a critical error (rc=VERR_FILE_AIO_NO_REQUEST) during operation. Falling back to failsafe mode. Expect reduced performance
02:21:31.362615 AIOMgr: Error happened in /wrkdirs/usr/ports/emulators/virtualbox-ose/work/VirtualBox-4.3.10/src/VBox/VMM/VMMR3/PDMAsyncCompletionFileNormal.cpp:(1664){int pdmacFileAioMgrNormal(RTTHREADINT*, void*)}
02:21:31.362639 AIOMgr: Please contact the product vendor
02:22:01.364390 AHCI#0P0: Cancelled task 10
02:22:01.364425 AHCI#0: Port 0 reset
02:22:31.368833 AHCI#0P0: Cancelled task 13
02:22:31.368875 AHCI#0P0: Cancelled task 14
02:22:31.368884 AHCI#0P0: Cancelled task 15
02:22:31.368891 AHCI#0P0: Cancelled task 16
02:22:31.368900 AHCI#0: Port 0 reset
02:23:01.372525 AHCI#0P0: Cancelled task 13
02:23:01.372554 AHCI#0: Port 0 reset
02:23:31.375544 AHCI#0P0: Cancelled task 14
02:23:31.375600 AHCI#0: Port 0 reset
02:24:01.379376 AHCI#0P0: Cancelled task 15
02:24:01.379418 AHCI#0: Port 0 reset
02:24:31.382768 AHCI#0P0: Cancelled task 16
02:24:31.382805 AHCI#0: Port 0 reset
02:25:01.386624 AHCI#0P0: Cancelled task 13
02:25:01.386653 AHCI#0: Port 0 reset
02:25:31.390469 AHCI#0P0: Cancelled task 14
02:25:31.390496 AHCI#0: Port 0 reset
02:26:01.394339 AHCI#0P0: Cancelled task 15
02:26:01.394373 AHCI#0: Port 0 reset
02:26:31.397658 AHCI#0P0: Cancelled task 16
02:26:31.397707 AHCI#0: Port 0 reset
02:27:01.401564 AHCI#0P0: Cancelled task 13
02:27:01.401593 AHCI#0: Port 0 reset
02:27:31.405379 AHCI#0P0: Cancelled task 14
02:27:31.405415 AHCI#0: Port 0 reset
02:28:01.408757 AHCI#0P0: Cancelled task 15
02:28:01.408790 AHCI#0: Port 0 reset
02:28:31.412583 AHCI#0P0: Cancelled task 16
02:28:31.412621 AHCI#0: Port 0 reset
02:29:01.416390 AHCI#0P0: Cancelled task 13
02:29:01.416417 AHCI#0: Port 0 reset
02:29:31.419787 AHCI#0P0: Cancelled task 14
02:29:31.419828 AHCI#0: Port 0 reset
02:30:01.423429 AHCI#0P0: Cancelled task 15
02:30:01.423470 AHCI#0: Port 0 reset
02:30:31.427425 AHCI#0P0: Cancelled task 16
02:30:31.427466 AHCI#0: Port 0 reset
02:31:01.430615 AHCI#0P0: Cancelled task 13
02:31:01.430645 AHCI#0: Port 0 reset
02:31:04.320810 Changing the VM state from 'RUNNING' to 'RESETTING'.
02:31:04.321151 CPUM: SetGuestCpuIdFeature: Enabled APIC
02:31:04.321175 PIT: mode=3 count=0x10000 (65536) - 18.20 Hz (ch=0)
02:31:04.321553 PDMR3Reset: after    0 ms, 1 loops: 1 async tasks - ahci/0
02:31:05.360012 ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={480b372c-c0b5-4c23-9bd7-dcbb85b1594c} aComponent={Display} aText={Could not take a screenshot (VERR_NOT_SUPPORTED)}, preserve=false
02:31:05.360041 PDMR3Reset: after  1039 ms, 2 loops: 1 async tasks - ahci/0
02:31:09.351513 ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={480b372c-c0b5-4c23-9bd7-dcbb85b1594c} aComponent={Display} aText={Could not take a screenshot (VERR_NOT_SUPPORTED)}, preserve=false
02:31:09.351546 PDMR3Reset: after  5030 ms, 4 loops: 1 async tasks - ahci/0
02:31:09.353727 PDMR3Reset: after  5032 ms, 5 loops: 1 async tasks - ahci/0
02:31:13.359035 ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={480b372c-c0b5-4c23-9bd7-dcbb85b1594c} aComponent={Display} aText={Could not take a screenshot (VERR_NOT_SUPPORTED)}, preserve=false
02:31:13.359077 PDMR3Reset: after  9038 ms, 6 loops: 1 async tasks - ahci/0
02:31:17.082389 VBVA: VRDP acceleration has been disabled.
02:31:17.393924 ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={480b372c-c0b5-4c23-9bd7-dcbb85b1594c} aComponent={Display} aText={Could not take a screenshot (VERR_NOT_SUPPORTED)}, preserve=false
02:31:20.359187 ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={480b372c-c0b5-4c23-9bd7-dcbb85b1594c} aComponent={Display} aText={Could not take a screenshot (VERR_NOT_SUPPORTED)}, preserve=false
02:31:20.359238 PDMR3Reset: after 16038 ms, 10 loops: 1 async tasks - ahci/0
02:32:05.221905 PDMR3Reset: after 60901 ms, 12 loops: 1 async tasks - ahci/0
02:32:05.517656 ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={480b372c-c0b5-4c23-9bd7-dcbb85b1594c} aComponent={Display} aText={Could not take a screenshot (VERR_NOT_SUPPORTED)}, preserve=false
02:32:05.517689 PDMR3Reset: after 61196 ms, 13 loops: 1 async tasks - ahci/0
02:32:05.520095 PDMR3Reset: after 61199 ms, 14 loops: 1 async tasks - ahci/0
02:32:24.446781 ERROR [COM]: aRC=VBOX_E_IPRT_ERROR (0x80bb0005) aIID={480b372c-c0b5-4c23-9bd7-dcbb85b1594c} aComponent={Display} aText={Could not take a screenshot (VERR_NOT_SUPPORTED)}, preserve=false
02:32:24.446829 PDMR3Reset: after 80125 ms, 15 loops: 1 async tasks - ahci/0
02:32:24.449322 PDMR3Reset: after 80128 ms, 16 loops: 1 async tasks - ahci/0
02:32:30.312669 Console::powerDown(): A request to power off the VM has been issued (mMachineState=Stopping, InUninit=0)
02:34:38.836573 ERROR [COM]: aRC=VBOX_E_INVALID_VM_STATE (0x80bb0002) aIID={8ab7c520-2442-4b66-8d74-4ff1e195d2b6} aComponent={Console} aText={The virtual machine is being powered down}, preserve=false
02:48:28.499067 ERROR [COM]: aRC=VBOX_E_INVALID_VM_STATE (0x80bb0002) aIID={8ab7c520-2442-4b66-8d74-4ff1e195d2b6} aComponent={Console} aText={The virtual machine is being powered down}, preserve=false
02:48:34.633272 ERROR [COM]: aRC=VBOX_E_INVALID_VM_STATE (0x80bb0002) aIID={8ab7c520-2442-4b66-8d74-4ff1e195d2b6} aComponent={Console} aText={Invalid machine state: Stopping}, preserve=false

So the next question is Virtual box is running version... 4.3.10_OSE... now the latest version is 4.3.12... so would there be any milage in upgrading... I think there might be a bug here... but as I'm not using "on official build" I can bugger of!

 

[cyberjock]

Well, the kernel driver files need to stay in sync with the Virtualbox installed version. So if you choose to upgrade 4.3.12 in your jail the box may or may function. If you are up for experimenting you are welcome to give it a go.

 

[Wonderjacky]

Hello,

Did you manage to upgrade to Virtualbox 4.3.12?
Did you register a Freenas ticket for this matter?
I would like to buy the same processor as yours (with a MBD X10 SSL-F) and I need to have iTunes running inside a Virtualbox too.
I'm anxious it comes from an Hardware incompatibility.

Thanks in advance.

François


Sent from my iPhone using Tapatalk

 

[solarisguy]

Virtualbox 4.3.12 is now included in FreeNAS 9.2.1.6-RC
http://forums.freenas.org/index.php?threads/freenas-9-2-1-6-rc-is-now-available.21711/

 

[andyclimb]

Not yet. I'm travelling and won't be able to upgrade till end of July. Unfortunately. But yes I'm very keen to try 4.3.12. The change log does indicate some stability updates that might help me!


Sent from my iPhone using Tapatalk

 

[andyclimb]

I thought I'd post an update. Came back form holiday, updated to 9.2.1.6-RC , created new jail with .12 version of virtualbox. Opened my Windows 7 VM and I'm not on day 3 of running iTunes with 0 crashes. So it looks the new version has fixed the problem!! great..

Now if anyone has any ideas of how to upgrade the virtual box jail without recreating the whole thing and going the the user permissions, added storage thing. For some reason the /home/ folder is no longer visible to the virtual box GUI.

Thanks for the hard work team.

A