VM no longer boots - inaccessible boot device

Status
Not open for further replies.

birt

Dabbler
Joined
Nov 9, 2015
Messages
26
I have a Windows 2012 R2 VM that was working fine until a couple of days ago when I upgraded from 11.0-U3 to 11.0-U4. Now when it attempts to boot a blue screen comes up with the message Inaccessible boot device.

I tried going back to 11.0-U3 and I am getting the same error so I'm not entirely sure if it's related to U4 or not. It's worth noting that prior to the U4 update I manually shut down the VM cleanly just to avoid this sort of issue.

My configuration: FreeNAS 11.0-U3/U4 booting from an USB stick on an ASRock C2750D4I with 32GB ECC RAM. When it comes to the VM, other than the VNC adapter, it has a VirtIO NIC (tried to change to Intel E82545, still no go) and an AHCI ZVol disk that resides on a SSD drive (also tried to change this one to VirtIO but of course it didn't work which is pretty normal seeing that the VM doesn't have the viostor driver). Trying to boot in safe mode results in the same blue screen and no C:\windows\ntbtlog.txt file is created, something that according to Microsoft can only happen if the OS cannot get a handle to the disk to write the file. I even specifically tried to select the Enable Boot Logging option - same result, no log file. Trying to select Last known good configuration results in the same blue screen.

What is strange is that when it boots in automatic repair mode, I can open a command prompt and there is no problem if I try to create a file on the C: drive, it works fine... Also, chkdsk reports no problems. Tried sfc /scannow with /offbootdir and /offwindir a couple of times, the first time it reported repairing something but the 2nd time it reported no integrity violations yet the blue screen keeps showing up. I did nothing to the ZVol or to the VM configuration in the past couple of weeks and no Windows updates were installed recently (it's configured for manual update installation) - basically it simply stopped working when I updated to 11.0-U4.

I checked the FreeNAS logs and nothing relevant appears to happen when the VM boot fails, which is not completely unexpected seeing that the error occurs inside the VM.

Did anyone encounter similar problems? Is there anything that I am overlooking that might be causing this behavior? Any suggestions on what I could do at this point?
 

m0nkey_

MVP
Joined
Oct 27, 2015
Messages
2,739
You say it boots into repair mode. Have you tried hitting the repair button? Sounds to me like an update or driver is not playing nice causing a crash.
 

birt

Dabbler
Joined
Nov 9, 2015
Messages
26
That was one of the first things I tried. Sadly, it fails.

I completely agree: it looks like a driver issue, specifically the driver needed to access the AHCI boot device (because it does not even write the boot log). When booting from the install ISO, the repair option also resulted in a similar "inaccessible boot device" message which makes me think that the problem may not be related to the Windows installation itself, but rather to bhyve somehow.
 

m0nkey_

MVP
Joined
Oct 27, 2015
Messages
2,739
What about creating a new Windows VM? It'll be interesting to see if you get it with that too. Make sure you have the latest VirtIO drivers slip-streamed into the installer.
 

birt

Dabbler
Joined
Nov 9, 2015
Messages
26
I'm traveling now but I'll try that this weekend when I get back.

When installing I've tried about 10 (no exaggeration) different versions of the viostor drivers including the so-called stable versions and absolutely all of them had problems with Windows 2012 R2, resulting in a straight blue screen or in the VM becoming completely unresponsive and requiring me to kill the associated bhyve process. The only option that was stable was using AHCI. The network driver also has problems, it took trying 4-5 versions to find one that works and it was a version much older than the current stable.

Anyway, my point is that I won't even bother with VirtIO, I'll give it a try with AHCI which was working perfectly fine until a few days ago.
 

birt

Dabbler
Joined
Nov 9, 2015
Messages
26
Tried to install another Windows 2012 R2 VM in 11.0-U4 and that one seemed to be working fine. I guess I will have to blame Windows and reinstall.

On the same topic, I also tried the viostor driver in several VirtIO versions (105, 126, 141) in combination with a few different Windows versions (7, 10, 2012 R2) but it appears to be completely broken in bhyve, which crashes as soon as the driver is loaded with the following messages in /var/log/middlewared.log:
Code:
[2017/10/15 13:50:08] (DEBUG) ServiceBase.run():128 - Backup: Assertion failed: (n >= 2 && n <= BLOCKIF_IOV_MAX + 2), function pci_vtblk_proc, file /freenas-11-releng/freenas/_BE/os/usr.sbin/bhyve/pci_virtio_block.c, line 216.
[2017/10/15 13:50:08] (DEBUG) ServiceBase.run():128 - Backup: fbuf frame buffer base: 0x942800000 [sz 16777216]
[2017/10/15 13:50:09] (WARNING) ServiceBase.destroy_vm():149 - ===> DESTROYING VM: Backup ID: 3 BHYVE_CODE: -6


I guess the issue is the same one described at https://reviews.freebsd.org/D10581
 

mattyakel

Cadet
Joined
Aug 6, 2013
Messages
6
Oh my goodness thank you so much for this post! I was slamming my head on the desk trying to get a WinServer2008R2 installed with VirtIO but it crashes after the drivers try to load during the install.


Sent from my iPhone using Tapatalk
 

birt

Dabbler
Joined
Nov 9, 2015
Messages
26
Oh my goodness thank you so much for this post! I was slamming my head on the desk trying to get a WinServer2008R2 installed with VirtIO but it crashes after the drivers try to load during the install.

If it makes you feel any better, I also spent quite a few hours trying various things before I figured out it was a bhyve issue. AHCI appears to work fine, though.
 

EsTaF

Contributor
Joined
Sep 20, 2013
Messages
163
Have that problem too.
Just run win2k12 in ~5 time (first 4 timeы all're ok) and get error:
"!SAC>
Your PC ran into a problem and needs to restart.
If you'd like to know more, you can search online later for this error:
INACCESSIBLE_BOOT_DEVICE

0xFFFFD000C872D790
0xFFFFFFFFC0000034
0x0000000000000000
0x0000000000000000"
I always use achi-hd option. (-s 4,ahci-hd,/dev/zvol/temp01/win2k12). Not virtio.
I can remove specs settings from windows (injecting a standart drivers settings into system hive of windows journal) - it's not help.
 
Last edited:

birt

Dabbler
Joined
Nov 9, 2015
Messages
26
Even though I suspect the problem was not related to the VM itself, I ended up reinstalling :(
 

EsTaF

Contributor
Joined
Sep 20, 2013
Messages
163
Just the predictability = 0.
You install system, add some apps, settings etc. working. And in one nice time have a not working one, and generally moment - you don't know, why it's lucked and don't know nobody.
don't understand.
 

birt

Dabbler
Joined
Nov 9, 2015
Messages
26
Yeah, well, I've had the same - software installed and configured and it suddenly stopped working. Luckily I had a backup of the configuration so it wasn't a huge issue but I did waste quite a lot of time trying to figure out what went wrong before reinstalling.

Unfortunately, bhyve doesn't seem like a very stable hypervisor at the moment nor is it straightforward to use. I guess we're the early adopters and it's probably going to be a few years until it reaches the stability and ease of use of the other hypervisors.
 

EsTaF

Contributor
Joined
Sep 20, 2013
Messages
163
And, for example, I add a some ppt variable (https://murf.se/iohyve-and-pci-passthru/) to loader.conf !!! over web gui. After it I can't view that one in /boot/loader.conf. If that ppt variable will be not right, I can't load freenas os. How I can edit that one in offline?
For example, I detach flash drive and can mount it on other bsd os (https://headcrash.industries/refere...-from-zfs-boot-drive/#mounting-the-zfs-volume).
I'd add ppt variable for xhci usb in loader.conf over web gui as 2.0.0. have that value for xhci. After it, restart os and have error 2 boot. But I add usb flash to uchi (usb 2.0) already. anyway.
How can I delete that variable from pseudo loader.conf from freenas "sql"?

because, from txt' loader.conf I can't remove it

1 14:21 zpool import
2 14:21 mkdir /tmp/freenas
3 14:21 zpool import -R /tmp/freenas freenas-boot
4 14:21 zpool import -f -R /tmp/freenas freenas-boot
5 14:21 zpool status -v
6 14:22 zfs list
7 14:22 zfs get all
8 14:23 zfs mount freenas-boot/ROOT/11.0-U4
9 14:23 cd /tmp/freenas/
10 cat boot/defaults/loader.conf | grep ppt
ng_pptpgre_load="NO" # PPTP GRE protocol netgraph node type

nothing
 
Last edited:

yottabit

Contributor
Joined
Apr 15, 2012
Messages
192
Thanks for the link to FreeBSD. I have the VirtIO NIC working fine in Windows 10 which improved the stability, but have also had zero luck getting the VirtIO disk driver to work without crashing. The AHCI driver is exceedingly slow. Until this gets sorted out I'm considering running KVM in Linux and then Windows inside KVM, or going through monumental hassle make Windows install to an iSCSI target. Both options are less than ideal.
 

kleinem

Dabbler
Joined
Jul 29, 2014
Messages
10
Has anyone bothered to open a bugreport for this?
I did a fresh install for 11.1 today and i'm running into the same problem.

However, i can't even seem to start a proper installation.
I would start Windows Setup, load the viostor drivers from a 2nd CD drive.
The disks get detected afterwards but the vm crashes after selecting a disk and hitting next.
Code:
Assertion failed: (n >= 2 && n <= BLOCKIF_IOV_MAX + 2), function pci_vtblk_proc, file /freenas-11-releng/freenas/_BE/os/usr.sbin/bhyve/pci_virtio_block.c, line 216
 

birt

Dabbler
Joined
Nov 9, 2015
Messages
26
You might as well forget about using the viostor drivers for now. It just won't work.

I haven't opened a FreeNAS bug report but if I remember correctly there was one already on the FreeBSD tracker. Actually now that I scroll up I see I even posted the URL above: https://reviews.freebsd.org/D10581
 

kleinem

Dabbler
Joined
Jul 29, 2014
Messages
10
Yeah i kind of missed it the first time i scrolled through the Thread.
I'n not an expert in software development and release cycles etc, i would just like to know if that commit is going to make it into any stable release, or if we can get a patched bhyve before that.

However, if i understand that Bugfix correctly, it will make Windows work but break Linux?!
It's really a pity, since the AHCI emulation performance is horrible.
 
Last edited:

birt

Dabbler
Joined
Nov 9, 2015
Messages
26
There is an "abandoned" tag on that issue, I believe its current status is "going nowhere fast". I have no idea what to do from here but I believe the FreeNAS team doesn't handle bhyve bugfixing so the ball is likely in FreeBSD's court.

Personally, I just got used to the idea that I have to contend myself with AHCI.

What's much worse is that even with AHCI I get random blue screens with a Windows 7 VM after 5-10 days of uptime and there's not much that indicates what may be the cause even though I spent several hours searching the Windows logs and FreeNAS logs. My $0.02 is that bhyve is quite far from ready to be a stable VM environment when it comes to Windows; I will probably migrate the VM I'm using to a different server when I have some time. Perhaps I will try Windows running in bhyve once more in some 3-5 years.
 
Status
Not open for further replies.
Top