SOLVED Unable to start or connect Virtual Machine - FreeNAS 11 - Intel i7 2600k

[moosethemucha]

Hi guys, I'm having some issues with getting VM's to work on my machine. I've followed the steps here https://doc.freenas.org/11/vms.html, and also watched this https://www.youtube.com/watch?v=GjMIgkC5REM video just to make sure I was doing it right.

Basically I set up the VM and when I hit start, I receive a successful flash message but the status of the VM is still stopped. The config of the VM pretty straightforward

• UEFI (tried both)
• 1 virtual CPU (can't go any higher due to hardware restrictions)
• RAM 2048
• NIC and VNC default (tried setting up manually but same result)
• DISK created a 100GiB zVol
• CD-ROM using Ubuntu 14.04 server amd64 (will try ubuntu 16 tomorrow)

Build FreeNAS-11.0-U4 (54848d13b)
Platform Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Memory 16238MB

Now I think it might be my processor being slightly to old - which is what is causing the Virtual CPU restrictions (which is fine for my use case).

When I do press start the only entries made in /var/log/messages are

Code:

Dec 14 02:50:07 freenas tap0: Ethernet address: 00:bd:e4:64:17:00
Dec 14 02:50:07 freenas kernel: tap0: promiscuous mode enabled

I'm not really sure where bhyve and iohyve store there logs.

EDIT:

https://ark.intel.com/products/52214/Intel-Core-i7-2600K-Processor-8M-Cache-up-to-3_80-GHz

It appears me supports EPT are there any other requirements for bhyve.




Your help would be greatly appreciated.

 

[moosethemucha]

bump

 

[dlavigne]

Does updating to 11.1 resolve the issue?

 

[moosethemucha]

I did try to update to 11.1 but didn't properly pull over my config so was unable to mount my zpool, but I'll try again today.

 

[moosethemucha]

So I updated to 11.1 via the GUI (done that a couple times without any issues) - My volume is unable to be imported; even when I upload the config directly into the install. Reverting back to 11.0 fixes the problem; but im unable to test VM's as I don't have any storage, but maybe we try solving this problem first and see if I can get bhyve working 11.1. Currently this what my UI looks like

Im not really sure whats going on - just a note my zpool is made up of different sized disks so when I set them up I had to create the zpool via the command line - basically its 5*3tb + 1*4tb; I had a new 4tb lying around when I built the system and rather than buying another 3tb I used this spare 4tb. I've had the system running for almost 2 years without any stability issues at all.

NOTE: all these commands were run on the 11.1 system not on 11.0

Code:

zpool staus
pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:02:49 with 0 errors on Sat Dec 16 03:47:49 2017
config:

	NAME		STATE	 READ WRITE CKSUM
	freenas-boot  ONLINE	   0	 0	 0
	  da0p2	 ONLINE	   0	 0	 0

errors: No known data errors

Code:

zpool list
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
freenas-boot  29.5G  2.91G  26.6G		 -	  -	 9%  1.00x  ONLINE  -

Code:

glabel status
									  Name  Status  Components
							  label/efibsd	 N/A  da0p1
gptid/1440971d-5108-11e7-91c2-50e549e84010	 N/A  da0p1
gptid/0674f3e1-fbc7-11e6-a4c5-50e549e84010	 N/A  ada0p2
gptid/07aaa821-fbc7-11e6-a4c5-50e549e84010	 N/A  ada1p2
gptid/a23bf851-50b4-11e7-9238-50e549e84010	 N/A  ada2p2
gptid/053a947a-fbc7-11e6-a4c5-50e549e84010	 N/A  ada3p2
gptid/08659a6c-fbc7-11e6-a4c5-50e549e84010	 N/A  ada4p2
gptid/09fc61d5-fbc7-11e6-a4c5-50e549e84010	 N/A  ada5p2

Code:

camcontrol devlist
ST33000651AS CC43>				at scbus0 target 0 lun 0 (ada0,pass0)
<WDC WD30EZRX-00DC0B0 80.00A80>	at scbus0 target 1 lun 0 (ada1,pass1)
<WDC WD30EFRX-68AX9N0 80.00A80>	at scbus1 target 0 lun 0 (ada2,pass2)
<WDC WD4003FZEX-00Z4SA0 01.01A01>  at scbus1 target 1 lun 0 (ada3,pass3)
<TOSHIBA DT01ACA300 MX6OABB0>	  at scbus2 target 0 lun 0 (ada4,pass4)
<TOSHIBA DT01ACA300 MX6OABB0>	  at scbus3 target 0 lun 0 (ada5,pass5)
<Lexar USB Flash Drive 1100>	   at scbus5 target 0 lun 0 (pass6,da0)

This last one has me the most confused - as there are defintely no hardware faults and if swap to 11.0 and reboot everything is there as normal.

Code:

sudo zpool import
   pool: volume1
	 id: 10364792877333430154
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
	devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

	volume1										 UNAVAIL  insufficient replicas
	  raidz2-0									  UNAVAIL  insufficient replicas
		4139841096778600255						 UNAVAIL  cannot open
		354619203168905401						  UNAVAIL  cannot open
		gptid/a23bf851-50b4-11e7-9238-50e549e84010  ONLINE
		9055488112294957408						 UNAVAIL  cannot open
		16478642635572880347						UNAVAIL  cannot open
		12910193442291371281						UNAVAIL  cannot open

 

[moosethemucha]

Just thought I'd share the commands above running on FreeNAS 11.0

Code:

zpool status
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h2m with 0 errors on Sat Dec 16 03:47:49 2017
config:

	NAME		STATE	 READ WRITE CKSUM
	freenas-boot  ONLINE	   0	 0	 0
	  da0p2	 ONLINE	   0	 0	 0

errors: No known data errors

  pool: volume1
 state: ONLINE
  scan: scrub repaired 0 in 24h33m with 0 errors on Mon Nov 27 00:33:26 2017
config:

	NAME											STATE	 READ WRITE CKSUM
	volume1										 ONLINE	   0	 0	 0
	  raidz2-0									  ONLINE	   0	 0	 0
		ada0										ONLINE	   0	 0	 0
		ada1										ONLINE	   0	 0	 0
		gptid/a23bf851-50b4-11e7-9238-50e549e84010  ONLINE	   0	 0	 0
		ada3										ONLINE	   0	 0	 0
		ada4										ONLINE	   0	 0	 0
		ada5										ONLINE	   0	 0	 0

errors: No known data errors

Code:

zpool list
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
freenas-boot  29.5G  2.91G  26.6G		 -	  -	 9%  1.00x  ONLINE  -
volume1	   16.2T  9.12T  7.13T	 16.0E	20%	56%  1.00x  ONLINE  /mnt

Code:

glabel status
									  Name  Status  Components
gptid/a23bf851-50b4-11e7-9238-50e549e84010	 N/A  ada2p2
							  label/efibsd	 N/A  da0p1
gptid/1440971d-5108-11e7-91c2-50e549e84010	 N/A  da0p1

Code:

<ST33000651AS CC43>				at scbus0 target 0 lun 0 (pass0,ada0)
<WDC WD30EZRX-00DC0B0 80.00A80>	at scbus0 target 1 lun 0 (pass1,ada1)
<WDC WD30EFRX-68AX9N0 80.00A80>	at scbus1 target 0 lun 0 (pass2,ada2)
<WDC WD4003FZEX-00Z4SA0 01.01A01>  at scbus1 target 1 lun 0 (pass3,ada3)
<TOSHIBA DT01ACA300 MX6OABB0>	  at scbus2 target 0 lun 0 (pass4,ada4)
<TOSHIBA DT01ACA300 MX6OABB0>	  at scbus3 target 0 lun 0 (pass5,ada5)
<Lexar USB Flash Drive 1100>	   at scbus5 target 0 lun 0 (pass6,da0)

zpool import returns nothing - which is want we want.

 

[moosethemucha]

I'm going to try and set up a new usb stick with a fresh install of FreeNAS 11.1 and see what happens.

 

[moosethemucha]

Is there a reason that FreeNAS isn't available to download via torrent ? Would it be ok for me to host something - or is there something already out there. What I was thinking is something similar to:

http://openbsd.somedomain.net/

From what I can tell under FreeNAS's Creative Commons Attribution License this is completely legal - or have I got this completely wrong ?

 

[danb35]

just a note my zpool is made up of different sized disks so when I set them up I had to create the zpool via the command line

No, you didn't need to do this; the Manual Setup button in the Volume Manager would have done this. But whether this has anything to do with your problems I can't say.

Is there a reason that freenas isn't available to download via torrent?

I'm not aware of any such reason, but is there a reason it needs to be? I think it would be legal, but I have no idea what makes you think FreeNAS is under any sort of Creative Commons license--it isn't; it's under one of the BSD licenses instead.

 

[moosethemucha]

Thank you for your non-helpful response. Yes you are right about the license, I read this as being describing the actual operating system.

The FreeNAS® User Guide is freely available for sharing and redistribution under the terms of the Creative Commons Attribution License. This means that you have permission to copy, distribute, translate, and adapt the work as long as you attribute iXsystems as the original source of the Guide.

It's actually covered by the BSD-2-Clause] - Which I'll investigate further.
With your first point I didn't think that was available in 9.10 - the version i set up the zpool in.

https://forums.freenas.org/index.php?threads/freenas-9-1-zfs-volume-of-different-drive-size.13549/


I don't really understand how running something like this (not the command i ran but something similar)

Code:

zpool create -f data raidz2 ada0 ada1 ada2 ada3

and then doing the rest via the GUI would cause any problems. You might say FreeNAS does thing in the background, so lets look at the code. I haven't gotten very far and if someone could help me find the code which creates the manual disks that would be awesome. What I've found so far.

Also this is the nightly build so it could also be different - I'll be investigating that.

https://github.com/freenas/freenas/blob/9d66325f0aa3c0df133279c180f24ab994d5e996/gui/storage/forms.py#L298

This seems to call the method we are interested in does anyone know where I can find the source of this method ?

Code:

notifier().create_volume(volume, groups=grouped, init_rand=init_rand)

 

[danb35]

I don't really understand how running something like this (not the command i ran but something similar)

Code:

zpool create -f data raidz2 ada0 ada1 ada2 ada3

and then doing the rest via the GUI would cause any problems.

Well, for one thing, FreeNAS partitions the disks--it puts a swap partition (2 GB by default) at the beginning, then uses the rest for ZFS. What you did didn't write a partition table to the disks. I don't know that this should have caused a problem in re-importing the pool, but it's very suspicious that FN11.1 sees the one disk that FreeNAS put in (apparently you replaced a disk at some point--it's referred to by gptid in the pool, and is partitioned), but not any of the others.

Though as I look more closely, it gets stranger--11.1 does see partitions on all six disks (according to glabel status), but only partition 2. In 11.0, glabel status only indicates a partition table on ada2. The disks are either partitioned or not; I don't think of any reason why they'd appear differently under the two versions of the OS--though I suspect that difference gets to the root of your problem. What's the output of gpart show ada1 ada2 under both versions?

With your first point I didn't think that was available in 9.10 - the version i set up the zpool in.

It was most certainly available in 9.10; the four-year-old thread you linked to was for 9.1, which is a completely different version (9.10 was current until this year; blame iX for the confusing version numbers). But it looks like it wasn't present in 9.1; Manual Setup was added to the (ZFS) Volume Manager in 9.2.1, released 7 Feb 14 (see the 9.2.1 User Guide, p. 115; compare to the 9.2.0 User Guide, p. 111).

 

[moosethemucha]

Thank You for your latest response it is greatly appreciated.

This makes sense, it wasn't 9.1 I was on I was on 9. ten - that is confusing and why this could be occurring. I thought I created the partitions via the GUI but that could be false - I simply can't remember.

I can only run that command on ada2 (the gpt drive) which produces

Code:

sudo gpart show ada2
=>        34  5860533101  ada2  GPT  (2.7T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338696     2  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

with any of the others I receive this msg

Code:

gpart: No such geom: ada0.

Now I've got an idea - as seeing as the disk I replaced seems to be set up - why don't I swap out each drive and let FreeNAS rebuild the array one by one. All I need is one new drive and some time - I don't have anywhere to dump the 9tb of data I've got. This would hopefully partition each drive correctly and maybe I'll have a 'properly' configured zpool. then I can worry about 11.1 and bhyve. This might be risky but I think its going to easier than trying to recreate partitions on existing drives.

Thank You

 

[danb35]

Now I've got an idea - as seeing as the disk I replaced seems to be set up - why don't I swap out each drive and let freenas rebuild the array one by one. All I need is one new drive and some time

No need for a new drive. Go into the GUI, offline ada0, go to the CLI, zpool labelclear ada0, to back to the GUI, replace ada0 with ada0. FreeNAS will partition the disk and start the replacement using the second partition. When resilvering completes, repeat with the next disk until all disks are replaced. But was this:

with any of the others I receive this msg

Code:

gpart: No such geom: ada0.

under 11.0 or 11.1?

 

[moosethemucha]

Sorry both of those were run on 11.0 - below is on 11.1

Code:

gpart show ada1
=>		34  5860533101  ada1  GPT  (2.7T) [CORRUPT]
		  34		  94		- free -  (47K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  5856338696	 2  freebsd-zfs  (2.7T)
  5860533128		   7		- free -  (3.5K)

Thats weird isn't it ? So there is a partition table but it is corrupt?

Code:

gpart show ada2
=>		34  5860533101  ada2  GPT  (2.7T)
		  34		  94		- free -  (47K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  5856338696	 2  freebsd-zfs  (2.7T)
  5860533128		   7		- free -  (3.5K)

I'll hold on the resilvering

 

[danb35]

Thats weird isn't it ?

Yes, it is, and I don't think I have a good explanation of what's going on there. If I were to wildly speculate, I might guess something like this:

• The other disks (ada0-1 and 3-5) were previously used in a FreeNAS pool which was created through the web GUI. This resulted in their being partitioned as FreeNAS typically does.
  
• A GPT-partitioned disk has two copies of the partition table stored, one at the beginning of the disk and one at the end (this one I'm pretty sure is true)
• When you created a new pool at the CLI using the whole disks, that overwrite the partition table at the beginning of the disks, but not the one at the end.
• There's been some change in the disk/partition recognition code between FreeBSD 11.0 and 11.1, such that the remaining partition tables on the disks are seen in 11.1, but not in 11.0.

You presumably know if the first point is correct, and I'm pretty sure of the second. But the third and fourth are really just me guessing based on what we're seeing here. I'm not sure who around here knows FreeBSD internals well enough to confirm/refute this, or even to opine as to whether it makes any sense. @rs225? @jgreco?

I'll hold on the resilvering

Probably a good idea. I'm almost certain it would resolve this issue, but it'd be better to work out for certain what's going on if possible.

Edit: This discussion also seems orthogonal to your original question, and might be better split into its own thread. @Ericloewe?

 

[moosethemucha]

Yep point one is correct - The first box I built had the mobo die during the first week or two of use - It was second hand and honestly was a terrible decision on my part - I wanted to set up really bad and thought hey I'll use this mobo with some seriously weird BIOS continuity issues (facepalm)

Anyway I got a new mobo - set up the system and (couple year pause) and here we are.

What you said seems like a pretty legit explanation of what has happened - it makes sense and it lines up with actions I've taken throughout the life of the machine.

I agree with the splitting of the thread is that something I can do or are admins the gatekeepers.

and once again Thank You - I owe you a slab of beer for sure.

 

[danb35]

I agree with the splitting of the thread is that something I can do or are admins the gatekeepers.

The mods (of which Eric is one) would need to do that, I think.

 

[rs225]

I think you're on the right track with the corrupt partition table. Usually the logs will show something on boot if there is something wrong with the partition tables. Is there anything?

I have no idea what might have changed to cause this, but it would be nice to know. It might be a bug, not an intended change.

 

[moosethemucha]

Ok - I'll definitely hold of doing anything rash (re-silver the drives); so we can investigate the problem.

On 11.0

Code:

cat messages | grep GEOM
Dec 18 10:17:27 freenas GEOM_ELI: Device mirror/swap0.eli destroyed.
Dec 18 10:17:27 freenas GEOM_ELI: Detached mirror/swap0.eli on last close.
Dec 18 10:17:27 freenas GEOM_ELI: Device mirror/swap1.eli destroyed.
Dec 18 10:17:27 freenas GEOM_ELI: Detached mirror/swap1.eli on last close.
Dec 18 10:17:27 freenas GEOM_ELI: Device mirror/swap2.eli destroyed.
Dec 18 10:17:27 freenas GEOM_ELI: Detached mirror/swap2.eli on last close.
Dec 18 10:17:27 freenas GEOM: ada0: the primary GPT table is corrupt or invalid.
Dec 18 10:17:27 freenas GEOM: ada0: using the secondary instead -- recovery strongly advised.
Dec 18 10:17:27 freenas GEOM: ada1: the primary GPT table is corrupt or invalid.
Dec 18 10:17:27 freenas GEOM: ada1: using the secondary instead -- recovery strongly advised.
Dec 18 10:17:27 freenas GEOM: ada3: the primary GPT table is corrupt or invalid.
Dec 18 10:17:27 freenas GEOM: ada3: using the secondary instead -- recovery strongly advised.
Dec 18 10:17:27 freenas GEOM: ada4: the primary GPT table is corrupt or invalid.
Dec 18 10:17:27 freenas GEOM: ada4: using the secondary instead -- recovery strongly advised.
Dec 18 10:17:27 freenas GEOM: ada5: the primary GPT table is corrupt or invalid.
Dec 18 10:17:27 freenas GEOM: ada5: using the secondary instead -- recovery strongly advised.
Dec 18 10:17:27 freenas GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
Dec 18 10:17:28 freenas GEOM_ELI: Device ada2p1.eli created.
Dec 18 10:17:28 freenas GEOM_ELI: Encryption: AES-XTS 128
Dec 18 10:17:28 freenas GEOM_ELI:	 Crypto: hardware

on 11.1

Code:

Dec 18 11:30:20 freenas GEOM: ada0: the primary GPT table is corrupt or invalid.
Dec 18 11:30:20 freenas GEOM: ada0: using the secondary instead -- recovery strongly advised.
Dec 18 11:30:20 freenas GEOM: ada1: the primary GPT table is corrupt or invalid.
Dec 18 11:30:20 freenas GEOM: ada1: using the secondary instead -- recovery strongly advised.
Dec 18 11:30:20 freenas GEOM: ada3: the primary GPT table is corrupt or invalid.
Dec 18 11:30:20 freenas GEOM: ada3: using the secondary instead -- recovery strongly advised.
Dec 18 11:30:20 freenas GEOM: ada4: the primary GPT table is corrupt or invalid.
Dec 18 11:30:20 freenas GEOM: ada4: using the secondary instead -- recovery strongly advised.
Dec 18 11:30:20 freenas GEOM: ada5: the primary GPT table is corrupt or invalid.
Dec 18 11:30:20 freenas GEOM: ada5: using the secondary instead -- recovery strongly advised.
Dec 18 11:30:20 freenas GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
Dec 18 11:30:20 freenas GEOM_MIRROR: Device mirror/swap0 launched (2/2).
Dec 18 11:30:20 freenas GEOM_MIRROR: Device mirror/swap1 launched (2/2).
Dec 18 11:30:20 freenas GEOM_MIRROR: Device mirror/swap2 launched (2/2).
Dec 18 11:30:20 freenas GEOM_ELI: Device mirror/swap0.eli created.
Dec 18 11:30:20 freenas GEOM_ELI: Encryption: AES-XTS 128
Dec 18 11:30:20 freenas GEOM_ELI:	 Crypto: hardware
Dec 18 11:30:20 freenas GEOM_ELI: Device mirror/swap1.eli created.
Dec 18 11:30:20 freenas GEOM_ELI: Encryption: AES-XTS 128
Dec 18 11:30:20 freenas GEOM_ELI:	 Crypto: hardware
Dec 18 11:30:20 freenas GEOM_ELI: Device mirror/swap2.eli created.
Dec 18 11:30:20 freenas GEOM_ELI: Encryption: AES-XTS 128
Dec 18 11:30:20 freenas GEOM_ELI:	 Crypto: hardware

Let me know if there is any other info I can get thats going to be useful.

 

[moosethemucha]

Ok; so I couldn't wait - the corrupted tables seems like the most likely answer - and anyway i'd prefer to have my drives set up this way. There was definitely a GPT table on one of the disks. After off-lining one of the disks I was able to see it had a 2.1gb swap space and the rest being a zfs partition.

With resilvering I offlined the disk and then re-added it via the GUI with the force tag - does that delete all remaining data on the drive and then re-add it or should I run zpool labelclear DISK on each disk to be sure ?