So often errors and bugs while making a small changes...

[ujjain]

I'm currently booting the installer of a FreeNAS VM, but it's stuck when booting from the CD. I rebooted and it happened again.

The text on the console seems unrelated to the issue.

As the solution was to change the number of CPU's from 2 to 1 and amount of memory from 16 to 8 and the now installer boots fine.

Any explanation? I've had quiet a few of these bizarre issues with FreeNAS. Running on VMware, H330 HBA passthrough, 6x4TB disk's. Boot disk is 16gb vmdk.

 

[joeschmuck]

Seriously, you needed to reduce the CPU count and RAM amount to make it pass?

Could you provide specific details so I could try to reproduce it myself? I'd need to know what version of VMWare, the VM version, did you lock the RAM, the FreeNAS ISO, and anything else you can think of. The only think I can't replicate is your HBA and drives since I'll be trying this on a complete VM machine, I wouldn't expect the HBA to be the cause here. Also was the 16gb boot drive new or did it have a previous version of FreeNAS on it? Did you choose UEFI or Legacy? Hopefully Legacy, I don't think UEFI is reliable under VMWare, or should I say that I've not had good luck in the past with it.

 

[ujjain]

Manufacturer: Dell Inc.
Model: PowerEdge T330
CPU: 4 CPUs x Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GHz
Memory: 32 GB ECC 2400

Not UEFI in VMware, but BIOS.
Guest OS: FreeBSD (64-bit)
Compatibility: ESXi 6.5 and later (VM version 13)
VMware Tools: Yes
CPUs: 1
Memory: 16 GB (reserved)
HDD: 16 GB vmdk on SSD NVMe storage.
HBA: H330 in HBA-controller mode, not raid mode. http://i.imgur.com/7uEmrmH.png

http://i.imgur.com/pVaIRlP.png
http://i.imgur.com/pXmupLO.png
http://i.imgur.com/qE62m3F.png

This boots fine as long as I don't increase the number of CPU's to 2.

 

[joeschmuck]

This boots fine as long as I increase the number of CPU's to 2.

Isn't this contrary to your first posting?

Also I need to know which ISO image you used specifically, can you provide the file name? And was this a clean install or upgrade, if upgrade, from what?

 

[ujjain]

Isn't this contrary to your first posting?

Yes, sorry, it works only with a single CPU.

Also I need to know which ISO image you used specifically, can you provide the file name? And was this a clean install or upgrade, if upgrade, from what?

Clean install on a lazy zeroed VMDK.

ISO name is FreeNAS-11.0-U2.iso from https://download.freenas.org/11/11.0-U2/x64/FreeNAS-11.0-U2.iso, installed to disk.

 

[joeschmuck]

Okay, I have that ISO already. I'll give this a shot soon, right now I'm trying to get the wife out of the house, we need to drive into town for a few things.

 

[ujjain]

No worries, let me know if you need an tarball of my FreeNAS-VM directory. Would that help? Even though obviously you might have to change the HBA-passthrough unless you have a H330 around.

 

[joeschmuck]

So I think I misunderstood a little until I tried the scenario. During the bootup you never get to the installation screen because it hangs.

This is one odd issue to be honest with you. I just tried it and I configured the CPU a 1 CPU with 2 cores, and 2 CPUs with 1 core. I could not get it to hang. I installed the software without issue.

I am running VMWare ESXi 6.5.0 Update 1 (Build 5969303) and your version is the initial release. My advice would be to update to the current version. Have you used this upgrade tool before? If your ESXi has internet access then the upgrade is super easy. You just place your ESXi into Maintenance Mode, Enable SSH and open up an SSH window, Cut and Paste all the text from the link "My VMWare Patch Download". Then you wait while the updates are downloaded. Since you have a lot of updates you can expect the screen to not do anything for a little while. After a while it will fill up with all the upadtes applied and ask you ro reboot, then just type "reboot" and wait for the machine to come up again. Then in the GUI ensure you have the latest updates (check the version number) take the machine out of maintenance mode and try your VM again.

Let me know how that works if you do upgrade.

-Mark

 

[ujjain]

I have updated with vCenter, I hope that's fine too.

• Version: 6.5.0 Update 1 (Build 5969303)
• State: Normal (connected to vCenter Server at 192.168.86.192)
• Uptime: 0.01 days

• Image profile (Updated) Dell-ESXi-6.5.0-4564106-A00 (Dell)
• vSphere HA state Not configured
• vMotion Supported

Same issue, no boot with multiple CPU's. Without CPU's all fine.

Code:

Aug 19 17:59:48 freenas da1 at mrsas0 bus 1 scbus4 target 0 lun 0
Aug 19 17:59:48 freenas da1: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SPC-4 SCSI device
Aug 19 17:59:48 freenas da1: 150.000MB/s transfers
Aug 19 17:59:48 freenas da1: 3815447MB (7814037168 512 byte sectors)
Aug 19 17:59:48 freenas da1: quirks=0x8<4K>
Aug 19 17:59:48 freenas da4 at mrsas0 bus 1 scbus4 target 4 lun 0
Aug 19 17:59:48 freenas da4: <ATA WDC WD40EFRX-68N 0A82> Fixed Direct Access SPC-4 SCSI device
Aug 19 17:59:48 freenas da4: 150.000MB/s transfers
Aug 19 17:59:48 freenas da4: 3815447MB (7814037168 512 byte sectors)
Aug 19 17:59:48 freenas da4: quirks=0x8<4K>
Aug 19 17:59:48 freenas da5 at mrsas0 bus 1 scbus4 target 5 lun 0
Aug 19 17:59:48 freenas da5: <ATA WDC WD40EFRX-68N 0A82> Fixed Direct Access SPC-4 SCSI device
Aug 19 17:59:48 freenas da5: 150.000MB/s transfers
Aug 19 17:59:48 freenas da5: 3815447MB (7814037168 512 byte sectors)
Aug 19 17:59:48 freenas da5: quirks=0x8<4K>
Aug 19 17:59:48 freenas da3 at mrsas0 bus 1 scbus4 target 3 lun 0
Aug 19 17:59:48 freenas da3: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SPC-4 SCSI device
Aug 19 17:59:48 freenas da3: 150.000MB/s transfers
Aug 19 17:59:48 freenas da3: 3815447MB (7814037168 512 byte sectors)
Aug 19 17:59:48 freenas da3: quirks=0x8<4K>
Aug 19 17:59:48 freenas da2 at mrsas0 bus 1 scbus4 target 1 lun 0
Aug 19 17:59:48 freenas da2: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SPC-4 SCSI device
Aug 19 17:59:48 freenas da2: 150.000MB/s transfers
Aug 19 17:59:48 freenas da2: 3815447MB (7814037168 512 byte sectors)
Aug 19 17:59:48 freenas da2: quirks=0x8<4K>
Aug 19 17:59:48 freenas da6 at mrsas0 bus 1 scbus4 target 6 lun 0
Aug 19 17:59:48 freenas da6: <ATA ST4000DM000-2AE1 0001> Fixed Direct Access SPC-4 SCSI device
Aug 19 17:59:48 freenas da6: 150.000MB/s transfers
Aug 19 17:59:48 freenas da6: 3815447MB (7814037168 512 byte sectors)
Aug 19 17:59:48 freenas da6: quirks=0x8<4K>
Aug 19 17:59:48 freenas random: unblocking device.
Aug 19 17:59:48 freenas Timecounter "TSC-low" frequency 1848000000 Hz quality 1000
Aug 19 17:59:48 freenas taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
Aug 19 17:59:48 freenas taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
Aug 19 17:59:48 freenas Trying to mount root from zfs:freenas-boot/ROOT/default []...
Aug 19 17:59:48 freenas vmx_init: processor does not support VMX operation
Aug 19 17:59:48 freenas module_register_init: MOD_LOAD (vmm, 0xffffffff8281a620, 0) error 6
Aug 19 17:59:48 freenas GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
Aug 19 17:59:48 freenas pmc: Unknown Intel CPU.
Aug 19 17:59:48 freenas hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI>
Aug 19 17:59:48 freenas VMware memory control driver initialized
Aug 19 17:59:48 freenas kernel: vmx0: link state changed to UP
Aug 19 17:59:48 freenas kernel: vmx0: link state changed to UP
Aug 19 17:59:49 freenas root: /etc/rc: WARNING: failed to start watchdogd
Aug 19 17:59:49 freenas GEOM_ELI: Device da1p1.eli created.
Aug 19 17:59:49 freenas GEOM_ELI: Encryption: AES-XTS 128
Aug 19 17:59:49 freenas GEOM_ELI:	 Crypto: hardware
Aug 19 17:59:49 freenas GEOM_ELI: Device da2p1.eli created.
Aug 19 17:59:49 freenas GEOM_ELI: Encryption: AES-XTS 128
Aug 19 17:59:49 freenas GEOM_ELI:	 Crypto: hardware
Aug 19 17:59:49 freenas GEOM_ELI: Device da3p1.eli created.
Aug 19 17:59:49 freenas GEOM_ELI: Encryption: AES-XTS 128
Aug 19 17:59:49 freenas GEOM_ELI:	 Crypto: hardware
Aug 19 17:59:49 freenas GEOM_ELI: Device da4p1.eli created.
Aug 19 17:59:49 freenas GEOM_ELI: Encryption: AES-XTS 128
Aug 19 17:59:49 freenas GEOM_ELI:	 Crypto: hardware
Aug 19 17:59:49 freenas GEOM_ELI: Device da5p1.eli created.
Aug 19 17:59:49 freenas GEOM_ELI: Encryption: AES-XTS 128
Aug 19 17:59:49 freenas GEOM_ELI:	 Crypto: hardware
Aug 19 17:59:49 freenas GEOM_ELI: Device da6p1.eli created.
Aug 19 17:59:49 freenas GEOM_ELI: Encryption: AES-XTS 128
Aug 19 17:59:49 freenas GEOM_ELI:	 Crypto: hardware
Aug 19 17:59:50 freenas ntpd[1992]: ntpd 4.2.8p10-a (1): Starting
Aug 19 17:59:50 freenas nfsd: can't register svc name
Aug 19 17:58:13 freenas smbd: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Aug 19 17:58:14 freenas smbd: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Aug 19 17:58:15 freenas smbd: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Aug 19 17:58:16 freenas smbd: dnssd_clientstub ConnectToServer: connect() failed path:/var/run/mdnsd Socket:36 Err:-1 Errno:2 No such file or directory
Aug 19 17:58:21 freenas netatalk[2866]: Netatalk AFP server starting
Aug 19 17:58:21 freenas cnid_metad[2871]: CNID Server listening on localhost:4700
Aug 19 17:58:21 freenas netatalk[2866]: Registered with Zeroconf
Aug 19 17:58:21 freenas root: /etc/rc: WARNING: failed precmd routine for minio
Aug 19 17:58:21 freenas afpd[2870]: Netatalk AFP/TCP listening on 192.168.86.140:548
Aug 19 17:58:26 freenas mountd[1910]: unknown group:
Aug 19 17:58:27 freenas mountd[1910]: unknown group:
Aug 19 17:58:27 freenas mountd[1910]: unknown group:
Aug 19 17:58:29 freenas bridge0: Ethernet address: 02:e0:a9:ac:fb:00
Aug 19 17:58:29 freenas kernel: bridge0: link state changed to UP
Aug 19 17:58:29 freenas kernel: bridge0: link state changed to UP
Aug 19 17:58:29 freenas kernel: vmx0: promiscuous mode enabled
Aug 19 17:58:29 freenas epair0a: Ethernet address: 02:ff:e0:00:05:0a
Aug 19 17:58:29 freenas epair0b: Ethernet address: 02:ff:30:00:06:0b
Aug 19 17:58:29 freenas kernel: epair0a: link state changed to UP
Aug 19 17:58:29 freenas kernel: epair0a: link state changed to UP
Aug 19 17:58:29 freenas kernel: epair0b: link state changed to UP
Aug 19 17:58:29 freenas kernel: epair0b: link state changed to UP
Aug 19 17:58:29 freenas kernel: epair0a: promiscuous mode enabled

 

[ujjain]

And this is with 2 CPU's:

 

[ujjain]

Although this get a nice stacktrace with debug mode on this stime:

 

[joeschmuck]

I have updated with vCenter, I hope that's fine too.

I'm too poor to afford vCenter otherwise I'd have it. Good features using it.

I noticed on one of your screenshots that it appears you do not have "Hardware virtualization" and "Performance counters" enabled but rather they are disabled. For me these are turned on. Maybe this will help?

 

[joeschmuck]

Hey, try this one on for size so that both you and I are on the same page...

Create a new VM with two hard drives, the first one 8GB, the second 20GB (Virtual Device Controller = SCSI, SCSI Controller = LSI Parallel Logic). Two CPUs, 16GB RAM (not locked), turn on the hardware virtualization and performance counters, add an E1000 NIC, and the rest likely default settings and your FreeNAS ISO fo course. I'm curious if the issue is the difference between having your HBA or not. If you have the problem with a pure VM setup then I'm not sure where to go from here. While it doesn't make any sense, I'd download the ISO again just to rule it out, but I honestly do not see that as being the issue. Just trying to rule things out and making the complete VM with VM drives would remove the HBA from the issue.

Good luck!

 

[Stux]

You can just generate an md5 of the ISO and check it.

I noticed the stack trace mentioned "WITNESS" was enabled?

Anyway, try the basic version with two virtual disks like @joeschmuck recommended

 

[joeschmuck]

I noticed the stack trace mentioned "WITNESS" was enabled?

Now I'm going to have to look that up you know since I don't know what that means.

 

[ujjain]

2 vCPU's on a new FreeNAS VM seems to work fine.

On the original VM too without the HBA attached. So it does seem related to the PERC H330.

 

[joeschmuck]

Glad you figured it out, sorry it was the HBA. It's just odd. What if you had 4 CPUs and 16GB RAM? I'm curious if it's just the 2 CPUs or if you must have a single cpu core. Depending on how you use FreeNAS, 1 cpu core may be fine.

 

[Stux]

Glad you figured it out, sorry it was the HBA. It's just odd. What if you had 4 CPUs and 16GB RAM? I'm curious if it's just the 2 CPUs or if you must have a single cpu core. Depending on how you use FreeNAS, 1 cpu core may be fine.

Ix does specifically mention 2 vCPU as a hard minimum, Ie "at least 2".

http://www.freenas.org/blog/yes-you-can-virtualize-freenas/

 

[ujjain]

FreeNAS boots fine with 2 or 4 vCPU's when I don't attach the PERC H330, but not when I attach the PERC H330.

I personally don't mind so much, I'm probably not going to store that important data on FreeNAS yet until I feel I understand the software part and until I trust the combination of this hardware and FreeNAS.

Same issue on a fresh install. FreeNAS doesn't boot with multiple CPU's and the H330 configured as passthrough. I tried changing hpet0.present to false as per some tips on the internet, but that didn't help.

 

[joeschmuck]

Ix does specifically mention 2 vCPU as a hard minimum, Ie "at least 2".

http://www.freenas.org/blog/yes-you-can-virtualize-freenas/

His problem was he couldn't run 2 vCPUs while using the HBA. Also, 2 CPUs is not an absolute minimum, it's recommended. Trust me, FreeNAS will run with only a single vCPU. I've been doing this for a long time on my test rigs, 1 vCPU, 2vCPU, 4 vCPU, you get the point. Whatever feels good when I build the VM is what I do.