FreeNAS on ESXi serving VMs via NFS

chuckypp

Dabbler
Joined
Jan 29, 2015
Messages
11
Hi Everyone,

I would really appreciate some help on this. I got to the point where I couldn't deal with the poor performance of Bhyve VMs on FreeNAS (it really is broken) and recently have moved back to VMware for VMs.

My architecture is:
  • ESXi 6.7 on bare metal with a small local SSD hosting the FreeNAS 11.2-U2.1 VM.
  • FreeNAS has all the other drives passed through and has a volume serving back a datastore to ESXi for other VMs via NFS v4.1.
The performance of this is really good which I was surprised with but that's a good thing.

My question is when I reboot the server, ESXi comes up first obviously and tries to NFS mount the FreeNAS volume but can't (as FreeNAS hasn't started yet when ESXi first comes up) and so it looks stale and the VMs don't register. To get it working again effectively I have to remove the NFS mount and remount it and then re-register the VMs each time I reboot which isn't a great solution obviously.

Is there another way to deal with this? I would really appreciate any guidance.

Thank you :)

Charles
 

Turgin

Dabbler
Joined
Feb 20, 2016
Messages
43
From my notes when I first started down the AIO path:

• Configure passwordless SSH from FreeNAS to ESXi
  • Enable SSH on ESXi
  • From FreeNAS CLI run “ssh-keygen” do not set a passphrase
  • SSH from FreeNAS to ESXi and accept the key
  • Exit the ESXi SSH session
  • From FreeNAS CLI run “cat ./id_rsa.pub | ssh root@<esx mgmt ip> 'cat >> /etc/ssh/keys-root/authorized_keys'”
• Setup FreeNAS VM to make host rescan for storage post-init
  • Create a post-init task command: ssh <username>@<esx mgmt ip> esxcli storage core adapter rescan –all
 

Terydan

Dabbler
Joined
Jan 19, 2017
Messages
31
I'm trying to follow the commands in the post above but I get stuck at the following:
  • From FreeNAS CLI run “cat ./id_rsa.pub | ssh root@<esx mgmt IP> 'cat >> /etc/ssh/keys-root/authorized_keys'”
I get a command not found for the cat command after ssh'ing into esxi, any ideas?
 
Joined
Jul 2, 2019
Messages
648
Maybe try ssh-copy-id from the FreeNAS box to the VMware box? Make sure you have the .ssh (note the leading "dot") created on the VMware box. Make sure the correct permissions are on the .ssh directories/authorized_keys/etc.
 

Turgin

Dabbler
Joined
Feb 20, 2016
Messages
43
I get a command not found for the cat command after ssh'ing into esxi, any ideas?

Are you running that command from the ssh to esxi or from FreeNAS CLI? You're supposed to execute that from FreeNAS. The prior step is "Exit the ESXi SSH session "

Not sure how much help I can be as I don't host VMs on AIO storage any longer. The process was pieced together from much research and I made a how-to for myself.
 

Terydan

Dabbler
Joined
Jan 19, 2017
Messages
31
Are you running that command from the ssh to esxi or from FreeNAS CLI? You're supposed to execute that from FreeNAS. The prior step is "Exit the ESXi SSH session "

Not sure how much help I can be as I don't host VMs on AIO storage any longer. The process was pieced together from much research and I made a how-to for myself.
Pretty sure I exited from the ESXi session, I was triple checking the instructions so that I wouldn't bork up one of the commands and break something bad. I'll try the process again and make sure of it. It may be possible there was an update that makes the process not work anymore too. I'll try again
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
I'm trying to follow the commands in the post above but I get stuck at the following:
  • From FreeNAS CLI run “cat ./id_rsa.pub | ssh root@<esx mgmt IP> 'cat >> /etc/ssh/keys-root/authorized_keys'”
I get a command not found for the cat command after ssh'ing into esxi, any ideas?
Did you enter the above command explicitly? Or did you substitute the IP address of your ESXi server for <esx mgmt IP>?
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478

Terydan

Dabbler
Joined
Jan 19, 2017
Messages
31
Good! I was just checking the obvious...

The command you're trying to run is probably from the VMware article below. You may want to refer to it and make sure you're entering the backtics and such correctly:

https://blogs.vmware.com/vsphere/2012/07/enabling-password-free-ssh-access-on-esxi-50.html
I finally got the command to run. I was originally entering it with the " at the beginning and end like the code above, but after reading the VMware article I see those weren't part of the command. I was also getting a
Code:
./id_rsa.pub : No such file or directory
error and found that I had to use
Code:
cat /root/./id_rsa.pub
Once I made that change it ran without error. Now I can continue forward with my efforts. Thanks all for the assistance.
 

Terydan

Dabbler
Joined
Jan 19, 2017
Messages
31
P.S: The post-init command above has been added to my FreeNAS under Tasks/ Init/Shutdown Scripts as a command. To confirm, this will make ESXi rescan my NFS shares and re-mount them after a reboot?

My most critical NFS share is a 1TB share for my CCTV software VM where all my footage is archived. During a reboot it fails to mount and the VM won't start, therefore disabling my security camera recording. I did run the above command directly in the ESXi ssh terminal, but it didn't do anything and I had to remove/add the nfs share manually to get it running again. Obviously I'll have to reboot soon to confirm the above worked for me, but I just wanted to make sure I have checked all my boxes before taking everything down again to test.

My setup is identical to the OP except that I store all my VMs on the host NVMe datastore and use NFS shares for storing backups and long term files associated to my VMs. I am planning on installing PCIe card that will perform RAID 1 mirroring of my my NVMe which is why I'm not using the FreeNAS NFS datastores to host my VM data.
 

John Doe

Guru
Joined
Aug 16, 2011
Messages
635
May I ask why you use v4.1?

I had a similar issue, but since I use a vSwitch only for the NFS share (without password), there is no connection to any other entity. so I do not expect any security issue.

with that workaround computer boots up, starts esxi, starts FN and after some time, the vms will start.
 

Terydan

Dabbler
Joined
Jan 19, 2017
Messages
31
NFS 3 will automatically rescan until it the FreeNAS VM spins up? I wasn't aware of that, but then again I only recently have dabbled in NFS shares. I did attempt to use a separate switch for my "storage network" after seeing the recommendation on another thread, but for the life of me I couldn't get it to work. I created a second vswitch with no physical adapter and added a vkernel portgroup with <IP1>/16 for ESXi, and added a second adapter to my FreeNAS VM which was added to the vswitch. In FreeNAS I set the IP static with <IP2>/16. I could ping the host from FreeNAS, but the host could not ping or connect to FreeNAS VM. After a couple days on and off messing with it i gave up.
 

John Doe

Guru
Joined
Aug 16, 2011
Messages
635
sorry checked right now, VMs are not booting up, freenas is booting up.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
@Terydan / @John Doe : you have to configure ESXi to start particular VMs or start them from FreeNAS using the scripts in my resource here:

https://www.ixsystems.com/community/resources/utility-scripts-for-freenas-and-vmware-esxi.29/

You'll want to start the FreeNAS VM first/earliest, with FreeNAS issuing a command to rescan the datastores in its startup script.

Once the FreeNAS VM is running and its NFS datastore is available, you can then start your VMs that use NFS. Note that if you decide to use ESXi to start these VMs then you will need to explicitly configure it to start them, and may need to tweak the time delay between started VMs so that NFS will be available for them.

I find it more reliable to just start these from FreeNAS. That way there's no question whether or not the NFS service is available.
 

Terydan

Dabbler
Joined
Jan 19, 2017
Messages
31
@Terydan / @John Doe : you have to configure ESXi to start particular VMs or start them from FreeNAS using the scripts in my resource here:

https://www.ixsystems.com/community/resources/utility-scripts-for-freenas-and-vmware-esxi.29/

You'll want to start the FreeNAS VM first/earliest, with FreeNAS issuing a command to rescan the datastores in its startup script.

Once the FreeNAS VM is running and its NFS datastore is available, you can then start your VMs that use NFS. Note that if you decide to use ESXi to start these VMs then you will need to explicitly configure it to start them, and may need to tweak the time delay between started VMs so that NFS will be available for them.

I find it more reliable to just start these from FreeNAS. That way there's no question whether or not the NFS service is available.
I'll try out the settings I have now with my VMs that rely on NFS set to start manually, that way I can confirm that ESXI successfully rescans the data stores. Once that is working I'll apply the scripts you linked to in FreeNAS. I think using FreeNAS is the better way to start the VM like you said so that we know for sure NFS is available.
 

Terydan

Dabbler
Joined
Jan 19, 2017
Messages
31
So I rebooted my host system today, and had it set to boot my FreeNAS VM and no other. The script was set to rescan my datastores, but when it was all booted, I still couldn't see my NFS datastores, just show up as emptly. I manually ran the command
Code:
ssh <username>@<esx mgmt IP> esxcli storage core adapter rescan –all
from the FreeNAS Shell (FreeNAS 11.2U5) and I substituted root for the username and the proper IP for my Host ESXi install, but still no datastore. I had to manually remove and add it to get it to work. Does this command no longer force the rescan of NFS drives, or is there something else missing?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
"storage core adapter rescan" rescans block storage, not NFS.

"esxcfg-nas -r" restores NFS, but it is tagged with an INTERNAL USE ONLY flag which brings into question whether or not using it might cause some other problems.
 

Terydan

Dabbler
Joined
Jan 19, 2017
Messages
31
"storage core adapter rescan" rescans block storage, not NFS.
So that would only help if I was using iSCSI, is that correct? I am a blank slate when it comes to iSCSI so I haven't looked much into it. I took a peek when setting up my new AIO, but there was so much going on that I was unfamiliar with I just failed back to what I was comfortable with. The OP was specifically referring to NFS storage just like I'm using and that command was one of the proposed fixes.
I'll research the "esxcfg -nas -r" command and see if I can use it without any repercussions. Maybe in a commercial production environment with more features and services being used it may cause some errors, but for my smaller installation using only a fraction of ESXi capabilities, I hope the collateral damage will be nonexistent.
 
Top