2nd NIC for FreeNAS VM on ESXi 6.5 (AiO)

[cfcaballero]

I have an "All-in-One" (AiO) setup (details in my sig) like in this article referenced by many posts (like this) here on the forum.

I am ashamed to admit that when I got everything going, I skipped the part about creating the virtual storage network (an additional virtual switch to handle the NFS traffic from the ESXi host to the FreeNAS storage so the physical NIC and management network doesn't get saturated). Naturally, this becomes an issue when you add VMs to ESXi.

I'm now trying to rectify my initial mistake and can shutdown the FreeNAS VM and add the 2nd virtual NIC attached to the storage vSwitch. However, when I boot up the FreeNAS VM again after that, the first NIC no longer pulls a DHCP address from my LAN, just times out. Both NICs do show up when I use the VMware console, however. If I remove the 2nd virtual NIC, all works fine again.

Besides the obvious question of has anyone seen anything like this, I would love an opinion on troubleshooting approach.

Unfortunately, the only other machine I have available to test against (an MSI gaming laptop) doesn't have a NIC that ESXi supports, so I can't install it. I did install ESXi as a VM under VirtualBox on the factory Win10 install on the laptop, but I worry that any troubleshooting I do there might lead me down a rabbit hole because of the extra layer of virtualization.

This is normally the point where my brain would try and justify buying another computer, but I am trying to avoid that. The home office is too crowded already!

Thanks in advance for any ideas, tips, pointers, etc.

 

[Mirfster]

Some questions:

• Did you assign a Static IP to the VM Storage Kernel (on the Storage vSwitch)?
• Did you assign a Static IP to the 2nd NIC in FreeNas?
• I still run ESXi 6.0 U2. Don't like 6.5+ and FreeNAS 9.10... but here are some screenshots that may assist:

Edit: Forgot to add a screenshot of the FreeNAS VM Properties:

 

[cfcaballero]

Some questions:

• Did you assign a Static IP to the VM Storage Kernel (on the Storage vSwitch)?
• Did you assign a Static IP to the 2nd NIC in FreeNas?

First, thanks for the reply!

When I tried to add the Kernel NIC (with static IPv4) to the storage vSwitch, then the port group no longer shows up as a choice for the 2nd NIC on the VM. So, I thought I would try just getting the 2nd NIC working first in the VM.

I hadn't tried doing anything with the 2nd NIC in FreeNAS once I saw the first NIC was no longer pulling an IP. Will try and configure from the command line and see if that does the trick.

Sadly, it never occurred to me to check ESXi versions. All the posts I saw were 6.0, but they were old enough that I thought maybe just before 6.5 came out. I'm committed now, so will just have to barrel through.

 

[Mirfster]

IIRC only 1 NIC in FreeNAS can leverage DHCP, so that in itself may be the issue. May want to look into that.

Best of luck.

 

[cfcaballero]

IIRC only 1 NIC in FreeNAS can leverage DHCP, so that in itself may be the issue. May want to look into that.

Best of luck.

That may be just the lead I need. Thanks again! Will definitely report back in this thread once I sort it out.

 

[melloa]

That may be just the lead I need. Thanks again! Will definitely report back in this thread once I sort it out.

Also, if you have a open PCI slot, why not get a PCI NIC there in pass through? My ESXi has a Mellanox for its 10Gb vSwitch and a Chelsio in pass through. ESXI doesn't even see it as a interface as it doesn't have the drivers for it:

But it is there:

 

[cfcaballero]

Also, if you have a open PCI slot, why not get a PCI NIC there in pass through?

Thanks, but the issue isn't getting better network throughput out to the physical LAN, it's getting the VMs to connect to FreeNAS without clogging the same NIC as the physical LAN. Besides, I just have the motherboard NICs, no possibility for a PCI passthrough of just one.

 

[cfcaballero]

Update:

I changed the virtual NIC in the FreeNAS VM to static IP, all working well, then powered the VM down, added the 2nd NIC connected to storage vSwitch, and then, same problem. First NIC stops working, even when I configure the 2nd NIC for static via the console, reset networking, etc.

Either something very odd is going on here, or I am missing something obvious, or both . . .

 

[melloa]

Thanks, but the issue isn't getting better network throughput out to the physical LAN, it's getting the VMs to connect to FreeNAS without clogging the same NIC as the physical LAN.

My example with a 10Gb NIC is valid for any speed interface.

I just have the motherboard NICs, no possibility for a PCI passthrough of just one.

I'm sure that X10 has PCI slots and a NIC is around $18: 1Gb NIC

Either way, help me understand what you want to do: Have FN with two IPs/two vSwitchs? Have FN on vSwitchA and other VMs on vSwitch B?

 

[cfcaballero]

Either way, help me understand what you want to do: Have FN with two IPs/two vSwitchs? Have FN on vSwitchA and other VMs on vSwitch B?

Thanks, really appreciate the assistance.

It's an all-in-one setup, so I'm just trying to offload the NFS traffic between other VMs in ESXi and FreeNAS by putting it on a different vSwitch than the single physical NIC that handles traffic to the LAN. I grok how it should be done, but every time I add that 2nd NIC to the FreeNAS VM, the first NIC stops working.

I'm going to see if I can reproduce the problem in a virtualized ESXi this weekend.

 

[melloa]

FreeNAS by putting it on a different vSwitch than the single physical NIC that handles traffic to the LAN.

Ok, so, something like this:

Two vSwitches. In your case one dedicated to FreeNAS.

Start with checking your vSwitch0 to see what Physical NICs are being used. Create a second vSwitch with a unused one. If none are available b/c are being used as uplink, you will need to remove one from it. As long you keep the main one, the one obtaining the IP for your vSwitch, that shouldn't be a problem.

With an available NIC, just create a vSwitch and a Port group, assigned to it. Add that to the FN VM, play with the interfaces configuration to remove DHCP from the first one and add to the second one (I do find that function a lot confusing and it takes me several attempts to get it right).

When you have he 2nd one with an IP from the DHCP, shutdown the VM, remove he 1st vSwitch and test.

I never had a problem with a vSwitch stopping when creating a new one, by the way.

Good luck :)

 

[cfcaballero]

Start with checking your vSwitch0 to see what Physical NICs are being used. Create a second vSwitch with a unused one. If none are available b/c are being used as uplink, you will need to remove one from it. As long you keep the main one, the one obtaining the IP for your vSwitch, that shouldn't be a problem.

So does the switch dedicated to FreeNAS serving NFS to the other VMs need a physical NIC? I figured it didn't, as that traffic is never leaving ESXi. Anyway, will give it a try, thanks again!

 

[melloa]

So does the switch dedicated to FreeNAS serving NFS to the other VMs need a physical NIC? I figured it didn't, as that traffic is never leaving ESXi. Anyway, will give it a try, thanks again!

Take a look at this article and see what configuration will be the best for you: Understand How Virtual Machine Traffic Routes

 

[philosopher]

So does the switch dedicated to FreeNAS serving NFS to the other VMs need a physical NIC? I figured it didn't, as that traffic is never leaving ESXi. Anyway, will give it a try, thanks again!

I had this same problem adding a second vSwitch for my 10gbe traffic going to freenas. Both had static IP addresses but I couldn't get it to work properly. I ultimately had to passthrough my PCI 10gbe NIC to fix it.

 

[phier]

is there any fix for the issue? or passthrough is the only one solution?

 

[Crashdodson]

Ive found the issue with adding multiple vnics causes truenas to reorder the nics. So you have to get a shell and use ifconfig to compare the MAC addresses associated with the the VMX interface to the actual vnic in esxi. Once you see how they are mapped you can configure the NICs accordingly.