Cannot access FreeNAS server from a vm on separate ESXi server

[brentk]

I would be grateful if someone could shed some light on why I cannot access FreeNAS or a FreeNAS share from a virtual machine on a separate VMWare ESXi host.

Hardware Config:
FreeNAS server:
Motherboard make and model: ASROCK B85M PRO4
CPU make and model: intel Core i5 I5-4590S
RAM quantity: 32GB Corsair CMX16GX3M2A1333C9
Hard drives, quantity, model numbers, and RAID configuration, including boot drives: 6 WD Red 4TB, lz4
Hard disk controllers: SATA
Network cards: On board B85M PRO4, no interfaces

ESXI server:
Motherboard make and model: Supermicro X11SSM-F
CPU make and model: Intel Xeon E3-1230v6
RAM quantity: 64GB ECC CT16G4WFD8266
Hard drives, quantity, model numbers, and RAID configuration, including boot drives: 6 VMSF5 datastores
Hard disk controllers: SATA, Ubuntu vm is using SCSI, Windows vm is IDE
Network cards: On board X11SSM-F

Software Config:
VMWare ESXi 6.0.0, 3620759
FreeNAS 9.10.2-u6

What does NOT work:
Ubuntu 20.04 virtual machine in vmware esxi: Cannot ping FreeNAS, cannot access NFS Shares on FreeNAS
Windows Server 2012 R2 virtual machine in vmware esxi: Cannot ping FreeNAS, cannot access Windows SMB Shares on FreeNAS
Cannot ping FreeNAS from the ESXI server or any vm within the ESXI server
Cannot ping the the ESXI server or any vm within the ESXI server from FreeNAS

What DOES work:
Ubuntu 20.04 server outside of vmware: No issues pinging FreeNAS or accessing NFS Shares on FreeNAS
Windows Server 2012 R2 server outside of vmware: No issues pinging FreeNAS or accessing Windows SMB Shares on FreeNAS
Windows 10 Destop outside of vmware: No issues pinging FreeNAS or accessing Windows SMB Shares on FreeNAS
Can ping FreeNAS from any server or desktop outside of ESXI server
Can ping any server or desktop outside of ESXI server from FreeNAS
Can ping any server or desktop (other than the FreeNAS server) from the ESXI server and from any vm within the ESXI server

I have not set up network interfaces on FreeNAS. Here is my Network Summary:
Name IPv4 Address IPv6 Address
em0 192.168.1.190/24
Nameserver
192.168.1.1
Default route
192.168.1.1

All servers and the desktop mentioned above have static ip addresses. I have not seen any MAC Address changes/updates occurring on the ESXI virtual machines.

The firewall on the ESXI server is disabled. At least I think it is:
[root@esxi:~] esxcli network firewall get
Default Action: DROP
Enabled: false
Loaded: true
[root@esxi:~]
I have also tried setting the Default Action to PASS but that did not make a difference.

Any attempt to mount a FreeNAS share from the Ubuntu vm hangs indefinitely. Any attempt to map a FreeNAS share from the Windows server vm returns "The drive could not be mapped because no network was found", however shares from other servers can be mapping (inside or outside of ESXi)
Tracert to or from ESXi to FreeNAS returns nothing. * * * *
I cannot find anything in FreeNAS logs, although I may not know where to look. I simply expanded the log at the bottom of the web UI.

Any help I can get on this issue would be most appreciated.

 

[Spearfoot]

I would be grateful if someone could shed some light on why I cannot access FreeNAS or a FreeNAS share from a virtual machine on a separate VMWare ESXi host.

Hardware Config:
FreeNAS server:
Motherboard make and model: ASROCK B85M PRO4
CPU make and model: intel Core i5 I5-4590S
RAM quantity: 32GB Corsair CMX16GX3M2A1333C9
Hard drives, quantity, model numbers, and RAID configuration, including boot drives: 6 WD Red 4TB, lz4
Hard disk controllers: SATA
Network cards: On board B85M PRO4, no interfaces

ESXI server:
Motherboard make and model: Supermicro X11SSM-F
CPU make and model: Intel Xeon E3-1230v6
RAM quantity: 64GB ECC CT16G4WFD8266
Hard drives, quantity, model numbers, and RAID configuration, including boot drives: 6 VMSF5 datastores
Hard disk controllers: SATA, Ubuntu vm is using SCSI, Windows vm is IDE
Network cards: On board X11SSM-F

Software Config:
VMWare ESXi 6.0.0, 3620759
FreeNAS 9.10.2-u6

What does NOT work:
Ubuntu 20.04 virtual machine in vmware esxi: Cannot ping FreeNAS, cannot access NFS Shares on FreeNAS
Windows Server 2012 R2 virtual machine in vmware esxi: Cannot ping FreeNAS, cannot access Windows SMB Shares on FreeNAS
Cannot ping FreeNAS from the ESXI server or any vm within the ESXI server
Cannot ping the the ESXI server or any vm within the ESXI server from FreeNAS

What DOES work:
Ubuntu 20.04 server outside of vmware: No issues pinging FreeNAS or accessing NFS Shares on FreeNAS
Windows Server 2012 R2 server outside of vmware: No issues pinging FreeNAS or accessing Windows SMB Shares on FreeNAS
Windows 10 Destop outside of vmware: No issues pinging FreeNAS or accessing Windows SMB Shares on FreeNAS
Can ping FreeNAS from any server or desktop outside of ESXI server
Can ping any server or desktop outside of ESXI server from FreeNAS
Can ping any server or desktop (other than the FreeNAS server) from the ESXI server and from any vm within the ESXI server

I have not set up network interfaces on FreeNAS. Here is my Network Summary:
Name IPv4 Address IPv6 Address
em0 192.168.1.190/24
Nameserver
192.168.1.1
Default route
192.168.1.1

All servers and the desktop mentioned above have static ip addresses. I have not seen any MAC Address changes/updates occurring on the ESXI virtual machines.

The firewall on the ESXI server is disabled. At least I think it is:
[root@esxi:~] esxcli network firewall get
Default Action: DROP
Enabled: false
Loaded: true
[root@esxi:~]
I have also tried setting the Default Action to PASS but that did not make a difference.

Any attempt to mount a FreeNAS share from the Ubuntu vm hangs indefinitely. Any attempt to map a FreeNAS share from the Windows server vm returns "The drive could not be mapped because no network was found", however shares from other servers can be mapping (inside or outside of ESXi)
Tracert to or from ESXi to FreeNAS returns nothing. * * * *
I cannot find anything in FreeNAS logs, although I may not know where to look. I simply expanded the log at the bottom of the web UI.

Any help I can get on this issue would be most appreciated.

How do you have the ESXi networking set up? Are all of these devices connected to a switch? Are they all on the same subnet?

 

[brentk]

All devices are on the same subnet (192.168.1.x) and connected via a Linksys SE3008 switch.

 

[Spearfoot]

Odd! What do you mean by this?

I have not set up network interfaces on FreeNAS. Here is my Network Summary:
Name IPv4 Address IPv6 Address
em0 192.168.1.190/24
Nameserver
192.168.1.1
Default route
192.168.1.1

Because it looks like you have set up interfaces on FreeNAS if some machines can ping it.

Anyway, your problem is a bit of a stumper!

I'm guessing that Linksys switch doesn't support VLANs, correct? If so, then that's not the cause of the problem.

What device is providing DNS and routing at 192.168.1.1?

How is networking configured on your ESXi server? Using one or more motherboard NICs on a single vSwitch? Or do you have multiple vSwitches?

 

[brentk]

I should have been clearer regarding: I have not set up network interfaces on FreeNAS. Here is a clarification:
From the FreeNAS web UI, I have not configured any Interfaces on the Interfaces tab in the Network section. i.e. ISCSI interfaces.
In addition, my Global Configuration tab only contains:
Hostname: freenas
Domain: local

<I'm guessing that Linksys switch doesn't support VLANs, correct? If so, then that's not the cause of the problem.>
I expect that the SE3008 switch does not support VLANs as well because Linksys support states that it is an unmanaged switch.

<What device is providing DNS and routing at 192.168.1.1?>
The router is connected to the switch is an tp-link AC3150:
Archer C3150 V2 Wireless MU-MIMO Gigabit Router

<How is networking configured on your ESXi server? Using one or more motherboard NICs on a single vSwitch? Or do you have multiple vSwitches?>
There are two motherboard NICS and a single vSwitch.

 

[Spearfoot]

The FreeNAS Network->Interfaces settings are not iSCSI-specific; that's where you inspect or specify the IP address of your FreeNAS server.

It's possible to assign your FreeNAS server an IP address via DHCP -- and I'm guessing this is how your FreeNAS server is obtaining an IP address -- but generally it's not a good idea. FreeNAS is a server, and servers need to be nailed down so they're easy to find, i.e., they really ought to always reside at a fixed IP address.

So I suggest you configure a static IP address on your FreeNAS server, on the same subnet as the rest of your machines (i.e., an address in the 192.168.1.0/24 subnet) using the FreeNAS Network->Interfaces dialog on the GUI (or from the console).

Then specify your gateway and nameserver (192.168.1.1) on the FreeNAS Network->Global Configuration dialog (or, again, from the console).

There's no point in trying to access Samba or NFS shares until you can ping the FreeNAS server, so ping is the command to start out with. Once you can successfully ping it, you can go on from there.

If your FreeNAS server's hostname is freenas.local then you should be able to do a DNS lookup on that name from any of your machines attached to your nameserver. In other words, you may need to tinker with your TP-Link Archer's DNS configuration to make this work. I don't know any specifics about the Archer, but most routers have a way to define entries in DNS so that machines on your network can resolve hostnames to IP addresses. This lets you use the easier-to-remember hostname instead of its IP address.

Your goal is to be able to run the command:

Code:

nslookup freenas.local

on a Windows machine and get the IP address of your FreeNAS server as the response. In the UNIX world, the dig command (or similar) should yield the same result.

With respect to the other machines on your network, you'll want to insure they all refer to the same gateway and name resolver. This is usually taken care of by the DHCP service if they're assigned an IP address via the DHCP service -- but you'll want to make sure.

 

[brentk]

Thanks for your response and help so far Spearfoot.
I already had IP reservations in the router for all servers and desktops. I've made the updates to the FreeNAS Networking as suggested:

I am a rookie when it comes to networking, so I expect the next piece is going to take me a while.
Here is what I have happening so far:
I still cannot ping freenas or freenas.local from servers inside ESXi

dig is working externally from all Linux servers, but not internally:
ubuntu@plex_ubuntu:/etc$ dig freenas.local

; <<>> DiG 9.11.3-1ubuntu1.12-Ubuntu <<>> freenas.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 42355
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;freenas.local. IN A

;; AUTHORITY SECTION:
. 3152 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2020071201 1800 900 604800 86400

;; Query time: 2 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Sun Jul 12 20:52:49 UTC 2020
;; MSG SIZE rcvd: 117

ubuntu@plex_ubuntu:/etc$ dig www.google.com

; <<>> DiG 9.11.3-1ubuntu1.12-Ubuntu <<>> www.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62173
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.google.com. IN A

;; ANSWER SECTION:
www.google.com. 179 IN A 216.58.217.36

;; Query time: 2 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Sun Jul 12 20:53:09 UTC 2020
;; MSG SIZE rcvd: 59

ubuntu@plex_ubuntu:/etc$

nslookup is not working internally or externally from the Windows servers/desktop.
C:\Users\Brent>nslookup freenas
Server: UnKnown
Address: ::

*** UnKnown can't find freenas: No response from server

C:\Users\Brent>nslookup freenas.local
Server: UnKnown
Address: ::

*** UnKnown can't find freenas.local: No response from server

C:\Users\Brent>ping freenas.local

Pinging freenas.local [192.168.1.190] with 32 bytes of data:
Reply from 192.168.1.190: bytes=32 time<1ms TTL=64
Reply from 192.168.1.190: bytes=32 time<1ms TTL=64
Reply from 192.168.1.190: bytes=32 time<1ms TTL=64
Reply from 192.168.1.190: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.1.190:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Users\Brent>ping freenas

Pinging freenas.local [192.168.1.190] with 32 bytes of data:
Reply from 192.168.1.190: bytes=32 time<1ms TTL=64
Reply from 192.168.1.190: bytes=32 time<1ms TTL=64
Reply from 192.168.1.190: bytes=32 time<1ms TTL=64
Reply from 192.168.1.190: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.1.190:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Users\Brent>nslookup www.google.com
Server: UnKnown
Address: ::

*** UnKnown can't find www.google.com: No response from server

C:\Users\Brent>

 

[Spearfoot]

Ah, you never said you'd set up DHCP reservations for everything! With those, you should have been able to resolve hostnames: usually you can enter a hostname for a DHCP reservation. Now I wonder if any of your systems have a DNS server at all in their network configuration.

It sure looks like your Windows PC doesn't have a DNS server configured: it can't resolve www.google.com.

At least you can ping your FreeNAS server.

Why don't you paste the output of ipconfig /all into code tags and let's see how your Windows networking is configured.

Here's the output from my workstation. See how there is a default gateway and a DNS server? You seem to be missing at least the DNS server.

Code:

Windows IP Configuration

   Host Name . . . . . . . . . . . . : onyx
   Primary Dns Suffix  . . . . . . . : spearfoot.net
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : spearfoot.net

Ethernet adapter Local Area Connection 6:

   Connection-specific DNS Suffix  . : spearfoot.net
   Description . . . . . . . . . . . : Intel(R) Ethernet Server Adapter X520-1
   Physical Address. . . . . . . . . : 90-E2-BA-23-18-F4
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.16.10.40(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 172.16.10.1
   DNS Servers . . . . . . . . . . . : 172.16.10.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

 

[brentk]

I see that your workstation is part of a domain. My Windows PC and servers are not part of a domain, but instead are configured with a Workgroup and sharing that way. I tried updating the DNS settings on the IPV4 adapter to 8.8.8.8 and 8.8.4.4, but still nslookup did not work. If I try nslookup www.google.com 8.8.8.8 then I will get results. Should I be taking the time to setup a DNS server and domain or is it good enough just to update the hosts file on each Windows and Linux box? Even still, I cannot ping my freenas server from within ESXi or from the ESXi server itself, which is my biggest concern.

Here is the config of the Windows server outside of ESXi:
C:\Users\Administrator>ipconfig /all Windows IP Configuration Host Name . . . . . . . . . . . . : CasaVega Primary Dns Suffix . . . . . . . : Node Type . . . . . . . . . . . . : Hybrid IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No Ethernet adapter Local Area Connection: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Realtek PCIe GBE Family Controller Physical Address. . . . . . . . . : 54-04-A6-69-7C-D5 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::30f9:f6f1:4a3d:8a0c%11(Preferred) IPv4 Address. . . . . . . . . . . : 192.168.1.170(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Lease Obtained. . . . . . . . . . : Sunday, July 12, 2020 12:42:14 PM Lease Expires . . . . . . . . . . : Thursday, August 19, 2156 12:00:46 AM Default Gateway . . . . . . . . . : 192.168.1.1 DHCP Server . . . . . . . . . . . : 192.168.1.1 DHCPv6 IAID . . . . . . . . . . . : 240387238 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-16-4F-BB-BC-54-04-A6-69-7C-D5 DNS Servers . . . . . . . . . . . : :: 8.8.8.8 8.8.4.4 NetBIOS over Tcpip. . . . . . . . : Enabled

Here is the config of the Windows server inside of ESXi:
C:\Documents and Settings\Administrator>ipconfig /all Windows IP Configuration Host Name . . . . . . . . . . . . : casavm Primary Dns Suffix . . . . . . . : Node Type . . . . . . . . . . . . : Unknown IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No Ethernet adapter Local Area Connection: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection Physical Address. . . . . . . . . : 00-0C-29-02-1B-58 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes IP Address. . . . . . . . . . . . : 192.168.1.171 Subnet Mask . . . . . . . . . . . : 255.255.255.0 Default Gateway . . . . . . . . . : 192.168.1.1 DHCP Server . . . . . . . . . . . : 192.168.1.1 DNS Servers . . . . . . . . . . . : 192.168.1.1 Lease Obtained. . . . . . . . . . : Saturday, July 11, 2020 12:39:20 PM Lease Expires . . . . . . . . . . : Monday, January 18, 2038 8:14:07 PM C:\Documents and Settings\Administrator>

Here is the Windows 10 PC:
C:\Users\Brent>ipconfig /all Windows IP Configuration Host Name . . . . . . . . . . . . : D34010WYK Primary Dns Suffix . . . . . . . : Node Type . . . . . . . . . . . . : Hybrid IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No Ethernet adapter Ethernet: Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Intel(R) Ethernet Connection I218-V Physical Address. . . . . . . . . : EC-A8-6B-FD-02-06 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::5010:cc8e:d3e3:7cbe%23(Preferred) IPv4 Address. . . . . . . . . . . : 192.168.1.168(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Lease Obtained. . . . . . . . . . : Friday, July 10, 2020 5:47:21 AM Lease Expires . . . . . . . . . . : Thursday, August 19, 2156 12:09:26 AM Default Gateway . . . . . . . . . : 192.168.1.1 DHCP Server . . . . . . . . . . . : 192.168.1.1 DHCPv6 IAID . . . . . . . . . . . : 82618475 DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-1B-E7-44-7C-EC-A8-6B-FD-02-06 DNS Servers . . . . . . . . . . . : :: 192.168.1.1 0.0.0.0 NetBIOS over Tcpip. . . . . . . . : Enabled

 

[Spearfoot]

My systems are configured on a Workgroup like yours; I'm not running a domain controller. I do own the spearfoot.net domain name so I use it for my local systems.

I use pfSense as my router/gateway/firewall and it provides DNS and DHCP services.

Your network settings look fine; the problem seems to be that you're not getting DNS services from your TP-Link Archer at 192.168.1.1 and that is where you need to look. How do you connect to your ISP? And how is DNS configured on the Archer itself? It should forward DNS requests to external DNS servers you've configured if it can't resolve them locally (e.g., for IP allocations you've set up), and that is simply not happening.

Perhaps you need to go to a TP-Link forum and ask for help there on configuring your Archer.

A question about your Windows VM: are you passing the network adapter through to the VM? Because your ipconfig output shows an "Intel(R) PRO/1000 MT Network Connection" while my virtual machines always show a VMware virtual NIC, in most cases a "vmxnet3 Ethernet Adapter". I'm running ESXi 6.7 and if I recall correctly, you're using 7.0; has VMware changed how they name virtual NICs for virtual machines?

 

[brentk]

Thanks. I'll go through the TP-Link forum and the settings on my router. I'll also consider mirroring your approach and using a domain I have and pfSense, but that's definitely a project.

Regarding the Network Config on the VM, I am using ESXi 6.0 and here is what my Windows VM config looks like:

Here is my ESXi Network Config:

 

[Spearfoot]

All of your ESXi settings look good, too. And I see you're using 6.0, not 7.

I'm going to have to punt on this one as I don't know anything about your Archer router, which seems to be crux of your problems.

Perhaps another, more knowledgeable forum member will stop by and tell us what we've missed!

 

[brentk]

Spearfoot - thank you for you insight, suggestions, and checking my ESXi network config. Much appreciated. It's good to know that you are accessing your FreeNAS from ESXi 6.7 VMs. I am still investigating and looking at the router config. It's also interesting that we have a difference with our ipconfig Description on our VMs.

 

[brentk]

Issue Resolved.

Several months ago, I swapped motherboards between the freenas and esxi servers - needed more memory for a vm on esxi.

The esxi config still had the MAC address of the previous MB in /etc/vmware/esx.conf which was conflicting with the freenas MAC address.

I found this knowledge article and updated /etc/vmware/esx.conf to the correct MAC address - VMware Knowledge Base

Everything is working as expected now.