Esxi 6.7 + iSCSI zvol share - Immediately shows "Normal, Degraded" in Vsphere

[SmoothRunnings]

I saw a recent post for this issue here from 2017 which explained that multipath was the issue in ESXi 6.5. When I created another VMKernel NIC within the same subnet the FeeNAS iSCSI continues to only show up as 1 device. Does anyone have any other suggestions?

I can create a datastore and looks like I can use it, I just would like to make sure there are no headaches down the road with the Normal, Degraded device issue.

Thanks,

 

[kdragon75]

FeeNAS iSCSI continues to only show up as 1 device

This is normal. ESXi will see multiple paths but knows its only one device.
Are you working in vCenter or the ESXi Web Client? Essentially you want to look at the device and then check that there is more than one path.

Also don't forget to setup NIC binding on the iSCSI adapter in ESXi for each vmkernel (and interface, you have more than one right?) . It's a little tricky if you have never done it before but it not to hard.

On a side note, its technically incorrect to use multiple VMK ports on the same subnet. If only working with one layer 2 network, it's advisable to use VLANS to separate the iSCSI networks and have matching VLAN interfaces on the FreeNAS side. Ideally FreeNAS would have at least three NICs, two for iSCSI and one for management.

 

[SmoothRunnings]

This is normal. ESXi will see multiple paths but knows its only one device.
Are you working in vCenter or the ESXi Web Client? Essentially you want to look at the device and then check that there is more than one path.

Also don't forget to setup NIC binding on the iSCSI adapter in ESXi for each vmkernel (and interface, you have more than one right?) . It's a little tricky if you have never done it before but it not to hard.

On a side note, its technically incorrect to use multiple VMK ports on the same subnet. If only working with one layer 2 network, it's advisable to use VLANS to separate the iSCSI networks and have matching VLAN interfaces on the FreeNAS side. Ideally FreeNAS would have at least three NICs, two for iSCSI and one for management.

I have 1 interface per ESXi host.

ix0 - 192.168.90.100 - ESXi Host 01
ix1 - 192.168.100.100 - ESXi Host 02

I have both IPs added to the portal under iSCSI, Chap is enabled, etc.

I have never had to this before. My IBM DS3524 SAN does it automatically, just like previously my EqualLogic and MD3200's did automatically. :(

 

[kdragon75]

ok, so is each subnet on its own vlan? in ESXi each vmk port should be "assigned" only one NIC. In the port group view you should have a portgroup for each vmk port and on the port group select the NIC that corresponds to the correct vlan/subnet. Once thats dont back in the ESXi iSCSI configuration you can add the port bindings. this is all to insure we don't try to route to FreeNAS 192.168.100.100 from ESXi 192.168.90.xx and create extra, sub optimal (going over a router) paths.
Take your time and verify all of your settings on both ends.
On a side note, I don't bother with static IPs for my ESXi hosts interfaces for iSCSI. Just DHCP that nonsense!
On another side note, I use a pfSense router at home and that makes it super easy to manage firewall rules, vlans, DHCP, and more than most power users will ever need. Just another fun project for you.

 

[SmoothRunnings]

ok, so is each subnet on its own vlan? in ESXi each vmk port should be "assigned" only one NIC. In the port group view you should have a portgroup for each vmk port and on the port group select the NIC that corresponds to the correct vlan/subnet. Once thats don't back in the ESXi iSCSI configuration you can add the port bindings. this is all to insure we don't try to route to FreeNAS 192.168.100.100 from ESXi 192.168.90.xx and create extra, sub optimal (going over a router) paths.
Take your time and verify all of your settings on both ends.
On a side note, I don't bother with static IPs for my ESXi hosts interfaces for iSCSI. Just DHCP that nonsense!
On another side note, I use a pfSense router at home and that makes it super easy to manage firewall rules, vlans, DHCP, and more than most power users will ever need. Just another fun project for you.

Each subnet is on VLAN 0, the ix0 and ix1 (10GbE) NIC connects directly from FreenNAS to the ESXI hosts. The traffic between the two subnets doesn't cross the regular LAN traffic. It's a switchless configuration.

So to clarify. Each subnet should have its own portal instead of one portal for everything correct?
There is no DHCP server on the two the two connections. Will what you said about just DHCP still work?

I run Windows Server 2016, AD (DNS/DHCP), FS, EXC, MGT, VCenter, Web in my environment. I use PfSense as well but for no DHCP or VLANs. If I do VLAN's on my network I used my Cisco switches.

 

[kdragon75]

Each subnet is on VLAN 0, the ix0 and ix1 (10GbE) NIC connects directly from FreenNAS to the ESXI hosts. The traffic between the two subnets doesn't cross the regular LAN traffic. It's a switchless configuration.

Ahh, that clears things up a bit! don't bother with setting VLANs.

So to clarify. Each subnet should have its own portal instead of one portal for everything correct?
There is no DHCP server on the two the two connections. Will what you said about just DHCP still work?

You can disregard my comments on DHCP since you are using point to point.

So just go each setting for both links. In your case I would create two portals on FreeNAS just to keep it easy to read. Make sure al MTUs match if you have ever fiddled with them (perf. 1500). Get the port binding working. Let us know what you try and how it goes.

 

[SmoothRunnings]

OK I am really stuck now. lol

Instead of explaining everything I am going to use pictures first.

I decided for this experiment I was going to disable CHAP knowing I could circle back and enable it.

Portals and settings:

Initiators (self explanatory):

Authorized Access (no in use right now):

Targets and settings:

Extents:

Associated Targets and settings:

Will create another post with my ESXi settings.

 

[SmoothRunnings]

VMKernel NIC settings:

When I add 192.168.90.100 to the Configure iSCSI adapter I don't get any errors and the server does scan after configuring the iSCSI adapter. The Freenas device never shows up under the Devices tab (doing the setup this way) but I do a "freenas-san" datastore with 0B available, 0B free.

Oh and when when i do this with CHAP enabled I get a unable connect to iSCSI error even though my CHAP username and password are correctly entered.

Any ideas?

 

[SmoothRunnings]

I got it working, well sorta.

Both servers have single connections to 192.168.90.xxx or 192.168.100.xxx and I can see the Freenas under devices in each host (I am connected directly to them) but the Freenas device still shows Normal, Degraded beside it.

 

[kdragon75]

Yeah You can't multipath with one nic.

 

[SmoothRunnings]

In this post, are they just adding another VMKernel instead of adding another physical nic?
https://forums.freenas.org/index.ph...ately-shows-normal-degraded-in-vsphere.55068/

I tried adding another VMKernel pointing it to another IP within the same subnet, it doesn't seem to improve anything.

Thanks,

 

[kdragon75]

you would want a vmk port on each subnet. The screen shot is only showing one kernel port on your vswitch

 

[SmoothRunnings]

Yeah You can't multipath with one nic.

The iSCSI is still usable right?

 

[kdragon75]

Yes. as long as that one path does not get cut your fine.

 

[SmoothRunnings]

Yes. as long as that one path does not get cut your fine.

I am stuck again. When I create a new datastore from one of the hosts it doesn't appear on the other host. How do I fix this?

Thanks

 

[kdragon75]

Have you rescanned for new VMFSes? If all the hosts are configured with access to the iSCSI target and in the same cluster under vCenter the VMFS should automatically mount. Verify that each host can see the iSCSI disk.

 

[SmoothRunnings]

Have you rescanned for new VMFSes? If all the hosts are configured with access to the iSCSI target and in the same cluster under vCenter the VMFS should automatically mount. Verify that each host can see the iSCSI disk.

I had to create the new datastore under VCenter for it to work properly.

The only last question I have is how do I improve the performance?

It looks like I am only getting 10MB I/O on my HDDs and about 110M over my iSCSI target port.

 

[kdragon75]

Considering I don't know anything about your hosts, testing method, or FreeNAS server, I couldn't guess. You may want to search the forums a bit.

 

[SmoothRunnings]

Considering I don't know anything about your hosts, network, or FreeNAS servers I couldn't guess. You may want to search the forums a bit.

Hosts are IBM x3550 M3's 80GB of RAM, two x5675's, ESXi 6.7 runs off a mirror + hot spare 146GB 10k SAS drives, I have two of those.
My FREENAS is version 11U4 on a Dell R510, 64GB of RAM, 14 x 500GB SATA 7200RPM drives Raidz2, INtel dual port 10GbE Fiber with FW 2.4.16 installed. Single port cards are installed on my IBM's with the same FW revision. And all the servers are connected over multimode fiber. The Transceivers were recommended by my work, we use a lot of Intel 10GbE in our data centers.

Thanks,

 

[kdragon75]

Hosts are IBM x3550 M3's 80GB of RAM, two x5675's, ESXi 6.7 runs off a mirror + hot spare 146GB 10k SAS drives, I have two of those.
My FREENAS is version 11U4 on a Dell R510, 64GB of RAM, 14 x 500GB SATA 7200RPM drives Raidz2, INtel dual port 10GbE Fiber with FW 2.4.16 installed. Single port cards are installed on my IBM's with the same FW revision. And all the servers are connected over multimode fiber. The Transceivers were recommended by my work, we use a lot of Intel 10GbE in our data centers.

Thanks,

You beat me to my edit. What can you tell me about your testing methods? Also is your iSCSI vdev set to sync always? Because it should be. It will also kill write performance.