VMware ESXi 5.5 and FreeNAS 8.3 Incompatible iSCSI Implementation

[yottabit]

The CF boot on my ESXi 5.1 server died today. While rebuilding I decided to upgrade to ESXi 5.5. I'll never get those 5 hours of my life back.

Anway, to the point: the iSCSI implementation between ESXi 5.5 and FreeNAS 8.3 is broken. I was using the release build of ESXi 5.5 and FreeNAS 8.3-RELEASE-p10.

The iSCSI targets were previously working just fine with ESXi 5.1. There were no changes on the FreeNAS side but the iSCSI targets would not be seen by ESXi 5.5, whether doing dynamic (discovery) or static defined targets. The targets were seen by discovery, but were never attached. I never saw anything suspect in the logs of either system.

I also played with security settings. I was using CHAP, but I also tried no authentication and also Mutual CHAP, all without any success.

Finally, not wanting to upgrade FreeNAS yet, I downgraded ESXi back to 5.1 and it works as it should.

I searched the FreeNAS support tickets and I did not see anything related to this issue. I will submit this as a bug but I wanted to give some time for feedback from anyone else here that may (or may not) have this same problem.

Thanks!

 

[cyberjock]

How about you post the real FreeNAS version. There was no 8.3-p10, so I know that is wrong.

As for your ESXi problem that seems to be squarely in the "ESXi has a problem" field, so I'm not sure why you are posting here and not the ESXi forums. It seems they'd be more likely to handle your problem since you proved that 5.1 works fine but 5.5 does not.

 

[yottabit]

How about you post the real FreeNAS version. There was no 8.3-p10, so I know that is wrong.

Sorry, that was uname -a. FreeNAS version is FreeNAS-8.3.2-RELEASE-x64 (r12686+70c2cb8).

As for your ESXi problem that seems to be squarely in the "ESXi has a problem" field, so I'm not sure why you are posting here and not the ESXi forums. It seems they'd be more likely to handle your problem since you proved that 5.1 works fine but 5.5 does not.

Well, for starters:

• FreeNAS 8.0RC, iirc, did have an iSCSI problem breaking interoperability with ESXi and that was fixed.
• I have been active in the FreeNAS community because users and developers are generally helpful and friendly, and some folks at ixSystems actually check these forums.
• I have no VMware paid-for licenses, they do not check their forums, and I don't feel like engaging them to report anything since they'll blow me off.

The whole point of posting here before opening a ticket was to get more input and feedback, not to take snide comments. If I misconstrued your comments, I'm sorry.

I will eventually test ESXi 5.5 and FreeNAS 9, but I'm not there yet.

At the very least this post might help others that were in my situation rapidly losing hair because things don't work when they should. I'm trying to help; not point blame.

 

[cyberjock]

It wasn't snide. I'd prefer to direct you to the proper place for support if support here is not likely to help.

Most people here aren't using 5.5 because of the weird and screwed up licensing with 5.5. If you aren't wanting to spend money on ESXi you should go back to 5.1 right now. After the 60 or 90 day trial is up you'll have to pay for the management VM, and you'll be very unhappy with the cost. :D

I'm a free-version only user(disabled vet.. so I can't afford upgrades) and I put in a ticket with their forum service and amazingly I did get some good support! I felt the same way, I didn't want to engage them on something that I didn't pay for, but they called me one day about my ESXi experience and I explained to them my problem and they said to try their "community". Sure enough, it worked.

 

[cyberjock]

Also, if you put a ticket in, I think you are wasting your time. No further development on 8.x is going on, so the first thing that will be said in your ticket is "does this exist in 9.2.0?". I'd upgrade to 9.2.0 before putting in a ticket. Partly because iSCSI has been overhauled in 9.2.0 and is supposed to be faster now that it is a kernel service.

 

[yottabit]

Cool, thanks for the tip! I'll stay with 5.1 then. I only upgraded to 5.5 because it was there, haha. Either way, I can only get my environment back online at the moment with 5.1 anyway, so...

I'll check the new features, and if 5.5 has something I want, I'll engage their user community about the problem (if it doesn't work with FreeNAS 9 that I will eventually get around to)...

Thanks again!

 

[Laszlo Szabo]

Hi,

Just few comments regarding to the ESXi licenses...
@ cuberjock: "After the 60 or 90 day trial is up you'll have to pay for the management VM, and you'll be very unhappy with the cost. :D"

ESXi 5.1 and 5.5 are still free. People overlook when they register on VMware site. Try to regsiter a new account then download ESXi 5.5 and you will see the license on your myvmware site. Only thing you need to do is upload the free license onto your ESXi 5.5 server and you will have your server.
With an old account is quiet hard to find the free license when you try to get it. (not sure why, to buy the full version???)

I have got 5 VMware ESXi 5.5 server in production running more than 90 days now and I have not paid a penny. ;)
Anyhow if you leave the licence period run out and don't apply the free license earlier then God knows what will happen.

So you can use 5.5 with out paying anything, that is for sure.

 

[gpsguy]

Yes, we know that.

The real issue kicks in if you upgrade the virtual hardware to version 10, you can't manage the VM. You can't edit the settings with the vSphere client. VMware wants you to use either the Web Client or vCenter, neither of which is free (after the 60 day trial). I believe one might be able to use Workstation 10, but again, it's not free.

I don't consider manually diddling around with the .VMX files to be a workable solution either.

As long as you keep the virtual hardware at version 8, you can manage a VM with the free vSphere client. But, you can't take advantage of new features that the version 10 offers.

I'm one of those that haven't upgraded to 5.5 yet.

So you can use 5.5 with out paying anything, that is for sure.

 

[jgreco]

gpsguy is correct, the management strategy for 5.5 is schizo even for paid vSphere sites (web client MUST be used for some things, legacy client MUST be used for others), and a total train wreck for free (the whole v10 virtual hw thing).

It is a total shock to those of us on a legacy 4.1 deployment where everything just works like it was actually designed by a competent designer.

vSphere 5.5 is very much the Windows 8 of virtualization products.

 

[ZFS Noob]

These comments are making me happy I chose XenServer oh so many years ago as my virtualization platform. It's not perfect, and it's not as full featured, but...

 

[Laszlo Szabo]

You guys are right, you can get this problem if you want to upgrade the HW version to 10.
Also I used to mess around with the vmx files that's not a problem, I needed for Microsoft SQL cluster quorum disk the shared SCSI disk solution ages ago. So If you lover the HW version number as you mentioned below 10, then after that you can edit the machine from vSpeher client.

But to be honest I have not faced this HW upgrade issue just yet and I have not upgraded any of my machines at all. (about 35 right now)
I'm pleased with the HW version 8. Windows 2012 and even Windows 8 works fine with it and bunch of Linux boxes. Of course I can't add usb 3 and SSD cache, and no more than 2TB disk, but don't need any of those. Features like virtual tablet??? ;) https://www.vmware.com/support/ws10/doc/workstation-10-release-notes.html

 

[Laszlo Szabo]

These comments are making me happy I chose XenServer oh so many years ago as my virtualization platform. It's not perfect, and it's not as full featured, but...

Hm I've implemented few of them for customers and XenServer has license features also.
Free license expires every 1 year as I know.

The best solution is KVM on Linux if you want real freedom, that's for sure I guess...

 

[jgreco]

These comments are making me happy I chose XenServer oh so many years ago as my virtualization platform. It's not perfect, and it's not as full featured, but...

XenServer just sucks in different ways.

VMware's strategy of only providing web management via VCS is really annoying, especially given that often when you are dealing with a problem you may want to cut VCS out of the loop and manage a hypervisor directly. Unfortunate. 4.1 was really nice except for the Windows based VCS and Windows based client.

 

[cyberjock]

vSphere 5.5 is very much the Windows 8 of virtualization products.

BURN! Too bad VMWare wasn't here to feel the heat!

 

[stoooo]

Here's an interesting article on the ESXi 5.5 and iSCSI targets problem... http://www.vaspects.com/2013/11/01/esxi-5-5-crashes-linux-iscsi_trgt/. I know it discusses Linux rather than BSD, but the issue may be the same.

I am currently runing ESXi 5.0 against FreeNAS 8.0.4, and all is well. However, I would like to upgrade to ESXi 5.5 this year, so I came to the FreeNAS site today to try and find what version of the iSCSI target are in use in which versions of FreeNAS. Anybody know ? Or can point me in the direction of a list of such things ? Thanks.

 

[jgreco]

Awesome. I had tried integrating a 5.5 box into our 4.1 environment late last summer and it had plowed a BUNCH of iSCSI SAN all to hell. I had never gotten around to actually researching the issue but the big 5.5 box had been mostly off in a corner quarantined from the shared storage. Thanks VMware...

 

[jgreco]

Here's an interesting article on the ESXi 5.5 and iSCSI targets problem... http://www.vaspects.com/2013/11/01/esxi-5-5-crashes-linux-iscsi_trgt/. I know it discusses Linux rather than BSD, but the issue may be the same.

I am currently runing ESXi 5.0 against FreeNAS 8.0.4, and all is well. However, I would like to upgrade to ESXi 5.5 this year, so I came to the FreeNAS site today to try and find what version of the iSCSI target are in use in which versions of FreeNAS. Anybody know ? Or can point me in the direction of a list of such things ? Thanks.

And as of right now FreeNAS 9.2.0 appears to be using istgt version 0.5, extra 20121028. Many of the FreeNAS 8's used istgt version 0.4, extra 20111008. Someday soon that will be replaced with the kernel iSCSI target code. https://bugs.freenas.org/issues/3811

 

[cyberjock]

Wait, what was all this talk about 9.2.0 having the kernel iscsi in IRC? Was that all crap? I could have sworn it was in the release notes. Am I out to lunch? Now to go read as that's not what's been disseminated to the masses. Note that I'm using iscsi for testing on my machine only so I really don't care about iSCSI performance, but I can respect the fact that it does affect others.

 

[jgreco]

It possibly does have the kernel iSCSI, but that doesn't mean it necessarily uses it at this point.

 

[jgreco]

I've got one of our spare 4.1 hypervisors and the 5.5 hypervisor both running data to and from a FreeNAS 9.2.0 iSCSI target and VMFS3. I am not noticing any problems, though admittedly I am not making it hard on the system (the 5.5 is hosting some crap so I can't reboot it, which is one of the times I expect it might go shooting fancy S.M.A.R.T. commands out its arse).