NFS & VMWare - Loss of Connectivity Affected Datastores Unknown

[ggodar]

I have a Areca 1882i-4GB (4GB of BBWC) Raid Controller with 4 different Raid Sets

4 x Samsung 250GB Evo SSD (Raid 0)
9 x Western Digital Green 5400 RPM 2TB (Raid 5)
5 x Western Digital Black 7200 RPM 1TB (Raid 5)
5 x Seagate 15K SAS 146GB (Raid 5)

I am running Freenas off of a ADATA 32GB SSD

I have Freenas up and running with the latest and greatest stable release. I am running vmWare 5.0 Update 1

I provisioned ZFS volumes on Freenas
I provisioned NFS shares off of those ZFS volumes on Freenas
I provisioned CIFS shres off of those ZFS volumes on Freenas

From vmWare I can upload / download files to any of those NFS datastores
From Windows I can read/write/execute files to any of those CIFS datastores

When I try to install a virtual machine (windows or linux) to any of those datastores it will initially load the operating system disc (for install, which is also housed on the datastore in question) it will eventually comeback with either a blue screen of death (windows) or loss of disk (linux)

In vmWare i see the following error:

Lost connectivity to storage device blah blah blah. Path to blah blah blah is down. Affected datastores unknown

Is there something I have missed in the configuration or should have done differently?

I have thought about updating the Areca firmware, vmware ESXi version, etc but I don't really know what is causing the issue...

I have also thought about using UFS and seeing if that makes a difference

 

[ggodar]

No Replies? BTT

 

[cyberjock]

Well, you confused the hell out of me with your first post... first you mentioned a hardware RAID controller(which you should NEVER EVER EVER be doing with ZFS), then you mentioned hardware RAID configurations(which you should not have for ZFS).

But you have no information on your FreeNAS hardware, if you are virtualizing FreeNAS or not(Hint: If you are doing this you fail FreeNAS 101).

There's plenty of information that makes me think whatever you are wanting to do is nothing but an epic fail. But there's no solid evidence that what you are doing is smart or not. There's also no clue as to what you might have right or wrong. Since it works in VMWare without a VM I have to think it's a VMWare problem.

 

[ggodar]

cyberjock,

So I am not virtualizing Freenas it is on an SSD in the Norco 4224 box. I was using ZFS on top of hardware raid so epic fail there, my bad.

What would be my other options for presenting storage to VMWARE from the Norco box? Iscsi? UFS? Any recommendations?

 

[Nick2253]

You could use iSCSI, but NFS should work, so it would be wiser to fix your configuration, and then troubleshoot the NFS issue, because it's probably a deeper underlying issue than you think it is.

 

[ggodar]

Well wait didnt cyberjxock just tell me not to do NFS on Freenas on top of Hardware raid? I want to use Hardware Raid as I have an Areca 1880 controller with 4GB of caching 6x8087 sas ports into the backplanes on the Norco 4224.

 

[Nick2253]

No, you should not be using ZFS on top of hardware RAID. The whole reason to use ZFS is because ZFS is better than hardware RAID for redundancy and data reliability. You can't use both, because that screws up ZFS.

 

[ggodar]

Thanks sir for the clarification. So either ISCSI, or use something other than Freenas.

 

[ser_rhaegar]

Or switch to an HBA instead of a RAID controller. The m1015 is pretty cheap on eBay and extremely reliable.

Though that is just to fix the issue of ZFS on RAID. Your NFS issue might be related or it might be something different. I had that issue when running NFS on a ZFS mirror of two Crucial M500 480GB drives. But I don't have the issue on my RaidZ2 with 6 2TB spinners using NFS. I ended up returning the SSDs and going with iSCSI for now.

 

[ggodar]

Yea switching to something else is not really an option I just spent 1400 bucks on that Raid Card.

I had the NFS issue on 4 different Raid Sets but it only comes up when trying to run VM's off of the set. I can upload and download files fine.

I think I have two options:

Go with UFS.. Not sure what the advantages or disadvantages would be

Go with ISCSI..Not sure what the advantages or disadvantages would be

Go with Windows Storage Server ISCSI..Not sure what the advantages or disadvantages would be.

 

[Nick2253]

Thanks sir for the clarification. So either ISCSI, or use something other than Freenas.

I think you're confusing a few things. NFS is not ZFS. They are completely separate and distinct. It seems to me that you have the following issues with your setup:

1. Using ZFS on Hardware RAID. If you want to use ZFS, stop using hardware RAID. If you are dead-set on using hardware RAID, stop using ZFS and use UFS. You'll lose the redundancy/reliability benefits of ZFS in that case.
2. Using RAID5. You should read cyberjock's link in his signature: "RAID5/RAIDZ1 is dead".
3. VMware connectivity to your NAS. This could be caused by a VMware setting, bad hardware in your VMware or NAS box, or an issue with your NFS settings. It would behoove you to try the NFS connection from another machine, and try transferring lots of data to test connection reliability. I mean, I've seen problems like this caused by a bad network cable, so you need to rule out the connection before you start tearing things apart.

Also, when you say you are using the "latest and greatest" version, that's always up for change. Please confirm the actual version of FreeNAS you are using.

Yea switching to something else is not really an option I just spent 1400 bucks on that Raid Card.

I hate to be a negative Nancy about this, but it sounds like you don't really understand what FreeNAS is about if you go out and buy a $1400 RAID card. If your use case is as a VM storage device, that money could be well spent elsewhere to generate much more performance, by expanding your RAM, improving your processor, or adding cache drives. If possible, you might want to look into returning the card.

 

[ggodar]

Nick 2253,
Understood on all fronts. The article has a lot of interesting comments though to the contrary.

Freenas I am running 9.2.1.4

I have tried the NFS connection from different Raid Sets and from different ESXi hosts. Upload / Download works fine. It is when you try to run a VM off of the NFS share that I start seeing the connectivity issues.

I can't return the card. I have the Norco 4224 so I needed something that had 6 SAS ports to connect into the backplanes. I had a HP card that I fooled around with for around a month with an HP sas expander trying to upgrade it to recognize the drives. It was so much hassle and I have little time to devote to troubleshooting. I'm a busy person that travels a lot so I needed something that works out of the box. The Areca accomplished that and I am very happy with it.

I can still do UFS or ISCSI it sounds.

Sorry to seem like a noob but I am. I have done a lot of reading but I am new to the storage world outside of traditional desktop raid type stuff. I am an app guy getting into the other realms of networking & security. Appreciate all the feedback thus far.

 

[cyberjock]

Ok.. being a newb I'll simplify this down for you..

1. That hardware RAID controller has to go. Period. If you want to use FreeNAS with ZFS, it cannot stay. UFS is dead. FreeNAS 9.2.2 will have no UFS support, so if you go with UFS you are looking at never having any upgrade options, ever.

2. Trying to troubleshoot your performance problems before you replace that RAID controller with an HBA is pointless. You need to do #1 before you worry about performance. I can guarantee you that replacing the RAID controller will help performance. How much is a different question.

 

[Matt Vaughan]

Are you using ESXi 5.5 U1? If so, it's a known issue.

See: http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=2076392

 

[jgreco]

Are you using ESXi 5.5 U1? If so, it's a known issue.

See: http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=2076392

Wow. View: http://m.youtube.com/watch?v=OEFxX0dfVY8