Freenas as a VM with Fibrechannel-based Storage

[Jason Brooks]

Hello,

I have inherited a freenas system running as a vm under vmware. It is using vmdk files located on a Nimble storage server. the nimble storage system is connected to all of the esxi systms using fibrechannel. I really don't like that they did this. But I need to explore options. I'd like to get away from using the vmdk files.

The back-end filer (Nimble) only has fibrechannel Luns available, I don' t think it does iscsi.

freenas 11.1-u5
vsphere 6.7


I could:
Build a custom LUN to vmware and make them available to freenas with raw device mapping so only freenas would use them?
( I remember RDM was a problem but I wasn't certain it meant only with vmdk files....)
Does freenas (freebsd) support a virtual fibrechannel device so I could connect the LUN directly to freenas? (at least that way vmware would just pass the signals)

What do you all think?

Thank you for your time...

--jason

 

[HoneyBadger]

What do you all think?

... I don't think the forum etiquette allows for my initial knee-jerk reaction here, so I'll say "What the *heck* made someone decide to stand up FreeNAS as a virtual front-end to another SAN/NAS product?"

What was the original implementor's core idea/problem they were solving by standing up a FN VM in a hilariously unsupported (and potentially dangerous-to-your-data) configuration?

I know this might not be a popular answer here, but I'd be looking to eliminate the virtual FreeNAS entirely.

 

[Jason Brooks]

:)

I pretty much agree. But then most of the objections I have seen seem to be surrounding latency times, sync/asynchronous writes, and difficulties due to waiting on software to act like hardware on behalf of you...

Hence my question when it comes to direct fibrechannel...

--jason

... I don't think the forum etiquette allows for my initial knee-jerk reaction here, so I'll say "What the *heck* made someone decide to stand up FreeNAS as a virtual front-end to another SAN/NAS product?"

What was the original implementor's core idea/problem they were solving by standing up a FN VM in a hilariously unsupported (and potentially dangerous-to-your-data) configuration?

I know this might not be a popular answer here, but I'd be looking to eliminate the virtual FreeNAS entirely.

 

[HoneyBadger]

The only VM setup that gets real approval is an HBA with PCI passthrough. Direct RDM from a SAN will work better than a VMDK but it still introduces potential for weird cache interactions.

What is the FreeNAS VM currently serving, and why isn't it being handled by the Nimble itself?

 

[Jason Brooks]

Yeah, I remember as much regarding the HBA.

That's a good question about the Nimble: I've simply been told it doesn't do nfs or smb shares which is why the freenas system was allocated in the first place. I will probably build a linux system and export nfs for the moment to get out of the freenas wierd-zone. It's really kind of too bad anyway: I really like all that freenas has to offer.

Thanks!

--jason

The only VM setup that gets real approval is an HBA with PCI passthrough. Direct RDM from a SAN will work better than a VMDK but it still introduces potential for weird cache interactions.

What is the FreeNAS VM currently serving, and why isn't it being handled by the Nimble itself?

 

[HoneyBadger]

Yeah, I remember as much regarding the HBA.

That's a good question about the Nimble: I've simply been told it doesn't do nfs or smb shares which is why the freenas system was allocated in the first place. I will probably build a linux system and export nfs for the moment to get out of the freenas wierd-zone. It's really kind of too bad anyway: I really like all that freenas has to offer.

Thanks!

--jason

It's possible that it's an older Nimble CS unit; if I recall those could be configured in such a way that they were locked to SAN protocols only, and the Ethernet ports would only serve the management UI.

Unfortunately I think I have to concur on using a non-ZFS solution here; I'd hate to have you get a bad impression of the ZFS filesystem or FreeNAS by being in an unsupported and "weird" setup.

But hey, if they get some spare bare-metal machines, build up a testbed and you'll probably be surprised with how well it compares to the Nimble box ... ;)

 

[Jason Brooks]

Hello,

I have a follow-up question: I took a deeper look at the aforementioned Freenas virtual server. I also found out there is a second one in another datacenter. Sigh.

But the details boil down to: it's not using raidz or mirroring: Each system has five Vdevs (vmdk images provided by esxi). but they are configured as simple pools rather than using mirroring or raid. Is there any reason to assume they are more stable or am I still looking at potential catastrophy?

Again, thanks for your help!

--jason

 

[HoneyBadger]

Welcome back.

But the details boil down to: it's not using raidz or mirroring: Each system has five Vdevs (vmdk images provided by esxi). but they are configured as simple pools rather than using mirroring or raid. Is there any reason to assume they are more stable or am I still looking at potential catastrophy?

That's about what I expected, honestly; if they're just using FreeNAS as a NAS front-end with a friendly UI, you'll just rely on the Nimble unit to deliver the redundancy, and it will be inherited from the VMFS that the VMDKs reside on. I suppose this makes it friendlier to migrate around, as you can use svMotion to move the underlying vdevs ... but you've got data swinging in the breeze certainly.

The admin who set it up this way effectively removed ZFS's ability to correct corruption - it will still detect mismatched checksums, but without any redundancy at the ZFS level it has no ability to correct it.

You could try including a virtual FC device with the FreeNAS VM, leaving it in initiator mode, and presenting RDMs to it from the Nimble SAN - but you're still going to be left with some oddities no doubt. It's definitely well outside the more appliance-like usage of FreeNAS though, so you might find that the benefits of the included UI aren't worth some of the limitations it imposes; a stripped-down Linux or FreeBSD install of your own favorite flavor might be a better option. Present an RDM from the Nimble, attach it to the NAS-Filer-VM, throw ext4/LVM on top, and make some NFS exports.