[Question] Solution besides FreeNAS for ZFS?

Status
Not open for further replies.

supercouille

Dabbler
Joined
Jul 15, 2013
Messages
11
So i'm looking at a solution to run a zpool that would be accessible to VMs running on the same machine.

Basically my plan is to boot on a OS that is ZFS-friendly and then run a software in that os to start VMs that will use the ZFS pool for data.

Do you think there is anything out there to help me achieve this?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
What OS do you plan to run that is ZFS friendly? If its FreeNAS I don't know how you plan to run other OSes aside from the jail(FreeBSD). If you plan to run ESXi, that's discussed in the http://forums.freenas.org/threads/a...ide-to-not-completely-losing-your-data.12714/ (warning #4 at the bottom)

The threads you should read through before going further are

http://forums.freenas.org/threads/p...nas-in-production-as-a-virtual-machine.12484/
http://forums.freenas.org/threads/a...ide-to-not-completely-losing-your-data.12714/

Running FreeNAS in a VM is risky. Plenty of people that claimed to have experience with ESXi and virtualization have lost their pools. If you choose to virtualize be sure you understand the risks.

And thanks for posting in off-topic. That was an excellent place for this question. :)
 

supercouille

Dabbler
Joined
Jul 15, 2013
Messages
11
Well the plan is to not virtualize the ZFS pool but to use software virtualisation inside the OS that is going to build the ZFS pool. I don't know if I'm writing this correctly but I don't really want to virtualize the main OS, just want to run some other OS in virtualisation.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
There is no option to run anything except the FreeBSD jail at the present time as a VM on FreeNAS. There are plans to add support for Linux as a VM in the future, but there is currently no ETA for that feature.
 

supercouille

Dabbler
Joined
Jul 15, 2013
Messages
11
Basically a FreeNAS box is *only* a freenas box as of now, right?
Is there anything else that will allow my to run a ZFS pool that is reliable besides FreeNAS?
The only other I know of is ZFS On Linux and I don't really know if it works well. Any advices?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
Yes, a FreeNAS box is only a FreeNAS box right now.

When it comes to ZFS, there aren't many options. ZFS is limited to FreeBSD and Linux. There is a OSX version of ZFS but I have no idea how reliable it is. ZFS on Linux worked well when i tested it but you must do all zpool manipulations from the command line.
 

supercouille

Dabbler
Joined
Jul 15, 2013
Messages
11
Well CLI is fine with me, I have currently a Raspberry PI set up as a download box all through SSH.

OK thanks, ill head towards linux zfs solutions and then I will find a way to virtualize windows in the linux environment. Do you think software VM is reliable? it is mainly to do scheduled backups and maybe run some dedicated game servers.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
software VM depends on alot of things. There's plenty of places where i'd never do VMs As such, that's strictly a personal opinion.

Keep in mind that ZFS on Linux is still a work in progress. Keep backups! Their forum has lots of users that have lost data.
 

supercouille

Dabbler
Joined
Jul 15, 2013
Messages
11
Do you think I should forget about ZFS altogether and focus on another software(or hardware?) raid solution? Is there anything out there that you would use besides ZFS?

(small question that is off-topic: can you 'brick' hard drives with raid or the worst that happens is lose data?)
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
Whether to use a hardware or software RAID, ZFS or non-ZFS is strictly a personal choice. I like ZFS and I have no plans to go back to hardware RAID in the future.

You can do things that wear out hard drives prematurely. Any data loss(or lack thereof) would be based on your backups, your choice of parity drives, and the health of your other disks.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Is there a compelling reason not to use something like ESXi and vmfs for basic VM storage?
 

supercouille

Dabbler
Joined
Jul 15, 2013
Messages
11
Well the main reason I'm building a server is to hold my data with redundancy. I bought some hardware without informing myself so i have this build that is overkill for only a data server. I want to use my server to its fullest so I want to host some VM for other purposes.

Now with esxi and vmfs I can't use FreeNAS and ZFS for redundancy (correct me if i'm wrong). Would I be able to use esxi with vmfs and then run a FreeNAS VM for my data outside of this box? Would that be reliable or is it just like any other FreeNAS virtualisation?

If I had a FreeNAS box and a ESXi box all problems would be solved but I was dumb now I have to find the best set-up for my hardware which is stupid. (fuck me)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
It is perfectly possible to have an ESXi host that runs FreeNAS in a VM to provide general filesharing services to the network. It is, however, a very bad idea to have ESXi use the FreeNAS instance as a datastore. ESXi wants its datastores available during startup; a FreeNAS VM will by definition be unavailable during ESXi startup, a paradox resulting in badness.

However, there are a bunch of specific issues related to virtualizing FreeNAS. Listed in two often-quoted posts. Basically you need to be suuuuuuuper careful and avoid clevering yourself to death; we have seen many people clever their bits into oblivion, which isn't what you want, right? I describe both the caveats I'm aware of and the general formula we use here with great success. If you do it "my way" it has the benefit of closely resembling an actual bare metal FreeNAS install, which is why I believe it to be safer than the random foo others have done resulting in bitloss.

I do not encourage you to try such a project without understanding the risks.
 

supercouille

Dabbler
Joined
Jul 15, 2013
Messages
11
Can you tell me more about the risks? Is it really that risky even with your method described in the post ? How would one clever himself to death?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
As cyberjock has repeatedly indicated to me, people will see it as a license to go ahead and do it and ignore one or more critical bits.

You really need some familiarity with both FreeNAS and ESXi in order to have a reasonable chance of success. You really need to understand the issues I outlined so that you can avoid them. I think there's probably a lot of room for improvisation, but there is also room for disaster.

For example, if you don't understand that ESXi might helpfully format all attached storage it sees as vmfs datastores during install, then you might make the mistake of setting up and transferring data to a FreeNAS system, dropping in a RAID controller for some datastores, then install ESXi, and find ESXi has helpfully just destroyed your pool. I don't think I even warn about that possibility, it's just something that I expect admins would be aware of.

But really, there's a history on the forums of people virtualizing their FreeNAS in what I would consider to be various ways that seemed to be clever (to them). And then things going wrong. So you make a virtual disk file on a bunch of ESXi datastores, thinking that you can then "ZFS" them together into a big pool. Well, yes, you can, and it works brilliantly. Until one day one disk starts to fail and ESXi gets tetchy about it (ESXi ... synonym for "not horribly tolerant of hard faults) and all you can tell at the VM level is that things seem slow. Then the disk dies, a week later. Now ESXi freaks and hangs the VM while trying to do disk I/O on its behalf. In the meantime you cannot even boot your FreeNAS VM because one of the datastore resources upon which it depends are not available. And replacing it? Wow, that becomes a bit of a logistical challenge, because you have to manage both VMware aspects and FreeNAS ZFS aspects to the recovery. So you screw it up, inadvertently messing up another disk on your RAIDZ1, and your pool is toast, and your data, maybe gone forever.

That's not even particularly clever. It is a straightforward, "obvious" implementation of FreeNAS with ZFS on top of ESXi in the manner that even an experienced admin might attempt. It will appear to (and indeed actually WILL) work great. Right up to the point where you would normally be getting SATA S.M.A.R.T. errors from the hardware that are trying to warn you of an imminent drive failure, which ESXi helpfully masks from you. Then all hell breaks loose, because most admins simply don't have experience dealing with complicated VM's with vmdk files on multiple nonredundant datastores.

And there are other risks. Going the PCI-passthrough route is a promising way to avoid some of the layering issues, but needs server-grade hardware, and relies on the motherboard, controller, ESXi, and the OS to all work together to support it.

Clevering oneself to death: Get a machine. Install ESXi. Install FreeNAS and a bunch of VM's on an NFS datastore hosted elsewhere. Configure FreeNAS to serve up NFS, thinking "I'm going to use ZFS to protect my VM's ... ZFS has great error protection features". True, dat. Set up FreeNAS to serve up an ESXi datastore. This *works*. Then storage vmotion all the VM's onto that datastore. This also *works*. Time passes. Now reboot. It's all gone, because you vmotion'ed the FreeNAS VM onto storage being provided by FreeNAS... wot a mess.
 
Status
Not open for further replies.
Top