FreeNAS As a Mini-Data Centre?

[MichaelCropper]

Hi all, newbie to FreeNAS here. After doing an awful lot of research, I've a few questions before I jump in and get a bunch of hardware to power FreeNAS to attempt to configure what I'm aiming to achieve.

What I'm looking for is essentially a mini-data centre type infrastructure for personal use. I.e. A single tower PC (a beast of a PC with RAID, lots of RAM & HDD space, with FreeNAS running). Ultimately with this hardware, the primary requirements are mainly software based though.

I'm looking for a setup that is capable of running every point below securely inside individual Jails for each point;

1) Running OpenVPS
2) Running OwnCloud (which I believe is available as a FreeNAS plugin)
3) Pointing www.website-1.com to an Apache installation (with full flexibility of what software is installed on this instance)
4) Pointing www.website-2.com....as above, same flexibility.

So the question being, is this possible, and if so, how?

I'm also going to be running this on a home network initially which is fine for local access, but adds extra complexity with domain > IP mapping with multiple websites. I.e if I port forward 80 for HTTP and 443 for HTTPS towards the PC running FreeNAS within the main router, how would that work in practice at the FreeNAS level to determine which Jail to direct the traffic to, OpenVPN, OwnCloud, www.website-1.com, www.website-2.com etc.?

I've figured out the Dynamic IP address type technology which should be able to be overcome with something like No-ip.com which I have been experimenting with and seems to work well for a few other projects I've been working on.

And one final thought, how would all of this fit in with something like VMWare? Is this something that could fit on top of FreeNAS, or separate, or something different? I've not researched VMWare in a great deal of detail at the moment, so I can't really talk with a lot of confidence about this for now, although from what I do know, this seems to somehow fit within this kind of setup, somewhere.

For reference, I'm a technical guy, but more focused on the web technology side, not on the networking/hardware side so I'm still getting to grips with the the underlying hardware / infrastructure design aspects and how this all works in practice. Keen to learn a lot more about this and best practice etc.

 

[Stux]

Mini Data Centre: https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

 

[Dice]

And one final thought, how would all of this fit in with something like VMWare? Is this something that could fit on top of FreeNAS, or separate, or something different?

ESXi goes on the metal, other utilities on top.

If your hardware has dual NICs and a separate HBA you are setup for a 'whatever you want' style mini data center, including the possiblity to cheaply run advanced firewall/NAT/filtering/VPN functionalities virtualized since most of your intended machines run on the ...same machine ;)

I run ESXi on metal, put in pfSense for firewall, FreeNAS for running datastores to the VMs(pihole for adblocking, win7 for windows utilities that run daily) and my regular storage.
In your setup, I'd probably look to setup a simple linux VMs to run off ESXi to host your different www applications (if not want/cant be run on the same install).

Owncloud is no longer an official plugin, but replaced in NextCloud.

 

[adrianwi]

I always think of mine as a mini-data centre :D

These are all available from mine as external services via https behind a reverse proxy in FreeNAS jails - emby as a media server, owncloud for file sharing, openoffice for document collaboration, calibre as a e-book library, wordpress running multiple sites, and openvpn for secure access to my network. I have a few other services running internally on a mix of jails and VirtualBox VMs.

 

[Ericloewe]

The correct buzzword is "hyperconverged", because buzzwords generate more buzz if they're hyper.

 

[MichaelCropper]

Thanks for the information, I've had a good read through those links and learnt a lot there. This has raised a few more questions though.

1) VMWare / ESXi - Is this open source like FreeNAS? Or is this a premium product. I honestly can't nagivate their website at all, it is one of the most confusing I have ever come across with the various products / pricing / free etc.

2) If the recommendations is to put ESXi on the metal and FreeNAS on one of the VMs, I'm a little confused. My understanding, which could be incorrect, was that FreeNAS was a operating system that essentially creates a RAID system (Personally I'd probably look at RAID6 to ensure 2x redundant hard drives) which protects against hard drive failures. If this is the case, am I correct in thinking that the best approach is to use a VM setup (VMWare Hypervisor appears to be free), which sits on the metal (I've no idea how to even install / set this up yet, or how the VMWare software aligns with the underlying RAID6 hardware configuration, but let's go with the theory for now... - all of which by the way, I assume that VMWare manages the RAID6 setup?), then FreeNAS comes into the picture. If this is the case, what actually is FreeNAS in this picture. Is the best way of thinking about FreeNAS as a C:/ drive, which runs FreeNAS instead of Windows?

3) Backups. My initial thought was to create a duplicate system and use something like rsync to transfer data from a-->b, but I'm now not sure how this entirely works within the system design discussed above.

It is so long since I've played with the underlying hardware with computers, I feel I've lost touch a little, so I'm playing a bit of catch up here, please bear with me.

Feeling a bit confused at the moment. The more research I do, the more confused I seem to be at the moment, feeling a bit like Alice at the moment... tumbling down the rabbit hole

 

[Ericloewe]

I assume that VMWare manages the RAID6 setup

Not unless you want to lose data.

You should probably take a step back. Why do you want to virtualize things? Is any of the software you need not available for FreeBSD?

Now take another step back so that everyone can get on the same page. What is your day job - if irrelevant, what is your set of skills?
I ask because you're asking a number of questions that suggest you're in over your head and we need to be careful to structure our advice according to the water level.

3) Backups. My initial thought was to create a duplicate system and use something like rsync to transfer data from a-->b, but I'm now not sure how this entirely works within the system design discussed above.

Virtualization has mostly zero impact on how backups are made.

Is the best way of thinking about FreeNAS as a C:/ drive, which runs FreeNAS instead of Windows?

No, mostly because what you said makes little sense. The conflation of storage and processing also worries me somewhat, but I'll chalk it up to weird terminology for now.

FreeNAS provides Network-Attached Storage. Besides this main task, additional software can be run in jails and take advantage of the same storage and other resources.
All storage in FreeNAS uses ZFS, which takes a number of drives in certain configurations to provide reliable storage (keyword being reliable, unlike most filesystems), effectively replacing traditional hardware RAID controllers.

 

[adrianwi]

All of the things you mentioned in the OP (OpenVPN, ownCloud and Web Servers) can be done quite easily using FreeBSD jails inside FreeNAS, so no need for VMWare anything. If you do need to run something on FreeNAS that isn't available on FreeBSD, you can also run VM's on FreeNAS.

Creating a backup of a FreeNAS box is also very simple using ZFS Replication.

Personally, from what you've said you need it to do, VMWare is just adding an additional layer of complexity you don't really need.

 

[joeschmuck]

VMWare ESXi is a free product for home use. With that said, your experience is likely going to dictate if you use VMWare or just FreeNAS. If you use VMWare then you MUST be very careful and disiplined in how your run your system as you can destroy it easily. If you just stick with FreeNAS on bare metal then you don't have many of the pitfalls of running FreeNAS on VMWare ESXi. I run ESXi and FreeNAS on top of that so I do speak from experience.

My advice is to know exactly what you need to do and if FreeNAS can deliver that, which it soundls like it can do this for you based on your previouis postings.

Last piece of advice is to grab a copy of VMWare Workstation Player and install it on a Windows computer with lots of RAM and free hard drive space. Create a VM of FreeNAS and play with it. This is how a lot of people should be introduced to FreeNAS, this way you can play with the VM and setup jails to provide you the web servers and OpenVPN and OwnCloud. If you find that the FreeNAS VM can support what you want then you can buy the proper hardware to support this adventure and move forward.

 

[MichaelCropper]

Thanks everyone for the information, this is really useful in helping me get up to speed quickly. To answer a few of the questions above, and to ask a few more follow on questions...

Why do you want to virtualize things? Is any of the software you need not available for FreeBSD?

I'm looking to virtualise things for security reasons, so that if one system gets compromised, then this cannot infect the other systems running. And secondly so that this gives complete flexibility in the future, i.e. if one of the virtualised systems needs moving to it's own bigger box then it can be transferred easily. And thirdly, to ensure that every 'thing' that is running that is virtualised can run whatever software it needs, i.e. maybe a Linux Apache PHP MySQL setup on one VM, OwnCloud / NextCloud on another VM, a Windows Server machine on another VM (I've no need for this last one, just an example), etc.

I'm looking to virtualise either through a VM or a Jail - I'm not actually sure what the real difference is here at the moment thought? Is it that with a VM you state up front how much CPU/RAM the VM will have, whereas with a Jail, this is flexible? Or have I missed something?

What is your day job - if irrelevant, what is your set of skills?

Full stack software developer, web/online focused +10 years. Never really touched the hardware side in recent times though, the most advanced I got in this area was custom built PCs back in the day. So I've generally run systems on a LAMP stack on VMs, dedicate services, using linux based software from the command line etc. Again, very web focused, so all the hardware systems I've used have been in data centres which are managed infrastructures so I've had little visibility of this in practice hands on.

I ask because you're asking a number of questions that suggest you're in over your head and we need to be careful to structure our advice according to the water level.

I completely agree This is certainly not my area of expertise (yet), it's an area that I'm wanting to really up my skills in.

Virtualization has mostly zero impact on how backups are made.

I thought that was the opposite. My understanding was that through virtualised systems, backups at least when Linux is running on the VM, is handled at the VM level rather than the box level. Does this mean that backups are handled at the box level rather than the VM level then? And if so, through what mechanism / software / commands ?

The conflation of storage and processing also worries me somewhat, but I'll chalk it up to weird terminology for now.

Your assumptions are right, I'm not familiar with the difference here. In this world I am quite a newbie. So after reading about the dual NICs (which seem important/good), I also read about the separation of processing and storage but I've yet to fully understand what this actually looks like in practice with everything being inside one physical box. I believe that it is possible to extend FreeNAS to run over multiple boxes full of HDDs, but I've yet to understand the finer details of what this actually means, how this is set up and how this is all managed both at the software level and hardware levels - i.e. I take it that somehow with multiple boxes, the dual NICs on a single box are used to connect Box 1 to a Switch, Box 2 to the same Switch, then maybe have NIC 2 on both Box 1 and Box 2 connect to a different switch to ensure that if Switch 1 went down, Switch 2 would kick in? Does that sound about right?

All storage in FreeNAS uses ZFS, which takes a number of drives in certain configurations to provide reliable storage (keyword being reliable, unlike most filesystems), effectively replacing traditional hardware RAID controllers.

This bit I understood previously This was one of the main reasons for initially looking a solution and stumbling on FreeNas - As one of my hard drives has started to sound a bit wheezy and clearly is going to go at some point in the near future. Currently I use a variety of cloud based hosted solutions for file storage, but the monthly subscriptions soon add up - and I thought I could probably build a system that does all this for me with the right knowledge.

If you do need to run something on FreeNAS that isn't available on FreeBSD, you can also run VM's on FreeNAS.

This is an interesting question and one that is probably better as a reverse question for a newbie like me. What can you NOT do on FreeBSD? I only came across the term while researching FreeNAS so I'm not familiar with the finer details of that either.

Creating a backup of a FreeNAS box is also very simple using ZFS Replication.

That's good to know and answers one of the questions I mentioned above in this post. I'll do some more research around that to understand that further.

If you use VMWare then you MUST be very careful and disiplined in how your run your system as you can destroy it easily. If you just stick with FreeNAS on bare metal then you don't have many of the pitfalls of running FreeNAS on VMWare ESXi.

I assume here you're referring to accidentally deleting a VM which has all your data on and isn't easy to restore. If this is the case, that sounds like quite a risk for a newbie like me in this area, so I may just stick to FreeNAS as you suggest for simplicity.

Last piece of advice is to grab a copy of VMWare Workstation Player and install it on a Windows computer with lots of RAM and free hard drive space. Create a VM of FreeNAS and play with it. This is how a lot of people should be introduced to FreeNAS, this way you can play with the VM and setup jails to provide you the web servers and OpenVPN and OwnCloud. If you find that the FreeNAS VM can support what you want then you can buy the proper hardware to support this adventure and move forward.

That sounds like a sensible idea. I'm sure I can cobble together a reasonable PC from spare parts around the house to have a test of these things. Out of interest, is it possible to set this up on a Raspberry Pi to test? Or is that a little ambitious?

Overall, I'm still a little unsure how the setup outlined above translates into practice. From what I've picked up throughout the discussions so far, this is entirely managed by a reverse-proxy. So I'll go and have a good read up about those too. Lots of reading to do and I'm still working my way through all the topics / forum posts I can about this too. Lots to learn

Thanks for help to far, much appreciated.

 

[Ericloewe]

I thought that was the opposite. My understanding was that through virtualised systems, backups at least when Linux is running on the VM, is handled at the VM level rather than the box level. Does this mean that backups are handled at the box level rather than the VM level then? And if so, through what mechanism / software / commands ?

Okay, I see a source of confusion. You're thinking of the environment (OS + tools + etc.), which, in a virtualized environment, you'd backup by taking a copy of the storage device. However, your real data, the stuff that you're doing things to, shouldn't be mixed in with the OS and related bits.

For FreeNAS, specifically, if it were running virtualized, the boot pool would live on some hypervisor-provided virtual block device. Then, the actual data would live in a pool made up of disks attached to an HBA that is passed through to the VM via VT-d and/or related extensions.
To backup the former, you snapshot (not a ZFS snapshot, necessarily, I mean it generically) the virtual disk and back that up. To backup the actual data, you copy it elsewhere with zfs send | recv or rsync or something else.

Your assumptions are right, I'm not familiar with the difference here. In this world I am quite a newbie. So after reading about the dual NICs (which seem important/good), I also read about the separation of processing and storage but I've yet to fully understand what this actually looks like in practice with everything being inside one physical box. I believe that it is possible to extend FreeNAS to run over multiple boxes full of HDDs, but I've yet to understand the finer details of what this actually means, how this is set up and how this is all managed both at the software level and hardware levels - i.e. I take it that somehow with multiple boxes, the dual NICs on a single box are used to connect Box 1 to a Switch, Box 2 to the same Switch, then maybe have NIC 2 on both Box 1 and Box 2 connect to a different switch to ensure that if Switch 1 went down, Switch 2 would kick in? Does that sound about right?

No, you're overthinking things.

Forget the physical box. It's just a convenient enclosure. What matters is the CPU, DRAM and peripherals, mass storage being just another peripheral, of which you can add as much as you want, subject only to interface limitations. Mass storage is either provided via SATA, SAS or directly via PCIe (with the NVMe software interface) for high-end SSDs. SAS, in particular, can easily be extended outside the chassis as it is designed for this - so, you can run an SAS cable to an external chassis to add more disks. The additional chassis only contain disks, fans, the SAS backplane with expander chip, PSUs and, sometimes, a fancy power control board to provide monitoring and stuff.

Yes, you could do this over the network, but you don't, in practice.

This is an interesting question and one that is probably better as a reverse question for a newbie like me. What can you NOT do on FreeBSD? I only came across the term while researching FreeNAS so I'm not familiar with the finer details of that either.

It's a general-purpose OS on classical computer. It can do anything a Turing Machine can.

The better question, what software are you not willing to write and is not already available? Apparently, all the stuff you mentioned is available for FreeBSD.

I assume here you're referring to accidentally deleting a VM which has all your data on and isn't easy to restore. If this is the case, that sounds like quite a risk for a newbie like me in this area, so I may just stick to FreeNAS as you suggest for simplicity.

No, because it should never even come to that. VMs are not perfect and neither is the software running within. Combining the two without care is a great way to research just what causes how much data loss. Specifically for FreeNAS and ZFS, the big bullet point is using a real HBA that is passed through, instead of any kind of virtual disks for your data.

Out of interest, is it possible to set this up on a Raspberry Pi to test? Or is that a little ambitious?

No, you need an x64 machine with 8GB of RAM (you may get away with less if you're just learning and don't mind losing whatever data is stored, but do aim for the 8GB).

I'm looking to virtualise either through a VM or a Jail - I'm not actually sure what the real difference is here at the moment thought? Is it that with a VM you state up front how much CPU/RAM the VM will have, whereas with a Jail, this is flexible? Or have I missed something?

They operate on very different levels.

In a traditional VM, you emulate an entire computer. As such, the VM runs an operating system much like would happen on bare metal.

A jail (see also Illumos Zones, which are a close cousin) is an isolated environment running within the same FreeBSD operating system. You get some virtualized subsystems where it makes sense, but the environment is still a (locked-down) FreeBSD environment and you directly run applications like you would outside the jail. Since the abstraction is much more lightweight, performance is much closer (in many cases it's even equal) to non-jail performance.

 

[joeschmuck]

Out of interest, is it possible to set this up on a Raspberry Pi to test? Or is that a little ambitious?

LOL, well I got a laugh out of that one. Yea, an RPI is far from handling this type of task. I'm giving my RPI to a friend because he wants to play with it now. I think an RPI is a great tool and toy to play with but it is very limited in what it can do when you start to compare it to a full size computer.

 

[MichaelCropper]

This is amazing information, that has really helped to put things into context. You're right, I think I have been over thinking things. The dots are starting to join up now Let me go and do a bunch more research on the different parts now I know more about what to look at specifically. I think I've got enough information, at least for the next few weeks of research.

 

[Ericloewe]

Don't forget to check out the Resources section, plus Cyberjock's guide (link in my sig).

 

[JohnDigital]

What you are wanting to do, is how I run with quite alot more going on that. Everything runs on FreeNAS inside a few FreeBSD jails. I could not be happier. Any questions, shoot them over.

I'm also going to be running this on a home network initially which is fine for local access, but adds extra complexity with domain > IP mapping with multiple websites. I.e if I port forward 80 for HTTP and 443 for HTTPS towards the PC running FreeNAS within the main router, how would that work in practice at the FreeNAS level to determine which Jail to direct the traffic to, OpenVPN, OwnCloud, www.website-1.com, www.website-2.com etc.?

Lots of different web facing softwares have alternate ports they listen on, which is no problem to deal with, because they arent public. But maybe you would like them to be? But then there are ones that need to all listen on port 80, or 443 and ARE public. Or, they need to appear to listen on port 80 or 443. Nobody wants to have to go to domain.com:8456 to get to your website, but it can reside there but the address bar says only domain.com. This is accomplished *fairly easily with a bit of reverse proxying in the Apache vhost file. The jail you install Apache in will have its own IP. This is the IP you forward from your router to ports 443 and 80. Not your main FreeNAS IP. For instance my different domains point to the same WAN IP, Apache decides what should go where if necessary (ex. plex.domain3.com, owncloud.domain1.com, nextcloud.domain3.com, domain1.com, domain2.com, domain3.com and all have a DNS A record of 123.123.123.123). Apache diciphers incoming requests and it and passes things along to the right jails/IPs/ports/directories, with the end user being none the wiser.

OpenVPN I would put in its own jail. It doesnt have any issues running from a standalone jail.

You also dont need a seperate jail for every piece of software you intend to run, it makes permissions and storage mounting a real pain. This too many jails tends to happen when the PBI's are used. But you also dont want to cram too much into all one jail either. Get in the habit right from jump street to install your stuff manually into manually created similar jails. It makes it much easier to manage and update as well. I manually install similar softwares in the same jail to keep from having 20 jails, but this is just a matter of preference I suppose.

At any route, good luck, great choice that FreeNAS a.k.a. My Personal Datacenter. Any questions im semi around.

 

[MichaelCropper]

Ok, after doing a bunch more research, I'm starting to think I'm coming at this from the wrong angle.

Previously my initial thought was to use FreeNAS as THE solution to all challenges, but now I'm starting to think in a different way.

Due to the nature of the setup I'm thinking about, I'm not 100% convinced that FreeNAS is even part of the solution. I may be wrong though, so happy to hear thoughts from the FreeNAS experts out there.

Here are my latest thoughts;

Configure primary router to use a DMZ and point all incoming traffic to;
BeastBoxV1.0 - Leave the Raid to a hardware controller
CentOS for primary OS
CentOS to run an Apache Reverse Proxy to point incoming requests to relevant VM
VirtualBox to control each individual VM
Each piece of software I need to run inside its own VM for security reasons

I'm not sure where FreeNAS sits in this setup. I can see the benefits of FreeNAS as a pure storage solution attached on the local network, for example on a home or small business network. And while FreeNAS is capable of doing different things within Jails such as the things I'm trying to achieve, I'm still contemplating weather FreeNAS on the metal is the best setup? I guess I could replace CentOS in the above example with FreeNAS, which would technically achieve the same thing without the reliance on a hardware Raid controller. Am I missing something or is my new line of thinking along the right lines?

I'm getting closer to purchasing the kit required, just wanting to do as much pre-purchase research as possible on the software side. Wanting to run everything on open source technologies where possible for obvious reasons.

Thoughts?

 

[Stux]

A modern data center is basically a collection of SANs and VM compute nodes.

FreeNAS can function as a SAN, ESXi can provide the bare metal hypervisor, and you can mash them all into one system to get quite literally a “mini data Centre”.

Of course, If you just want to host a lamp stack, then that is overkill.

If you want to learn how it all works in practice, in the data centre, then perhaps the FreeNAS/ESXi approach is valid.

 

[MichaelCropper]

Good to know. Sounds like I'm on the right tracks with everything then. Now just need to get the equipment and have a play. Thanks for all the help.