IPFW: How to Make it Go?

[sd1a]

Hello New Friends and Forum Mates!

Having trouble getting the IPFW firewall to run with my own access rules. I've read most of the recent threads I could find on search here, and google (or other googely search engines). I've also I have the script ready to go, but having trouble getting IPFW to run it. I can get IPFW to run via the shell. It runs the FreeNAS rules. I've learned from the readings here where to put the script and rules, in /mnt/volumename/blablabla Forgive my terminology (see below###Me.). I also understand FreeNAS is not an OS to be hacked (and have no desire to). I did read that IPFW is an option, but can't seem to find out how to work with it. I also understand FreeNAS is not meant to be placed on an open network, and have read comments that indicate you might not want a firewall on your FreeNAS. I think I should give you some background on me and my system now.

###Me.
I'm mostly a network guy. At my current job, designed and implemented a dual DMZ with many secure VLANs, remote VPN (a dozen or so years ago). More recently redesigned it, moving from a now very old Cisco ASA to a new ASA and added site to site VPN. Oh, also experienced the shock of AnyConnect and the dreaded DAP policies to control who connects to what (see below ###My system.). I am at a highly regulated NPO.

Somewhere along the way, I took the initiative to move us onto VMware, in order to keep us from being pushed into using an unregulated VM environment across open networks, and to harness all the power of our old servers. So, with some good training after the fact, I also built and manage a vSphere/esxi environment as a 2nd hat. I'm somewhat a hack at that, but have it running well and everything is redundant. Of course, along with that hat comes another hat:

Storage. I'm even more of a hack at storage. I started out using maxed out (9TB from 12TB per host, after formatting, RAID, and esxi'ing it), then moved on to a Synology configured as a SAN (iSCSI). Designed it for a maximum of 10TB, now up to about 20TB :\ Added a 2nd failover Synology, which ended in a pretty much horrific way. So, in order to use the local storage on the three hosts as backup, I now turn to freeNAS.

BSD (of any flavor). I'm kind of Linuxy, but very green to FreeBSD, openBSD and FreeNAS. I actually have the most experience with FreeNAS. I probably got about 30 or 40 hours in with the older 9.x several years ago. That was more of a proof of concept deal, and I failed to get it to work with iSCSI as a VM. Worked great for NFS and Samba shares. Now I have been assigned to get a FreeNAS on each esxi host running and available for NFS, so I have another bundle of hours on that over the summer. I am such a hack at it. I consider myself very inexperiance at storage, and more so at FreeNAS storage. I beg for mercy!

I am dyslexic (isdelxic?) and very bad at typing, spelling and grammar. 10% of us are, knowingly or not. I have dogs and raise (at a hack level) some bonsai trees. I also make noise with bass guitar. Not music, just noise.

###My System.

I mentioned above, dual DMZ. You are storage experts here and I'm sure you know what a DMZ is, but in case some are not familiar with a dual DMZ, I'll explain now. I have one DMZ (AKA a true DMZ), which has any servers that can be touched by the outside (public/Internet). In other words, web servers. The other DMZ (aka a quazi DMZ), can only be touched by internal computers, or in a controlled manner, the web servers in the first DMZ. So it has SQL, file shares, DC's, etc. Both DMZ's run Linux and MS servers, the latter might be a security concern for many, especially the Windows IIS servers on the true DMZ.

The esxi environment is three old Dell 2950's, all upgraded to the last revision with the best CPU's (4 core X 2 sockets) and 64 GB RAM. Works very well for our needs. The three FreeNAS's, one on each esxi host, each get 4 x 3Ghz cores and GB RAM. They seem to run very well. Depending on who to what, I have 2 to 4Gbps between the hosts, Internet, and other resources outside my network. Only two of the three esxi hosts have good storage. I have the disks to upgrade the third, but holding, because I am to get another host (also Dell 2950) soon, like 2 years ago. I would like to make that a bare metal FreeNAS if I don't have to use it as a 4th esxi host.

FreeNAS: Latest version on the two hosts with good storage (11.1-U6, upgraded from U5). The host with anemic storage has the beta version with upgraded GUI, and that's what I'm working with now as far as IPFW goes. That beta would be 11.2-BETA2 updated to yesterday. Right now I'm giving 1 to 2 TB to FreeNAS, but it's easy to add more. Each FreeNAS has three NICs (vmnics), one for each DMZ and one for the management VLAN.

###Goal.

Not speed. This is for backup of Linux text files, Linux log files (for now... need a new logfile server!), and storage of uploads to the web sites (~1TB currently, but soon to include video and I expect it will explode in TB's). This will be the only copy of uploads, so one FreeNAS will take the uploads, then it needs to rsync to the other at night. All NFS for now. The upload speeds are very limited by the public's ISP.

Security. Oh my. I have to be concerned with real security and perceived security. I really need to not have the web GUI available on the DMZ's. I need to have only what's needed available on each DMZ, by IP address, and the management access available to my management VLAN. In other words, I need NFS to specific servers on each NIC, rsync between the FreeNAS's and the web GUI to my management VLAN (plus the usual other stuff, NTP, domain, ssh, etc. BTW: confusingly, service requests like NTP, domain, etc. will be sent to management VLAN and routed to the quasi DMZ :D

###Future.

While I prefer at least the critical servers have a "clone" off the SAN, to an esxi datastore, I could need to add SMB/SAMBA to the FreeNAS for veeam zips to backup VM's. Hopefully, any other future requirements will fall after I retire to a tropical island with sand between my toes! The goal for that is three years.

###TIA!

Thanks for reading what you read. I really respect your time and expertise on storage!

Cherio,
jon

 

[kdragon75]

Storage. I'm even more of a hack at storage. I started out using maxed out (9TB from 12TB per host, after formatting, RAID, and esxi'ing it), then moved on to a Synology configured as a SAN (iSCSI). Designed it for a maximum of 10TB, now up to about 20TB :\ Added a 2nd failover Synology, which ended in a pretty much horrific way. So, in order to use the local storage on the three hosts as backup, I now turn to freeNAS.

At some point you should gut the storage from the hosts and run All VMs from iSCSI LUNs. This will give the option of vMotion and VMware HA for your VMs. I STRONGLY suggest picking up the official VMware VCP cert guide and reading cover to cover at least twice. Anyway, once you have all of your storage on PHYSICAL FreeNAS boxes, you can use ZFS replication for your backup. The nice thing is that with FreeNAS you can coordinate VMware VM snapshots with ZFS snapshots so your backups are application consistent. Just be mindful to not do that every hour or you will tank your performance! I would do plain ZFS snaps every hour with a one week retention and VMware coordinated ones one per day.

Now I have been assigned to get a FreeNAS on each esxi host running and available for NFS

That just sounds like a hot mess. Also this will circumvent 90% of ZFSs ability to keep your data safe. ZFS needs direct access to your hardware and you CAN NOT use a RAID card.

The esxi environment is three old Dell 2950's

I'll dig through the bin for you... I literally just tossed three IBM x3550s with Xeon x5660s and 32GB of RAM. If there is ANY way to get off of that hardware you should ASAP.. That is unless you use it to heat the building :D

each get 4 x 3Ghz cores and GB RAM

When running ESXi, always configure only the absolute minimum to meet real world requirements. More is almost always slower. Look into how the CPU scheduling works. Also I don't see the memory allocation but this needs to be t least 8GB.

I have the disks to upgrade the third, but holding, because

because your building proper storage servers? I can't stress this enough. If your running FreeNAS on ESXi, you need to use PCI passthrough for an HBA not a RAID controller. Even then this is FAR from something I would ever do.

has the beta version

Why would you run beta in production?

FreeNAS will take the uploads, then it needs to rsync to the other at night.

depending on the exact use case I would just use the replication built in to FreeNAS. with that you can use snapshots and keep a few versions.

Security. Oh my. I have to be concerned with real security and perceived security.

I have no clue what your using for a routing/firewall platform but you should take a look at pfSense. It will give you all the flexibility with firewalls, IPS/IDS, VLANs, IP-Sec, and even policy routing that you could ever want. It also runs well as a VM.

While I prefer at least the critical servers have a "clone" off the SAN, to an esxi datastore

Cart before the horse! the VMFS should reside ON the SAN not the other way around!

The goal for that is three years.

Lets hope you server last that long (they won't)

 

[kdragon75]

Do you have ANY budget at all?

 

[sd1a]

Thanks for your response kdragon!

At some point you should gut the storage from the hosts and run All VMs from iSCSI LUNs. This will give the option of vMotion and VMware HA for your VMs.

I do and I have vMotion and HA running. I just want to backup to the local datastores, plus the public uploads and have redundancy to that on a 2nd FreeNAS.

Anyway, once you have all of your storage on PHYSICAL FreeNAS boxes

I understand that, but have to deal with the powers that be... I am actually hoping the I can use the 4th promised host as a FreeNAS bare metal iSCSI storage device. Time will tell :)

That just sounds like a hot mess. Also this will circumvent 90% of ZFSs ability to keep your data safe. ZFS needs direct access to your hardware and you CAN NOT use a RAID card.

I know... :( It is what it is. So it runs off a VM in esxi with disks from the datastore.

I'll dig through the bin for you... I literally just tossed three IBM x3550s with Xeon x5660s and 32GB of RAM. If there is ANY way to get off of that hardware you should ASAP.. That is unless you use it to heat the building :D

I would actually take those over any newer Dell offerings. They last forever. I have spares of every single component. Other than upgrades, all I've ever replaced is power supplies. When the nukes drop, all that will survive is cockroaches, Peavey amps, and Dell 2950's. I pretty much hate dells, but there is a sweat spot. 2900's sucked and whatever came after 2950's also sucked. I also have a huge AC blowing air on the rack (and me :D ). Oh, and the disks are all fairly new.

When running ESXi, always configure only the absolute minimum to meet real world requirements. More is almost always slower. Look into how the CPU scheduling works. Also I don't see the memory allocation but this needs to be t least 8GB.

Sorry, I missed the 8. It is 8 GB.

because your building proper storage servers? I can't stress this enough. If your running FreeNAS on ESXi, you need to use PCI passthrough for an HBA not a RAID controller. Even then this is FAR from something I would ever do.

Yes, I have hopes that I can have the 4th host to be used as a FreeNAS, no RAID, just ZFS. We don't even touch the resources of our esxi hosts. There is no reason to have a 4th host in my environment, but there are powers that be...

Why would you run beta in production?

It's not production. At least not as far as use. The FreeNAS on that host will never be a production storage device during my life at the current institution. It's a production esxi host, but I have no faith in the storage.

depending on the exact use case I would just use the replication built in to FreeNAS. with that you can use snapshots and keep a few versions.

I will have to look into that. The Linux sysadmin likes rsync, but that doesn't mean it's the best way to go.

I have no clue what your using for a routing/firewall platform but you should take a look at pfSense. It will give you all the flexibility with firewalls, IPS/IDS, VLANs, IP-Sec, and even policy routing that you could ever want. It also runs well as a VM.

Cisco ASA. That's my day job. No problems there.

Cart before the horse! the VMFS should reside ON the SAN not the other way around!

OK, the VM's all run on the iSCSI SAN. All their disks are on the SAN, except for junk VM's that can be lost. The SAN has a full backup/HA SAN, which I am reviewing, after a God Awefull failover.

Lets hope you server last that long (they won't)

I hope they outlast me. But that is not the issue, at least for today ;)

Do you have ANY budget at all?

Only your tax money :D

Seriously, for now I really just need to have a software firewall on my FreeNAS installations. How can I get IPFW working?

I love your advice, really, really respect your expertise, if everyone is freshly showered I'll take a group hug. But I really, really need to get the web GUI and other stuff other than NFS off my DMZ's. This is not about what is the right way; it's about what will work and be compliant in the idiot situation I am in.

Oh, I can't be fired unless I kill someone. Yet, by law I can go to prison if I am found negligent in the protection of data. So the stakes are high, while performance expectations are low.

Hey, thanks again for your efforts!

jon

 

[kdragon75]

Yet, by law I can go to prison if I am found negligent in the protection of data.

With that hardware and that risk I would quit and warn them leaving a paper trail on the way out so they can't pin it on me. I kinda picky about the environments I work in.

As for ipfw, it's just FreeBSD. The big issue is that you will want to write a script to set up all the rules etc... Otherwise it will not survive an update if it even suvives a reboot. Write it up as a shell script, save it to your pool and add a startup task to call it.

 

[sd1a]

Thank you again for the response! Sorry I dyslexia's your name, which I edited.

As for ipfw, it's just FreeBSD. The big issue is that you will want to write a script to set up all the rules etc... Otherwise it will not survive an update if it even suvives a reboot. Write it up as a shell script, save it to your pool and add a startup task to call it.

That's what I got from reading other threads here. This was at the end of the day Friday when I finally found the information. I know it's dumb, but seems a common pitfall, I had been trying to get it to run from that information. So I had the script ready to go and I didn't get it to run. I probably just did something stupid and obvious, but gave up at that point. You have verified that is what I need to do. In my first post, I called it a volume, but, it's on my pool.

Back to it after reading your post today. I got it to run (just starting it by hand now. But it is ineffective. The last rule is default allow IP any any, and it seems I can't change that without hacking the kernel, which I'm not willing to do. Now that would be an update/upgrade nightmare. I'm probably tainted by Cisco firewall knowledge, but had hope my final rule, "deny IP any any" would drop all packets before the default. Cisco stops on the first match.

I'll look around more tomorrow, but most likely I'm giving up on ipfw, and I'll drop back to single NIC in the FreeNAS, then use the Cisco firewall to control who can see what from the DMZ's. 3rd option is to explain FreeNAS won't do it and just run a Linux VM with NFS. I like FreeNAS and appliances in general for ease of maintenance, so I'm going to try to make it work.

Don't worry, I'm not going to jail. I'm covered. I should have said "certain sensitive data." What's going onto the FreeNAS will not fall into that category. I don't see how the old VMware hosts are a security risk. The "hardware" in use by the VM's is esxi. I don't expose esxi to anyone but myself a couple of alternate administrators.

I noticed in your signature, "FreeNAS-11.1-U4 (Because U5 sucks)." Is it not best for me to run the latest U6?

Thanks again!

 

[kdragon75]

Is it not best for me to run the latest U6?

I haven't looked into it yet. I generally wait a few weeks after an update to see what the issues are.