FreeNAS 9.1.1 + VLAN + Squid/DNS cache cluster.

Status
Not open for further replies.

mijahe

Cadet
Joined
Feb 24, 2014
Messages
3
Hi folks,
As a recently new member to the FreeNAS community I thought I'd post this short HOWTO.

I'm fully aware that a lot of people want to keep FreeNAS purely for storage management.
There have been many discussions on this topic on the FreeNAS forums, (and a lot of angst!).
I don't have any issue with this. I'm posting this short HOWTO to show that it is possible to run a Debian Jail with Squid proxy servers using VLANs and SLB. It works extremely well. I just hope that my first post doesn't incite a revolution.


I delved into FreeNAS for several reasons:
1. I needed to add more storage, and frankly I wanted a much simpler management of my storage
array instead of running the overbloated Solaris 10 + ZFS combo.
2. I also needed to rejig my network infrastructure to use VLANs, because I was running out
of ports on my 48 port 6850 OmniSwitch, (and also effectively remove two aging Linux firewalls,
and put in a Linux based embedded firewall).

However, I wanted to retain my nice "internet services cluster" that spanned across three HP
N30L microservers. As long as one of the servers was up everyone in the house could access the
Internet. Each server contained around 4Tb available storage, and ran:
- Squid3 proxy - with low memory cache, but 512Gb disk cache.
- SquidGuard - to block stuff that I don't want my kids to see.
- Privoxy - remove ads and other creative sites that SquidGuard couldn't capture.
- Bind9 DNS caching - speed up DNS resolution.
- Local Apache services - for Home Automation, bandwidth consumption stats, and other stuff.
- DLNA server - play back videos on my Samsung TV.
- Samba shares - for the Astone media players, Macs, and the few Windows boxes.
- NFS shares - for Linux boxes.


Does it work? Yes!

The final config provides the same services, but now:
- Runs FreeNAS 9.1.1.
- Each of the N30Ls are identical, except one server acts as a 'master' just for configuration purposes.
- I beefed up the RAM to 8G.
- Several Debian Linux jails are running.
1. Squid + SquidGaurd + Privoxy + Bind9 DNS caching.
2. Apache.
3. LDAP + Samba AD.
4. syslog.
- FreeNAS samba - for Astone media players.
- FreeNAS NFS - for Linux boxes.
- FreeNAS AFP - for Macs.
- I gave up on DLNA as it's really flakey. Never worked properly before either.


Note: I tried to upgrade to 9.2 recently, but I've found that Linux jails are less supported by this FreeBSD
version. 9.1.1 provides me the ability to run 64bit Debian Wheezy with VIMAGE + VLANs enabled. 9.2 forced me
to go back to 32bit Debian, and I couldn't run VIMAGE.

Note 2: I was also intending to make use of a USB NIC, (Apple type), for Internet after I had consolidated some
unused ports on the switch, but gave up on this. I went down the path of pulling the source, building my own
release with the CURVNET patch, (http://lists.freebsd.org/pipermail/freebsd-bugs/2014-February/055298.html),
but it was taking too much time to resolve. So stuck with VLANs - much simpler!

Note 3: I wanted to make use of WCCPv2. Both switch and Squid support this, but sadly FreeNAS 9.1.1 doesn't have
GRE inbuilt. Pulling the if_gre.ko module from FreeBAS 9.1 source and attempting to load it doesn't work. Throws
a "link_elf_obj: symbol ip_do_randomid undefined" error. Sadly it's only a one line fix for this issue. When I
have time I'll build from 9.1.1 source and add this in.


HOWTO: Create a Debian Wheezy jail 'template'.
This first part sets up a jail 'template' that you can then create other Debian Wheezy jails from,
and will drastically speed up the initial installs.
1. Create a minimal standard jail from the GUI, (called 'debootstrap');

2. Open a shell an install debootstrap in this Jail:
% jexec debootstrap /bin/sh
% pkg install debootstrap perl5.18-5.18.1_2

3. Make sure FreeBSD LCL modules are loaded, (was some time back, and not sure why I had to do this):
% kldload fdescfs linprocfs linsysfs tmpfs

4. Create Debian base.
% debootstrap --include=apt,vim testing /home/ROOT http://mirror.optus.net/debian/

5. Chroot into this Debian base, and continue to update/add just the basic packages common to all hosts:
% apt-get update
% apt-get install locales ..... etc, etc, etc.

6. Now you have your Debian Wheezy deboostrap install. Copy the /usr/local/share/warden/linux-installs/debian-6-squeeze
template and add remove the line:
################################################################################
# debootstrap squeeze ${jDIR}
################################################################################
and replace with:
################################################################################
rsync -HaxP /mnt/<your ZFS pool name>/jails/debootstrap/home/ROOT/ ${jDIR} > ${jDIR}.log 2>&1
################################################################################

7. This drastically speeds up Debian installs.
% warden create <squid proxy jail name> --linuxjail /mnt/<your ZFS pool name>/jails/debian-6-wheezy


HOWTO: Setup VLANs SLB:
Summary of IPs, (all changed to protect my network).
- 24.24.24.4 - Intranet facing cluster IP.
- 24.24.24.14 - Intranet facing gambit1 IP.
- 24.24.24.24 - Intranet facing gambit2 IP.
- 24.24.24.34 - Intranet facing gambit3 IP.

- 42.42.42.1 - INTERNET facing firewall router.
- 42.42.42.2 - INTERNET facing OmniSwitch interface.
- 42.42.42.14 - INTERNET facing gambit1 IP.
- 42.42.42.24 - INTERNET facing gambit2 IP.
- 42.42.42.34 - INTERNET facing gambit3 IP.


1. On the Omniswitch I created a VLAN for the Internet side on 4 ports, (13,14,15,16):
vlan 3 enable name "Internet"
vlan 3 port default 1/13
vlan 3 port default 1/14
vlan 3 port default 1/15
vlan 3 port default 1/16
ip interface "InternetRouter" address 42.42.42.2 mask 255.255.255.0 vlan 3 primary ifindex 2

2. On the Omniswitch enable VLAN tagging on N30L ports:
vlan 3 802.1q 1/9 "Internet tag"
vlan 3 802.1q 1/20 "Internet tag"
vlan 3 802.1q 1/23 "Internet tag"

3. On FreeNAS create a VLAN and arbitrarily assign it an IP of the Internet facing NIC if teh Squid server.
I called this 'VLAN3'.
# Set to 42.42.42.14

4. Create the load balancer:
ip slb admin enable

ip slb probe "SQUID" http
ip slb probe "SQUID" http period 120
ip slb probe "SQUID" http port 80
ip slb probe "SQUID" http expect "OK"
ip slb probe "SQUID" http url "/cgi-bin/proxy-status.pl"

ip slb cluster "SQUID" vip 24.24.24.4
ip slb server ip 24.24.24.14 cluster "SQUID" probe "SQUID"
ip slb server ip 24.24.24.24 cluster "SQUID" probe "SQUID"
ip slb server ip 24.24.24.34 cluster "SQUID" weight 2 probe "SQUID"

5. Back on your Squid proxy server create the CGI file to be used in the OmniSwitch probes:
# /usr/lib/cgi-bin/proxy-status.pl
################################################################################
#!/usr/bin/perl


sub vlan3_check()
{
system('/sbin/ifconfig vlan3 > /dev/null');
if ($? >> 8)
{
$gsStatus .= "DOWN - vlan3 dead\n";
$gsFail = 1;
}
else
{
$gsStatus .= "UP - vlan3\n";
$gsFail = 0;
}
}

sub lo1_check()
{
system('/sbin/ifconfig lo1 > /dev/null');
if ($? >> 8)
{
$gsStatus .= "DOWN - lo1 dead\n";
$gsFail = 1;
}
else
{
$gsStatus .= "UP - lo1\n";
$gsFail = 0;
}
}

sub ping_check()
{
system('ping -qnc 2 10.57.0.1 > /dev/null');
if ($? >> 8)
{
$gsStatus .= "DOWN - Optus DHCP dead\n";
$gsFail = 1;
}
else
{
$gsStatus .= "UP - Optus DHCP\n";
$gsFail = 0;
}
}

sub squid_check()
{
$RETURN=`/etc/init.d/squid3 status`;
chomp($RETURN);
if ($RETURN =~ /squid3 is running./i)
{
$gsStatus .= "UP - Squid\n";
$gsFail = 0;
}
else
{
$gsStatus .= "DOWN - $RETURN\n";
$gsFail = 1;
}
}

sub privoxy_check()
{
$RETURN=`/etc/init.d/privoxy status`;
chomp($RETURN);
if ($RETURN =~ /privoxy is running./i)
{
$gsStatus .= "UP - Privoxy\n";
$gsFail = 0;
}
else
{
$gsStatus .= "DOWN - $RETURN\n";
$gsFail = 1;
}
}


$gsFail = 0;
$gsStatus = '';

if ($0 =~ /proxy-status.pl/)
{
lo1_check();
vlan3_check();
ping_check();
squid_check();
privoxy_check();
}


if ($gsFail)
{
print("Status: 202 Accepted
Content-type: text/plain

$gsStatus
");
exit(1);
}
else
{
print("Status: 200 OK
Content-type: text/plain

OK
");
exit(0);
}
################################################################################


HOWTO: Create master squid proxy server.
1. Create the first Squid jail. I called it 'gambit'.
% warden create gambit --linuxjail /mnt/<your ZFS pool name>/jails/debian-6-wheezy

2. Update Jail stop/start files to make sure the VLAN3 NIC is created and assigned. Add this to the end:
# jail-post-start
################################################################################

echo ifconfig vlan3 vnet $JAILNAME
ifconfig vlan3 vnet $JAILNAME
################################################################################

# jail-pre-stop
################################################################################

echo ifconfig vlan3 -vnet $JAILNAME
ifconfig vlan3 -vnet $JAILNAME
################################################################################

3. Create a sysvinit startup file to ensure you assign IPs to lo1 and VLAN3:
(Note: lo1 is REQUIRED because the OmniSwitch load balancing will send packets to each of the server with a DST IP of the cluster IP.)
# /etc/init.d/load-balance
################################################################################
#! /bin/sh
### BEGIN INIT INFO
# Provides: load-balance
# Required-Start: $local_fs $remote_fs $network
# Required-Stop: $local_fs $remote_fs $network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Load balance for OmniSwitch.
# Description: Used for creating lo1 interface definition under FreeBSD jails.
### END INIT INFO


# insserv -v
# update-rc.d load-balance defaults

NAME=load-balance
PATH=/bin:/usr/bin:/sbin:/usr/sbin

trap "" 1

# See how we were called.
case "$1" in
start)
printf "Starting $NAME: $NAME"
/sbin/ifup lo1
/sbin/ifup vlan3
/bin/hostname `cat /etc/hostname`
echo "."
;;
stop)
printf "Stopping $NAME: $NAME"
/sbin/ifdown lo1
/sbin/ifdown vlan3
echo "."
;;
restart | force-reload)
$0 stop
sleep 1
$0 start
;;
*)
echo "Usage: $NAME {start|stop|restart}"
exit 1
;;
esac

exit 0
################################################################################

# /etc/network/interfaces
################################################################################
# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d

auto lo1
iface lo1 inet static
pre-up/sbin/ifconfig lo1 plumb
address24.24.24.4
netmask255.255.255.0
broadcast24.24.24.255
post-down/sbin/ifconfig lo1 unplumb

auto vlan3
iface vlan3 inet static
address42.42.42.14
netmask255.255.255.0
broadcast42.42.42.255
gateway42.42.42.1
################################################################################


4. Startup the gambit jail.
% warden start gambit


5. You should see these interfaces:
################################################################################
(JAIL) root@gambit1 [~] 1 # ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair3b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 42:42:42:42:10:4
inet 24.24.24.14 netmask 0xffffff00 broadcast 24.24.24.255
inet6 fe80::4042:42ff:fe42:1004%epair3b prefixlen 64 scopeid 0x2
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 24.24.24.4 netmask 0xffffff00
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=1<RXCSUM>
ether 68:b5:99:72:cb:40
inet6 fe80::6ab5:99ff:fe72:cb40%vlan3 prefixlen 64 scopeid 0x4
inet 42.42.42.14 netmask 0xffffff00 broadcast 42.42.42.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
################################################################################

6. You should be able to ping stuff:
################################################################################
(JAIL) root@gambit1 [~] 538 # ping -c 1 google.com.au
PING syd01s19-in-f31.1e100.net (74.125.237.223): 56 data bytes
64 bytes from 74.125.237.223: icmp_seq=0 ttl=56 time=12.087 ms
--- syd01s19-in-f31.1e100.net ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
################################################################################


6. The rest is stock standard configuration of Squid in an "HTCP cluster", and bind9 caching.
There are plenty of HOWTOs on this. If anyone is desperate I can certainly post my config here,
but I think it'll be verging on "off topic".

Things to watch out for:
* Make sure you don't bind all interfaces. Keep Squid only listening internally.
* Of course I assume that you have a firewall on your Internet side.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Mind if we move this to the guides section?
 
Status
Not open for further replies.
Top