AIO FreeNAS and Sophos UTM (32GB ECC, E3-1281, C226 Board, 10GB Fibre...)

[Jono_K]

Hello,

I'm considering moving to an all-in-one set-up for my current FreeNAS installation. I know, I know, AIO, a risky idea with ZFS, but I really think it will be fine....

My usage is fairly low, and consists of two full time users with limited data storage needs. Sophos UTM has been great, and I'd like to continue using it to secure our home network, and use the web server protection feature to secure a NextCloud instance. I've been using FreeNAS for the past three years, and was on Nexenta before then. I'd like to stay with FreeNAS for storage, although I'm not keen on using it as a hypervisor for other OSes.

My Sophos instance has been running reliably on ESXI for over a year, and I feel comfortable adding more pieces at this stage. As such, I'm preparing to migrate my existing FreeNAS and Sophos UTM from two discreet machines to a single all-in-one set-up.

Intel® Server Board S1200V3RPL - http://ark.intel.com/products/71384/Intel-Server-Board-S1200V3RPL (already own)
Proprietaty LSI add-on card provides RAID 1 (already own) with MFBU (will consider buying)
Intel® Xeon® Processor E3-1281 v3 - http://ark.intel.com/products/80907/Intel-Xeon-Processor-E3-1281-v3-8M-Cache-3_70-GHz (already own)
2 x Crucial 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3L 1600 (PC3L 12800) Server Memory Model CT2KIT102472BD160B (already own)
PCI-E Supermicro LSI 2008 SATA/SAS card in IT mode (already own)
2 Port Mellanox ConnectX 10G (plan to buy, appears ESXI and Sophis UTM both support)
2 x Finistar 10G optical transceiver (plan to buy)
Ebay 1GBE Copper transceiver (plan to buy)

Lexar 32GB JumpDrive P20 USB 3.0 Flash Drive, Speed Up to 400MB/s (LJDP20-32GCRBNA) (already own)
2 x Intel® SSD DC S3520 Series (240GB, 2.5in SATA 6Gb/s, 3D1, MLC) - http://ark.intel.com/products/93012/Intel-SSD-DC-S3520-Series-240GB-2_5in-SATA-6Gbs-3D1-MLC (plan to buy)

8 x 4TB HGST (already own)
Currently at 40% capacity, with limited growth expected

ESXi will run from the USB flash drive.
ESXi VM data store will be on a RAID 1 volume (MFBU will provide power loss protect for 512MB RAM cache on RAID card) on two Intel DC s3520's (redundancy on super-capacitor power protected drives)

FreeNAS will be installed with the 8 port 2008 LSI based IT mode card in pass-through
Installed on the RAID 1 set (20GB)
Log will be on the RAID 1 set (20GB)

Sophos will be on the RAID 1 set-up (50GB)
Mellanox in passthrough (connect to local network - fibre transceiver to Netgear 1GBE switch with same transceiver) (this will be in pass-through)
Mellanox in passthrough (connect to modem - ebay copper transceiver) (also in pass-through)

*Next Cloud will be on the RAID 1 set-up (50GB)
Storage will be mounted via NFS from FreeNAS (NFS vs iSCSI, NFS appears to come out ahead for my limited needs.)

Back-up of my data is to a Storage Spaces data-set (4 x 3TB WD REDs) on a Window 10 machine (using SyncBack from 2BrightSparks), which will, in turn, be using Crash Plan to sync to a 8TB WD external drive off-site. These machines are not using ECC RAM, and are using NTFS as the file system. Their purpose is to provide back-up in the case of a catastrophic failure on the FreeNAS machine.

That's the plan. Thank you for taking the time to read through this, and please do offer any feed back through you might consider useful if you were considering a similar endeavour.

*I'm not currently running NextCloud, so this is more of a nice to have.

 

[Spearfoot]

I run a couple of AIO systems (see 'my systems' below for details). You seem to be on the right track.

The only change I suggest is that you boot ESXi from your RAID1 datastore instead of a USB flash drive. Both of my AIO's are set up this way; I have a local datastore comprised of a pair of mirrored SSD's connected to an LSI 9211 HBA in IR mode (or clone thereof, e.g., IBM M1015/Dell H200). I install ESXi and the FreeNAS VM on this local datastore, with another HBA passed through to the FreeNAS VM. All of my HDDs are then run by the FreeNAS VM.

Neither ESXi or FreeNAS need much disk space, so your 240GB SSD local datastore should have plenty of room for the Sophos UTM VM.

Good luck!

 

[RodyMcAmp]

How are you booting ESXi from your raid 1 data store? I am a little confused. I can't think of a way to boot ESXi and FreeNAS from a data store without some trickery. ESXi install uses a full disk and so does FreeNAS,

I run a couple of AIO systems (see 'my systems' below for details). You seem to be on the right track.

The only change I suggest is that you boot ESXi from your RAID1 datastore instead of a USB flash drive. Both of my AIO's are set up this way; I have a local datastore comprised of a pair of mirrored SSD's connected to an LSI 9211 HBA in IR mode (or clone thereof, e.g., IBM M1015/Dell H200). I install ESXi and the FreeNAS VM on this local datastore, with another HBA passed through to the FreeNAS VM. All of my HDDs are then run by the FreeNAS VM.

Neither ESXi or FreeNAS need much disk space, so your 240GB SSD local datastore should have plenty of room for the Sophos UTM VM.

Good luck!

 

[gpsguy]

ESXi doesn't use a full disk. It only needs 5.2GB (1GB for the OS + 4GB for a scratch partition). The rest of the disk is usable for a datastore.

Spearfoot created a mirror using his Dell PERC controller and installed ESXi on it. He installed FreeNAS as a VM on that datastore (first one by default). Then he "passed through" his M1015 controller with 2TB disks for his RAIDz2 volume.

ESXi install uses a full disk and so does FreeNAS,

 

[RodyMcAmp]

learn something new. I am going to have to try this out now.
thanks

ESXi doesn't use a full disk. It only needs 5.2GB (1GB for the OS + 4GB for a scratch partition). The rest of the disk is usable for a datastore.

Spearfoot created a mirror using his Dell PERC controller and installed ESXi on it. He installed FreeNAS as a VM on that datastore (first one by default). Then he "passed through" his M1015 controller with 2TB disks for his RAIDz2 volume.

 

[Jono_K]

Thank you, Spearfoot.

After you suggested this I did a little looking into it and will definitely be taking your suggestion.

I run a couple of AIO systems (see 'my systems' below for details). You seem to be on the right track.

The only change I suggest is that you boot ESXi from your RAID1 datastore instead of a USB flash drive. Both of my AIO's are set up this way; I have a local datastore comprised of a pair of mirrored SSD's connected to an LSI 9211 HBA in IR mode (or clone thereof, e.g., IBM M1015/Dell H200). I install ESXi and the FreeNAS VM on this local datastore, with another HBA passed through to the FreeNAS VM. All of my HDDs are then run by the FreeNAS VM.

Neither ESXi or FreeNAS need much disk space, so your 240GB SSD local datastore should have plenty of room for the Sophos UTM VM.

Good luck!