New FreeNAS build for VMWare datastore

[ubellavance]

Hi,

I have used FreeNAS for non-critical (file sharing, lab ESXi) use for a while now and I'm about to build our first production FreeNAS box and I'd like to have your advice. The plan is to use this server:

• HP DL380 G7
• 48 GB RAM
• 2x Xeon X5680
• Smart Array P410i RAID controller
• 2 x 2TB Crucial MX300 SSD drives (with 1 hot spare)
• 8 x 450 GB 10K drives (with 1 hot spare)

Our plan is to use VMWare ESXi on this server and run FreeNAS in a VM because we may run other VMs on this server since it has capacity. We would make a mirror (RAID 1) device for the SSD drives and one RAID 5 device for the spinning disks. The SSD drives will run our VMs and the spinning drives will be for NAS functions (replica of non-critical files (drivers, installers, etc.)). We'll probably give 16 GB of RAM to the FreeNAS VM.

Our VMs will consist of a firewall, a reverse proxy server, a web server, and an 2 Oracle DB servers. These VMs will run on a separate ESXi server that will have 128 GB of RAM. The disk performance requirement is about 5,000 iops for reads, 1,000 for writes.

We don't know yet if the datastore will be iSCSI or NFS. Comments welcome. Is there an official guide to FreeNAS best practices for VMWare? I've seen many forum and blog posts, but nothing official.

Thanks in advance,

 

[Chris Moore]

Smart Array P410i RAID controller

Can it be flashed to IT mode so it just works as a SAS HBA instead of being a RAID controller? If not, it is not a good choice. If it is built into the system board, can it be disabled and a PCIe card added to run the drives?

Our plan is to use VMWare ESXi on this server and run FreeNAS in a VM because we may run other VMs on this server since it has capacity. We would make a mirror (RAID 1) device for the SSD drives and one RAID 5 device for the spinning disks. The SSD drives will run our VMs and the spinning drives will be for NAS functions (replica of non-critical files (drivers, installers, etc.)). We'll probably give 16 GB of RAM to the FreeNAS VM.

This sounds like you are planning to create a hardware RAID array and share that to a FreeNAS VM through ESXi and that is not how it is supposed to be done.
I don't know if I would call it official, but you should definitely look at this build log by @Stux as it goes for a deep dive into how ESXi should be setup with FreeNAS.
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

 

[ubellavance]

Can it be flashed to IT mode so it just works as a SAS HBA instead of being a RAID controller? If not, it is not a good choice. If it is built into the system board, can it be disabled and a PCIe card added to run the drives?

I don't know if it can be used a SAS HBA, but in my case, I don't think that it matters. I've read quite a lot about hardware RAID and FreeNAS and from what I've seen the cons of using a hardware RAID controller are:

• If the server breaks, you need to find a server with the same controller so that it can read the data, while otherwise you can read your data on any FreeNAS server. For that we will have a spare RAID controller on the shelf.
• What else?

This sounds like you are planning to create a hardware RAID array and share that to a FreeNAS VM through ESXi and that is not how it is supposed to be done.
I don't know if I would call it official, but you should definitely look at this build log by @Stux as it goes for a deep dive into how ESXi should be setup with FreeNAS.
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

That's an awesome guide, but as I said "not supposed to be done this way" is not sufficient for me... https://mangolassi.it/topic/12047/zfs-is-perfectly-safe-on-hardware-raid

My goal is not to be able to boot FreeNAS on bare metal if needed. I plan ton install FreeNAS OS on the datastore made of the 2 SSD drives, and this FreeNAS VM will have 3 vDisks:

• The FreeNAS OS disk
• A disk on the SSD datastore
• A disk on the spinning disks datastore

My setup will not use l2arc or SLOG devices, because I assume that SSDs will have more than what I need in terms of performance and the ARC will be sufficient. For the spinning disks, they're just replicated data so the performance requirement is almost 0. ESXi will see 2 disks, FreeNAS will see 3.

Apart from the fact that I rely on a proprietary chip for my data (hence the spare RAID card), I don't see anything wrong than that, but feel free to provide arguments.

 

[Chris Moore]

I don't know if it can be used a SAS HBA, but in my case, I don't think that it matters. I've read quite a lot about hardware RAID and FreeNAS and from what I've seen the cons of using a hardware RAID controller are:

• If the server breaks, you need to find a server with the same controller so that it can read the data, while otherwise you can read your data on any FreeNAS server. For that we will have a spare RAID controller on the shelf.

The whole point of using FreeNAS is to let FreeNAS have direct access to the individual disks so that the ZFS filesystem can be used to create a software RAID. Using ZFS the the main reason for using FreeNAS and you don't get the features of ZFS with hardware RAID...

That's an awesome guide, but as I said "not supposed to be done this way" is not sufficient for me

If telling you, this is how it is supposed to be done, is not good enough, I can't help you and I don't think anyone else can either.

 

[Chris Moore]

Apart from the fact that I rely on a proprietary chip for my data (hence the spare RAID card), I don't see anything wrong than that, but feel free to provide arguments.

I have been doing storage for over 20 years and ZFS with FreeNAS for the last 6. I am not going to argue.

 

[ubellavance]

I have been doing storage for over 20 years and ZFS with FreeNAS for the last 6. I am not going to argue.

I'm sorry, I meant explanations, not arguments. In French, my mother tongue, the word argument doesn't mean the same thing as in English.

 

[bigphil]

That RAID controller does not support HBA mode/pass-through. Listen to what @Chris Moore has told you. I wouldn't use it in a production FreeNAS server. You can source a compatible HBA for very little money, so why risk it? As for the storage protocol to use for VMware...I typically choose iSCSI for FreeNAS because it supports all of the vaai primitives...there is no support for NFS yet. It also depends on what your use case is. If you don't need access to vaai, then NFS may be best because of it's flexibility. Do you already have that hardware and are trying to repurpose it? I ask because there are very few components in your setup that I'd actually use in production for FreeNAS...even if it's virtualized. Another thing I just thought of...HPE G7's aren't even supported with ESXi 6.5, if that's what you plan to use...only ESXi 6...if using the HPE custom VMware iso. You can install the stand-alone version without issue or create a custom iso from the HP image to downgrade the one offending vib (HP smx if my memory is correct).

 

[brando56894]

just get this and flash it to IT mode, you'll be fine.

 

[ubellavance]

I'm repurposing this hardware. Actually, it isn't a real production scenario. For production we use redundant controllers storage arrays, but this one is for a remote site, which is currently on DAS. This site has replicas of backups of non-critical data, and a standby database. Thanks for your advice on iSCSI vs NFS. I don't think we'll need VAAI primitives, but I'll look into it. With iSCSC, sync=always is recommended right? Thanks for the heads-up about the ESXi version. I'll look at the differences between 6 and 6.5 but I don't think even v6 would be an issue.

 

[bigphil]

Yes, sync=always for iSCSI is a best practice for the most common usage scenarios.

 

[sfcredfox]

I'm repurposing this hardware. Actually, it isn't a real production scenario.

I can backup what these three have mentioned, your biggest problem is that RAID controller. Do what brando56894 said and get that HBA for your FreeNAS storage pool. You'll just need to follow one of the forum posted build guides for passing that HBA controller through to your FreeNAS VM.

Depending on the load you are planning to put on the FreeNAS VM for storage needs, you might want to consider more RAM. The importance of RAM for a FreeNAS system, especially one your planning to put an even slightly heavy work load on can't be over stated.

My system was a re-purposed G5, similar setup, but running FreeNAS bare metal. I have since switched to Super Micro to get away from 2.5" drives internally.

 

[ubellavance]

The load should be a maximum of 5 000 iops read/1 000 iops write for the SSD part. The rest is negligible. I'm planning on setting the vRAM to 16 GB and I don't plan using deduplication, but I think I'll use compression, at least on the spinning drives. I think that this hardware should be able to deliver this level of performance. The only bottleneck that I can see is the NICs that are only Gigabit, and the switch as well. The traffic should average below 120 MB/s (gigabit speed) but there will be peaks around 400 or 500 MB/s. This makes me think that iSCSI may be better because of multipathing. I think I have a quad port intel server NIC somewhere and I have plenty of free ports on the switch. I'll read on iSCSI multipathing on FreeNAS.

 

[Ericloewe]

Smart Array P410i RAID controller

IIRC correctly, that's just HP branding for PCH fakeRAID. So, plain ol' SATA.