FreeNAS with vCenter and 10GB Ports

[marlonc]

Hello,

I am redesigning my storage network and want to do it best way possible with what I have, so any recommended ideas are welcome.

The FreeNAS will be used to for storage of all VM's, files, etc.

Here is my current hardware:

FreeNAS:

Dell R710 with 6 WD Red 2TB NAS 5400 RPM Drives
8 port Gigabit Ethernet ports
2 x 10GB SPF+ Ports (Intel x520-DA2)
24 GB Memory
IBM M1015 HBA Card

Core Switch:

Enterprise Class 48 x Gigabit Ethernet Ports with 4 x 10GB SFP+ Ports Switch
Currently this switch is doing core routing and VLAN's for the entire network

ESXi Hosts for Resource Sharing:

2 x Dell 610 Servers with 72 GB Memory Each
Each server will use 2 USB drives in mirrored configuration to boot the servers
No hard drives
4 x Gigabit Ethernet ports in each server
2 x 10GB SPF+ Ports (Intel x520-DA2) in each server

Question:

1. I have read a few posts on here and it is recommended to have a separate switch layered in between the FreeNAS and Core Switch to do iSCSI and VMotion traffic. however, with my current hardware inventory, what would be the recommended setup?

2. What would be the recommenced config on the FreeNAS, RAID 6?

3. Should I boot the FreeNAS from mirrored USB drives?

Thanks,

Marlon

 

[bigphil]

1) no need, just use vlans on your current switch infrastructure to separate the traffic.
2) for vm storage, mirrored vdevs (raid 10 like). 6 wd reds isn't going to give you a lot of performance though.
3) you can if you want, just make sure to have config backups regardless.

You'll need more RAM for FreeNAS and a good SLOG device. There's plenty of information on the forum about building a decent setup for vm storage.

 

[Zredwire]

To answer your second question you need to tell us more information about your VMWare environment. Like how many VM's are you running, what type of VM, do you plan to run the VM's completely from the Freenas storage? The reason I ask is because your ESXI servers are much more robust than what you are proposing for Freenas. Seems like it could be a pretty bad mismatch depending on what your trying to do (at least when looking at the number and type of drives you have for Freenas).

 

[marlonc]

1) no need, just use vlans on your current switch infrastructure to separate the traffic.
2) for vm storage, mirrored vdevs (raid 10 like). 6 wd reds isn't going to give you a lot of performance though.
3) you can if you want, just make sure to have config backups regardless.

You'll need more RAM for FreeNAS and a good SLOG device. There's plenty of information on the forum about building a decent setup for vm storage.

Thank you for replying:

2. I could go with a the WD Red Pro@7200 RPM. I will investigate the SLOG device as that is something new to me and see if it fits in my budget. How much RAM would you recommend. I heard it should be 1GB RAM for each TB plus additional for cache?
3. I have a VEEAM backup server on a separate Windows 2012R2 server that is a physical server that will backup the configs and some critical files and some VM's. I also have snapshots configured as well.

 

[marlonc]

To answer your second question you need to tell us more information about your VMWare environment. Like how many VM's are you running, what type of VM, do you plan to run the VM's completely from the Freenas storage? The reason I ask is because your ESXI servers are much more robust than what you are proposing for Freenas. Seems like it could be a pretty bad mismatch depending on what your trying to do (at least when looking at the number and type of drives you have for Freenas).

Thank you for your reply.

My intention was to run all my VM's which are about 10-12 VM's that include Windows 2012/2016 Datacenter, Exchange 2016, some other application specific VM's like Palo Alto NGFW, etc. all from FREENAS without using local storage on the ESXi hosts. I know that local storage on each hosts would be more robust, but I dont have a commercial grade iSCSI SAN and VSAN is too picky, so hence the FREENAS design option. Its a very small network under 20 users, so not a lot of traffic and data being pushed. I also know that I/O performance could take a hit, however, I have core switch with 10GB SPF+ ports with deep buffering that will make a difference with DAC cables attached to the ESXi hosts and FREENAS.

 

[Chris Moore]

You will want to look at this build report to get some ideas for what you can do:
Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

 

[Chris Moore]

Thank you for replying:

2. I could go with a the WD Red Pro@7200 RPM. I will investigate the SLOG device as that is something new to me and see if it fits in my budget. How much RAM would you recommend. I heard it should be 1GB RAM for each TB plus additional for cache?
3. I have a VEEAM backup server on a separate Windows 2012R2 server that is a physical server that will backup the configs and some critical files and some VM's. I also have snapshots configured as well.

There is a FreeNAS config file that needs to be backed up so that you can recover from some of the failure modes of FreeNAS. There is a script for that any many of the other monitoring actions in FreeNAS:
Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

 

[Chris Moore]

FreeNAS:

Dell R710 with 6 WD Red 2TB NAS 5400 RPM Drives
8 port Gigabit Ethernet ports
2 x 10GB SPF+ Ports (Intel x520-DA2)
24 GB Memory
IBM M1015 HBA Card

This is very underpowered particularly with regard to the amount of RAM. More RAM is better, and if you want to be able to saturate a single 10GB connection, you probably need a minimum of 12 drives in mirrored vdevs, depending on the speed of the individual drive. Drives are usually the bottleneck. More is better, just like with RAM. Due to the small chassis of this system, it is less than ideal for the number of drives you need.
Are you trying to do this on a shoestring budget or do you have money that you can spend to do it right?

 

[marlonc]

You will want to look at this build report to get some ideas for what you can do:
Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

Hi Chris,

Thank you for the reply. The SLOG device can get pretty pricey. yes, Doing it with a shoestring budget with the hardware mentioned above. I cannot invest any more $$ for a bigger server. I might have enough for a SLOG device. Could i get away with two smaller SSD drives in a mirrored configuration for read and write, as I heard about this option from another user?

 

[Chris Moore]

Hi Chris,

Thank you for the reply. The SLOG device can get pretty pricey. yes, Doing it with a shoestring budget with the hardware mentioned above. I cannot invest any more $$ for a bigger server. I might have enough for a SLOG device. Could i get away with two smaller SSD drives in a mirrored configuration for read and write, as I heard about this option from another user?

You could get away with that in a pinch, but it will limit your performance significantly and 'regular' SSDs may not hold up for very long because of the amount of IO activity.

 

[marlonc]

You could get away with that in a pinch, but it will limit your performance significantly and 'regular' SSDs may not hold up for very long because of the amount of IO activity.

Chris,

Thanks, I will stick with the 6 x 2 TB WD Red's, the two USB as a mirrored config for the OS. For a system in production, what RAID is recommended for the 6 WD drives, RAID 10?

Thank you,

 

[Zredwire]

A SLOG will be of limited benifit for you if your using ISCSI on ESXI and don't force sync writes.
If you do choose to get a SLOG, you can get a 100GB Intel S3700 for $120.

My setup is very similar to what you are doing. I have 6 WD Red 3TB drives (its actually 4 WD RED and 2 Seagate Ironwolf but they are very similar) in mirrored VDEVs. I also use an Intel S3700 for a SLOG since I am running my ESXI server from NFS instead of ISCSI.
I run 8 VM's using 10Gbit between the ESXI host and my Freenas server. I have not had any bottlenecks with the disk, but my VM's are mainly service oriented and not database or Exchange. Mine are things like firewall, home automation, PLEX, etc.

 

[Chris Moore]

Chris,

Thanks, I will stick with the 6 x 2 TB WD Red's, the two USB as a mirrored config for the OS. For a system in production, what RAID is recommended for the 6 WD drives, RAID 10?

Thank you,

What @Zredwire said... I looked at the spec sheet for the WD Red drives and the 2TB and 3TB drive both have the same 147MB/s internal (mechanical) max transfer rate. So, if he is getting acceptable performance, you probably will too. I do suggest using a SLOG as the system (in this type of use) will be slow without it.

PS. In ZFS it is called mirrored vdevs, not RAID-10. RAID-10 is similar, but based on hardware RAID, where ZFS uses the system CPU to do all the calculations on where to put the data and must have direct access to the drives.
Speaking of which, did you flash the IT mode firmware on that SAS controller?

 

[marlonc]

What @Zredwire said... I looked at the spec sheet for the WD Red drives and the 2TB and 3TB drive both have the same 147MB/s internal (mechanical) max transfer rate. So, if he is getting acceptable performance, you probably will too. I do suggest using a SLOG as the system (in this type of use) will be slow without it.

PS. In ZFS it is called mirrored vdevs, not RAID-10. RAID-10 is similar, but based on hardware RAID, where ZFS uses the system CPU to do all the calculations on where to put the data and must have direct access to the drives.
Speaking of which, did you flash the IT mode firmware on that SAS controller?

Chris,

Thank you for pointing out the RAID 10 and ZFS mirrored features. Yes, I have downloaded the files to flash it to IT mode on the IBM M1015 HBA.

Time to hunt for a SLOG device as per Zredwire similar config.

 

[marlonc]

A SLOG will be of limited benifit for you if your using ISCSI on ESXI and don't force sync writes.
If you do choose to get a SLOG, you can get a 100GB Intel S3700 for $120.

My setup is very similar to what you are doing. I have 6 WD Red 3TB drives (its actually 4 WD RED and 2 Seagate Ironwolf but they are very similar) in mirrored VDEVs. I also use an Intel S3700 for a SLOG since I am running my ESXI server from NFS instead of ISCSI.
I run 8 VM's using 10Gbit between the ESXI host and my Freenas server. I have not had any bottlenecks with the disk, but my VM's are mainly service oriented and not database or Exchange. Mine are things like firewall, home automation, PLEX, etc.

Hello Zredwire,

Thank you for your input. Good to see you have a similar network design. Time to hit Ebay for a SLOG device. My Exchange resource consumption is not huge. Do you think 100GB SLOG device would be sufficient?

Thank you,

 

[Zredwire]

Time to hunt for a SLOG device as per Zredwire similar config.

You can sometimes find rebranded ones for less. I got a Dell. The one I got is below. I offered 120 and the seller accepted. You might could get it for even less.
https://www.ebay.com/itm/100GB-Dell...670319&hash=item56a79a542d:g:SQ8AAOSwK~RZ58ND

 

[Zredwire]

Do you think 100GB SLOG device would be sufficient?
Thank you,

Yes, actually your SLOG will almost certainly not ever use more than 7GB. Its just that you can't really find a drive suitable for SLOG that is less than 100GB. But being much larger than you need, the device can level its writes out over the entire disk and last longer.

 

[marlonc]

Yes, actually your SLOG will almost certainly not ever use more than 7GB. Its just that you can't really find a drive suitable for SLOG that is less than 100GB. But being much larger than you need, the device can level its writes out over the entire disk and last longer.

Ok, great and thank you for your insight. I have read some articles on NFS vs iSCSI and I think I will use iSCSI instead.

 

[marlonc]

Yes, actually your SLOG will almost certainly not ever use more than 7GB. Its just that you can't really find a drive suitable for SLOG that is less than 100GB. But being much larger than you need, the device can level its writes out over the entire disk and last longer.

Quick question:

In you hardware setup for your 6 WD Reds, how are you integrating your Intel SSD SLOG device since it is not a PCI express card like the P3700?

 

[Zredwire]

Quick question:

In you hardware setup for your 6 WD Reds, how are you integrating your Intel SSD SLOG device since it is not a PCI express card like the P3700?

Just hook it to your controller like your other hard drives. Then in FreeNAS you can designate it to be your SLOG.