My Dream System (I think)

[joeschmuck]

@Spearfoot
I have tried a ZIL before, just for testing and it didn't provide any benefits for me, but then again that was a long time ago. I have dedicated 16GB of RAM (locked) to this VM and that seems to be doing fine. Here is a test from a Windows 7VM on ESXi, same server hosting my FreeNAS server. Notice that the values are nice and high to the FreeNAS system. When I run the test on the ESXi datastore where my VM is actually installed, the values are only in the low 300 range, not great but not terrible either for an older SSD. I can't get ATTO to run on a network drive but that is fine. So the FreeNAS VM and all it's hardware is running nice and fast, it appears my limitation is the NIC and my network layout, not ESXi. But I'm very comfortable with my setup so far. Now I need to learn more about hosting Windows VMs and user accounts, etc... I know it's easier to just run a few dedicated Windows VMs but then I have nothing new to learn or get frustrated about. Even if it's not Windows, I'd be fine trying this with Ubuntu or Redhat. Something new is good too.

 

[Dice]

Great news. The early morning mist of doubts have cleared out, there <IS> meat left on the ESXi-Bone X)))

 

[Mirfster]

Just a scenario that I am pondering and wondering if anyone has tried or has insight on it:

1. Say you have ESXi/vSphere installed on two drives that are Hardware Mirrored (say connected to a LSI 9260 w/bbu)
   • Say they are 120 GB SSDs
2. Now you installed FreeNas on two Virtual Drives (So the OS is Mirrored)
   • Created in ESXi/vSphere
   • Say they are 32 GBs each
3. You are going to house ESXi/vSphere VMs on the FreeNas Pool; so you have
   • Properly added a HBA as a Pass-Through
   • Created the Virtual Switch and Virtual Kernel
   • Dedicated ample RAM and CPU (Not just allocated, but dedicated)
   • Created your vDev(s), Pool and NFS (or ISCI)
4. So everything is running

Here is my question, if one wanted to add a SLOG and wanted the benefits of BBU; would it be "sane" to simply create another Virtual Drive and add it to the FreeNas VM?

• Was thinking that it would then benefit from the BBU on the Hardware Raid
• Could easily be sized appropriately (instead of having to under-provision a larger dedicated drive)

Not sure of the pitfalls or benefits but was just wondering if it was feasible?

 

[joeschmuck]

I'm not sure about the SLOG being super fast just because of the added overhead from ESXi, even if you had a true dedicated SLOG on a pass-thru interface. Someone with personal experience will have to answer that one, or you could test it out but ensure you have some superior benchmark testing to prove it.

 

[jgreco]

Just a scenario that I am pondering and wondering if anyone has tried or has insight on it:

1. Say you have ESXi/vSphere installed on two drives that are Hardware Mirrored (say connected to a LSI 9260 w/bbu)
   • Say they are 120 GB SSDs
2. Now you installed FreeNas on two Virtual Drives (So the OS is Mirrored)
   • Created in ESXi/vSphere
   • Say they are 32 GBs each
3. You are going to house ESXi/vSphere VMs on the FreeNas Pool; so you have
   • Properly added a HBA as a Pass-Through
   • Created the Virtual Switch and Virtual Kernel
   • Dedicated ample RAM and CPU (Not just allocated, but dedicated)
   • Created your vDev(s), Pool and NFS (or ISCI)
4. So everything is running

Here is my question, if one wanted to add a SLOG and wanted the benefits of BBU; would it be "sane" to simply create another Virtual Drive and add it to the FreeNas VM?

• Was thinking that it would then benefit from the BBU on the Hardware Raid
• Could easily be sized appropriately (instead of having to under-provision a larger dedicated drive)

Not sure of the pitfalls or benefits but was just wondering if it was feasible?

No. You'd beat the crap out of the ESXi boot device SSDs unnecessarily, leading it to early failure. However, you can get an extremely fast, nearly unlimited SLOG device by doing almost the same thing but booting ESXi off of a RAID1 pair of HDD's.

See https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

Search on keyword "unorthodox"

 

[Mirfster]

No. You'd beat the crap out of the ESXi boot device SSDs unnecessarily, leading it to early failure. However, you can get an extremely fast, nearly unlimited SLOG device by doing almost the same thing but booting ESXi off of a RAID1 pair of HDD's.

Thanks for the link. I knew I had seen that mentioned somewhere as well as Intel 320s. Thinking I will do some testing since I have a few 32 GB Intel 320s and LSI 9260-8Is (only 512 MB cache though) to tinker with. While they are not the best devices, it may be worth trying out.

Excellent point on beating the crap out of the boot devices.

 

[jgreco]

By the way, when done on hard drive, it seems just fine. It is a nice win for low-to-moderate SLOG levels of activity (especially bursty) where you might normally avoid trying to do a SLOG.

 

[Mirfster]

I just upgraded to FreeNAS 9.10 so i could use the VMXNET3 driver

Are you sure you are actually using the vmxnet3 driver? From what I understand 9.10 still loads the vmxnet.ko (https://bugs.freenas.org/issues/15087). You need to do some additional steps to copy the vmxnet3.ko to FreeNas (From ESXi Tools) as well as add a System - Tunable - Loader (to load the driver) and Post-Init (to unload the vmxnet.ko).

As far as FreeNas are you running mirrors or raidz for the pool(s)?

 

[joeschmuck]

Are you sure you are actually using the vmxnet3 driver?

Good question, I don't know. I know it's VMXNET3 on ESXi side for sure but I did not do anything like copy file to FreeNAS to make that happen nor change the tunables. Maybe I need to look into this more, but I've already rolled back to the E1000 driver for a while, since I just upgraded to 9.10 and I want to ensure there are no other unforeseen side effects before I start playing around again with the network drivers. I'm very cautious and in no hurry and that keeps my VM running.

As far as FreeNas are you running mirrors or raidz for the pool(s)?

Nope, just six drives in a simple RAIDZ2.

 

[jgreco]

If the only network interface for your FreeNAS VM is configured as VMXNET3, it is VMXNET3. If you also have an interface added as E1000, then it could be either.

 

[Mirfster]

Nope, just six drives in a simple RAIDZ2.

Sorry to pester you, but is this pool just for data or is it housing ESXi VMs as well? Just curious if it is housing VMs and the performance I saw the posts earlier on read/write and was not sure if that was regarding just standard data or actual VMs.

 

[joeschmuck]

If the only network interface for your FreeNAS VM is configured as VMXNET3, it is VMXNET3.

In the FreeNAS network configuration menu the interface is called "vmx0". In ESXi I created the VMXNET3 interface. So if FreeNAS doesn't come with a VMXNET3 driver built in, it's possible it's using just the VMXNET driver.

 

[joeschmuck]

Sorry to pester you, but is this pool just for data or is it housing ESXi VMs as well?

My pool is just for storage, I currently have not defined any dataset for VM use. I can see that it could be a good thing but my current thinking is to just use a datastore on a SSD for extra VMs, but using FreeNAS storage isn't a bad idea since the internal traffic speed it pretty fast. Hum... I'll have to give it a try but only after I ensure FreeNAS has the VMXNET3 driver loaded.

 

[Mirfster]

So if FreeNAS doesn't come with a VMXNET3 driver built in, it's possible it's using just the VMXNET driver.

More than likely so. If you ran "kldstat" or "kldstat | grep vmxnet", I would presume the you would not see "vmxnet3" but "vmxnet" instead. This would mean you are using the older drivers that are currently in 9.10.

*** This is just from memory, so my syntax/commands may be wrong...

 

[Dice]

@joeschmuck @Mirfster
I managed (after a lot of newbie banging due to guides that misleadingly implied copy/paste - done deal) to extract the vmxnet3.ko to get into the /boot/modules/ folder according to the b3n-guide, added the tunable as described, but could not manage to unload the current vmxnet.ko driver. I've not access to my notes from the maneuver atm, but I think I ran something like a 'kldstat' and tried to kldunload vmxnet.ko followed by kldload vmxnet3.ko but did never recieve any confirmation in kldstat that the vmxnet3.ko was loaded.
As Mirfster suspects, I did not mange to get a kldstat display the vmxnet3.ko.
From what I gathered in the bug reports on this matter - you'll get vmxnet and not vmxnet3.

However, I did mange to get traffic through the vmxnet which is on a different subnet than the E1000 NIC. In my eyes, exxxxcellent speed improvements occurred.

 

[joeschmuck]

Yes, speed improvements were very good using the VMXNET driver.

 

[jgreco]

More than likely so. If you ran "kldstat" or "kldstat | grep vmxnet", I would presume the you would not see "vmxnet3" but "vmxnet" instead. This would mean you are using the older drivers that are currently in 9.10.

*** This is just from memory, so my syntax/commands may be wrong...

The original VMXNET uses the vxn driver which is either supplied by VMware or OpenVMTools. VMXNET3 uses a driver written for FreeBSD, the vmx driver, imported into FreeBSD 10. It is not clear to me that this is compiled into FreeNAS because I haven't looked. The VMware supplied VMXNET drivers are not highly recommended because they tend to do poorly during upgrades.

 

[Mirfster]

Yeah, still unsure if I really need them since from what I gather the main issue (according to the bug) was with jails. Since I am not looking at running jails it should not really affect me. Will try to do some testing to see if there is any speed improvements between the two when I get around to it; just for reference though.

 

[Dice]

the main issue (according to the bug) was with jails.

I'm positive I found more than one bug post on the vmxnet3 topic.
IIRC - one related to 2-3 year old arguments on whether this would be part of FreeNAS (aka- those who cannot manually swap drives should not even attempt to run ESXi).
Another report was more recent. Unfortunately I cannot recall it's 'argument'.

If anyone has a link/or instructions on how to unload the vmxnet.ko and verify that the 'tunables' really is enabled added as discussed in this thread / instructions on the point 9 in this tutorial:
https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

That's where I'm at right now.

 

[Mirfster]

If anyone has a link/or instructions on how to unload the vmxnet.ko and verify that the 'tunables' really is enabled added as discussed in this thread / instructions on the point 9 in this tutorial:

This is what I did; slightly different from the instructions you linked:

In VMware vSphere Client (I am running 6.0 Update 2)

• Right-Click on FreeNas VM
• Select "Guest" - "Install/Upgrade VMware Tools"
• When prompted by the "Install/Upgrade Tools" dialog
  • Select "Interactive Tools Upgrade"
  • Click [OK]

In FreeNas VM (via Putty); Commands

• mkdir /mnt/cdrom
• mount -t cd9660 /dev/iso9660/VMware%20Tools /mnt/cdrom/
  • * This is where my command differs...
• cp /mnt/cdrom/vmware-freebsd-tools.tar.gz /root/
• tar -zxmf vmware-freebsd-tools.tar.gz
• cd vmware-tools-distrib/lib/modules/binary/FreeBSD9.0-amd64
• cd vmware-tools-distrib/lib/modules/binary/FreeBSD10.0-amd64/
  
• cp vmxnet3.ko /boot/modules
• umount /dev/iso9660/VMware%20Tools
  • Unmounts the CD
• rmdir /mnt/cdrom
  • Deletes folder created for the CD Mount earlier

In FreeNas Web GUI

• In "Tasks" - "Init/Shutdown Scripts"
  • Click "Add Init/Shutdown Script"
    • Type: Command
    • Command: kldunload vmxnet
    • When: Post Init
    • Click [OK] to Save/Close
• In "System" - "Tunables"
  • Click "Add Tunable"
    • Variable: vmxnet3_load
      
    • Value: YES
      
    • Type: Loader
    • Comment: Load VMWare VMXNet3 Drivers
    • Enabled: [Checked]
    • Click [OK] to Save/Close

In VMware vSphere Client (I am running 6.0 Update 2)

• Right-Click on FreeNas VM
• Select "Guest" - "End VMware Tools Install"

Finally, Reboot FreeNas.

To verify it is loaded, run the command (via Putty or Shell): kldstat | grep vmxnet3.ko
You should get a result showing vmxnet3.ko is loaded; similar to:

Code:

# kldstat | grep vmxnet3.ko
2    1 0xffffffff81cb0000 8f48     vmxnet3.ko

Hope this helps and apologies to Joeschmuck if this is off topic.

*** Edited to correct the path to "FreeBSD10.0-amd64"