Virtualization with passed through LSI controllers and MSI-X

[mattlach]

Hey all,

First off, YES I know visualization is not supported, just hoping someone might be familiar with some of this stuff and be willing to provide some information.

I've been running FreeNAS as a guest inside ESXi since ESXi 5.0 and FreeNAS 8.3.0 happily and stably.

This weekend I am doing my "I got tired of VMWare and migrating to Proxmos instead" project.

I recall from when I first installed FreeNAS inside an ESXi guest, in order to get my LSI controllers working, I had to add the following config lines to loader.conf in order to avoid massive interrupt storms bringing the system to a halt:

hw.pci.enable_msix=0
hw.pci.enable_msi=1

I just did a fresh install of FreeNAS 9.10 in KVM on Proxmox, and even without those interrupt configuration strings, there are no interrupt storms and everything appears to be working normally.

Has something changed in the last few years making MSI-X now work? Or is there something different about how KVM handles these things? Maybe MSI-X is disabled by default?

I'd appreciate any input!

--Matt

 

[joeschmuck]

First off, YES I know visualization is not supported

Sure it is, it's just not the smartest thing for many people. FreeNAS has built in vmware support which allows ESXi to shutdown FreeNAS automatically.

FreeNAS 9.10 is FreeBSD 10 based so that is a big difference from FreeBSD 8. I know that doesn't really answer your question.

 

[jgreco]

Virtualization support isn't supported in these forums, would be a more correct assessment, and for all the reasons posted in the virtualization stickies, isn't suggested for the non-expert ESXi and FreeNAS user.

MSI/MSI-X has always seemed to be a little flaky. I wouldn't suggest it worth the time to get all OCD about it; do what works and move on with life.

 

[mattlach]

Well, Either KVM is not as well suited to passthrough with FreeNAS or the MSI/MSIX tunables really are still important, or I've just experienced some other coincidental hardware failure at the same time as doing my migration.

After running with passed through LSI controllers in ESXi for almost 5 years, my first scrub since transitioning to KVM resulted in seemingly random checksum errors across the pool and data loss in 48 files thus far. I'm glad I do offsite backups :)

I added the MSI/MSI-X tunables and restarted the FreeNAS VM, and now the scrub seems to be continuing with no checksum errors (yet).

It's too early to tell. I'm going to have to let the scrub complete, but I'm hoping that this is a matter of the MSI tunables above being a good idea, and not one where KVM can't handle the task.

Who knows, as part of the transition, I did also open up the case and move some cards around. It could also have been a hardware/cabling failure at the same time as my migration to KVM.

Going to let this scrub complete and see what the results are. This will be a long 30 hours.

 

[mattlach]

Well, so much for that theory. The checksum errors reappeared :(

Code:

pool: zfshome
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub in progress since Sun Apr 24 00:00:04 2016
        13.8T scanned out of 33.5T at 213M/s, 26h52m to go
        560K repaired, 41.29% done
config:

    NAME                                            STATE     READ WRITE CKSUM
    zfshome                                         ONLINE       0     0     0
     raidz2-0                                      ONLINE       0     0     0
       gptid/85faf71f-2b00-11e4-bc04-d8d3855ce4bc  ONLINE       0     0     3  (repairing)
       gptid/86d3925a-2b00-11e4-bc04-d8d3855ce4bc  ONLINE       0     0    13  (repairing)
       gptid/87a4d43b-2b00-11e4-bc04-d8d3855ce4bc  ONLINE       0     0     7  (repairing)
       gptid/887d5e7f-2b00-11e4-bc04-d8d3855ce4bc  ONLINE       0     0     7  (repairing)
       gptid/8cdbc2e5-cb62-11e4-8b23-000c29dbe1ad  ONLINE       0     0     4  (repairing)
       gptid/b62d0aa1-638a-11e4-8721-000c29dbe1ad  ONLINE       0     0    17  (repairing)
     raidz2-1                                      ONLINE       0     0     0
       gptid/56fb015b-2bfc-11e4-be49-001517168acc  ONLINE       0     0     9  (repairing)
       gptid/576cde68-2bfc-11e4-be49-001517168acc  ONLINE       0     0    12  (repairing)
       gptid/57dbbac1-2bfc-11e4-be49-001517168acc  ONLINE       0     0    15  (repairing)
       gptid/584a4dcc-2bfc-11e4-be49-001517168acc  ONLINE       0     0     9  (repairing)
       gptid/1e956fec-cb65-11e4-8b23-000c29dbe1ad  ONLINE       0     0     9  (repairing)
       gptid/abd7d2b7-63cf-11e4-8721-000c29dbe1ad  ONLINE       0     0    13  (repairing)
    logs
     mirror-2                                      ONLINE       0     0     0
       gptid/9ed5c1df-4a60-11e4-ba6b-000c29dbe1ad  ONLINE       0     0     0
       gptid/ff76c369-4a5f-11e4-ba6b-000c29dbe1ad  ONLINE       0     0     0
    cache
     gptid/ed739feb-040f-11e6-8254-7f15800a1674    ONLINE       0     0     0
     gptid/edf42c1f-040f-11e6-8254-7f15800a1674    ONLINE       0     0     0

errors: 48 data errors, use '-v' for a list

Going to have to stop the scrub and do a memtest, I guess. Damn that will take long with all this RAM.

 

[mattlach]

Well, I'm leaning very strongly towards this being a KVM issue at this point.

I exported the pool out of my FreeNAS VM, shut the VM down, and imported the pool directly on the host using ZFS for Linux, and am now rescrubbing it.

The scrub is going about 8 times faster than it did in the VM (800MB/s), and I'm about 1TB in, and have not had a single checksum error yet.

This is a bummer. I'm already so invested in Proxmox/KVM now that I might have to abandon using FreeNAS in favor of mounting the pool on the host, and forwarding the mount location through as a bind mount to an LXC container and do all of my NFS, SMB and AFP exports manually, which is a huge pain

 

[mattlach]

This is a bummer. I'm already so invested in Proxmox/KVM now that I might have to abandon using FreeNAS in favor of mounting the pool on the host, and forwarding the mount location through as a bind mount to an LXC container and do all of my NFS, SMB and AFP exports manually, which is a huge pain

Hmm. Is it possible to just mount a local host folder in a FreeNAS vm, and use FreeNAS just for the easy management of network shares?

 

[jgreco]

Well, I'm leaning very strongly towards this being a KVM issue at this point.

I exported the pool out of my FreeNAS VM, shut the VM down, and imported the pool directly on the host using ZFS for Linux, and am now rescrubbing it.

The scrub is going about 8 times faster than it did in the VM (800MB/s), and I'm about 1TB in, and have not had a single checksum error yet.

This is a bummer. I'm already so invested in Proxmox/KVM now that I might have to abandon using FreeNAS in favor of mounting the pool on the host, and forwarding the mount location through as a bind mount to an LXC container and do all of my NFS, SMB and AFP exports manually, which is a huge pain

Sorry to hear it. ESXi has been the only known-to-be-stable hypervisor for FreeNAS for years. https://forums.freenas.org/index.ph...ide-to-not-completely-losing-your-data.12714/

 

[joeschmuck]

This weekend I am doing my "I got tired of VMWare and migrating to Proxmos instead" project.

Maybe you will have to re-evaluate your original thoughts. I'm running on the free ESXi 6.0U1b version (6.0U2 is out now) and I use vSphere Client to manage it. I'd certainly like to manage my VMs using vCenter but I'm not going to pay a huge amount of money for that when I can just do a few things by hand if needed. I have been extremely happy with ESXi 6.0 to date.

Jumping back to your system setup, you upgraded form FreeNAS 8 to FreeNAS 9.10, a pretty large jump obviously so did you also upgrade the LSI controllers to the P20 firmware to maintain compatibility?

 

[mattlach]

Maybe you will have to re-evaluate your original thoughts. I'm running on the free ESXi 6.0U1b version (6.0U2 is out now) and I use vSphere Client to manage it. I'd certainly like to manage my VMs using vCenter but I'm not going to pay a huge amount of money for that when I can just do a few things by hand if needed. I have been extremely happy with ESXi 6.0 to date.

Yeah, I was really chafing using VMWare. It was a number of things, but the last straw was that there is a system breaking bug in 6.0 that hard crashes the server in certain circumstances, and I just could not get my system to upgrade to 6.0u1a (which fixes it) or 6.0u1b.

I tried every hack out there on how to get the update to install on my free version server, but it refused to do so, claiming my license was ineligible for the update, even thriough free licenses are supposed to qualify for updpates after 6 months.

All the arbitrary limitations (for instance, the lack of 802.3ad link aggregation, forcing me to use static linking, or what they call "route over IP hash", when these features are available freely in the open source code base they build their releases on. Their collaboration with hardware vendors (like nvidia) to prevent passthrough of consumer versions of their hardware, forcing people to buy expensive enterprise models also left a bad taste in my mouth.

The fact that I had to manage my ESXi server using a buggy client, that only runs in Windows (meaning, I had to run a virtualbox Windows VM on my Linux desktop in order to manage my linux-based ESXi server) was also a huge pain.

Then I read about Proxmox and how it combines both full virtualization using KVM, and LXC containers (which use much less system ram than a full blown VM) and I was pretty much sold. I like that Proxmox also essentially runs on top of a Debian install, which I am very familliar with based on my years of using Debian based Linux distributions (Ubuntu, followed by Mint Cinnamon edition after Unity reared its ugly head). The plattform independent web management interface is free, does not require some sort of Vcenter install, and just works. Thus far the FreeNAS complication has been the only problem with it. it forces me to do the file server thing manually, which is a pain, but once set up, it's done. (I'm in *nix user & group permission over NFS hell right now, and am becoming increasingly familliar with the lines that need to go into my /etc/exports configuration file)

Jumping back to your system setup, you upgraded form FreeNAS 8 to FreeNAS 9.10, a pretty large jump obviously so did you also upgrade the LSI controllers to the P20 firmware to maintain compatibility?

I started out on 8.3 (I think? can't remember) originally years ago, but it's not as if I let it sit until now. I gradally upgraded to each incremental release, and yes, when I went to 9.3 and the mpt2sas module was bumped up to p20, I reflashed my controllers to match, so that is not the problem. Thank you for your thoughts though.

I'll miss FreeNAS and it's easy management of the complex network file sharing setups and permissions, but ZFS itself is fairly simple to manage from the command line.

The biggest nuisence will be to get rid of all the Swap partitions on my drives in my pool :p I figure I'm going to offline them one by one, blow out the partitions and dd /dev/zero over the entire drives (to get rid of the ZFS drive label) and then replace the full drive back into the pool, one by one and grow it to reclaim that space. (it's only some 24GB over my 48TB 12 drive pool, but still it bothers me).

For whatever reason, the "zpool labelclear" command has never worked for me, forcing a very time (and write cycle if an SSD) consuming dd of the entire drive to to get rid of the labels, as I believe there are 4 in total spread over the drive (two close to the beginning, and two close to the end)

I wish the FreeNAS installer had offered an option on install to use a separate swap drive, and not put swap partitions on all the disks.

 

[joeschmuck]

Sorry to hear about the problems. I know I'm new to ESXi but just in case this works for you, assuming you even want to attempt it, here is the process I use to check for, download, and apply an upgrade. So far it's worked like a champ. Note that you must have a VMWare account and if you were able to download the ESXi server then it forced you to create an account. I tried the "normal" paid upgrade process and it of course fails but the CLI works fine for me. I actually have all this data in my calendar and once a month it reminds me to take a look to see if there is an update I should get.


Check VMWare for ESXi patches.

https://my.vmware.com/web/vmware/login

1. Go to the “More Available Free Trials” section and select View All.

2. Roll down to the bottom of the page and select vSphere Hypervisor (64 bit).

3. You should see your license and then the binaries. Open the binaries and check to see if they are newer, then download.

4. Do a Google search for “vmware esxi 6.0U2” (example) for the new version and see if you need to apply the update.





How to install the patch.

https://kb.vmware.com/selfservice/m...nguage=en_US&cmd=displayKC&externalId=2008939


Instructions:

1) Place ESXi into Maintenance Mode

2) Download patch and place zip file into datastore/patch (I created a directory called "patch" of course)

3) SSH into 192.168.1.41 (my ESXi server) and log in as root

4) Install the patch with this command: esxcli software vib update -d "/vmfs/volumes/datastore_HDD/patch/PatchName.zip"

5) Verify the patch was installed: esxcli software vib list

6) Reboot the system: reboot

7) Remove ESXi from Maintenance Mode

8) Cleanup by removing the zip file from patch directory

 

[cyberjock]

The biggest nuisence will be to get rid of all the Swap partitions on my drives in my pool :p I figure I'm going to offline them one by one, blow out the partitions and dd /dev/zero over the entire drives (to get rid of the ZFS drive label) and then replace the full drive back into the pool, one by one and grow it to reclaim that space. (it's only some 24GB over my 48TB 12 drive pool, but still it bothers me).

I'm assuming you know why FreeNAS uses gptids and not disks or partition ids directly? I'm assuming you also know that there may (or may not) be consequences for ZFS on whatever OS you are using? If not, you shouldn't be doing that stuff until you do your research and understand those questions and the answers...

 

[mattlach]

I'm assuming you know why FreeNAS uses gptids and not disks or partition ids directly? I'm assuming you also know that there may (or may not) be consequences for ZFS on whatever OS you are using? If not, you shouldn't be doing that stuff until you do your research and understand those questions and the answers...

Yes, I am very aware of the wisdom of using GPTID's (or at least some form of unique ID) so that the devices don't move around on reboot and confuse ZFS.

When I first imported the pool into Linux I was alarmed to discover that by default it chose to use the direct disk devices (/dev/sd*)

The first thing I did (after completing my scrub and repairing the pool) was to export it again, and reimported it using /dev/disks/by-id.

I would have preferred using UUID's, but none of the ZFS devices had UUID's and I didn't know if there would be any consequences by adding them manually to devices in an existing pool. Turns out all of my disk ID's contain their serial number, so for all intents and purposes /by-id/ should be equally unique, so that';s what I did.

Looks like this now:

Code:

# zpool status zfshome
  pool: zfshome
state: ONLINE
  scan: scrub repaired 0 in 7h25m with 0 errors on Wed Apr 27 05:55:41 2016
config:

        NAME                                             STATE     READ WRITE CKSUM
        zfshome                                          ONLINE       0     0     0
          raidz2-0                                       ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4EK8ZSX37     ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4EK8ZS6DS     ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4EF24PKAY     ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4EE3LX9RV     ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E5UJ0PX5     ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4EFP2610U     ONLINE       0     0     0
          raidz2-1                                       ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1636111     ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1647062     ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1646218     ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1646088     ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E5PEDRTY     ONLINE       0     0     0
            ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E5X4NKU1     ONLINE       0     0     0
        logs
          mirror-2                                       ONLINE       0     0     0
            ata-INTEL_SSDSC2BA100G3_BTTV425101D5100FGN   ONLINE       0     0     0
            ata-INTEL_SSDSC2BA100G3_BTTV4253038A100FGN   ONLINE       0     0     0
        cache
          ata-Samsung_SSD_850_PRO_512GB_S250NXAH313648V  ONLINE       0     0     0
          ata-Samsung_SSD_850_PRO_512GB_S250NWAG822223D  ONLINE       0     0     0

I appreciate the heads up though!