SOLVED Hardware Recommendations for FreeNAS and ESXi

[brando56894]

Problem: ATA timeouts while passing through an IBM ServeRAID M1015 (flashed to P20 IT) to ESXi v6 with FreeNAS 9.10 Guest VM.

Symptoms: ATA Status Error 41 shows in FreeNAS console when another VM starts

Code:

Sep  4 23:24:41 freenas (ada0:ata0:0:0:0): DSM TRIM. ACB: 06 01 00 00 00 40 00 00 00 00 01 00
Sep  4 23:24:41 freenas (ada0:ata0:0:0:0): CAM status: ATA Status Error
Sep  4 23:24:41 freenas (ada0:ata0:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Sep  4 23:24:41 freenas (ada0:ata0:0:0:0): RES: 41 10 00 00 00 00 00 00 00 01 01

Resolution: add the following loader Tunables
hw.pci.enable_msi=1
hw.pci_enable_msix=0
hw.pci.honor_msi_blacklist=0

Just for reference sake, the Linux equivalent to add to the kernel line is mpt3sas.msix_disable=1

 

[Spearfoot]

First off, let me say that I know that this isn't generally recommended and you need pretty beefy hardware to achieve this but I figured I'd give it a try as a learning experience, but I'm running into some issues and I'm wondering if they're hardware or software issues. I'm only really using this until FreeNAS 10 comes out in (hopefully!) 6 months since I need a beefy virtualization solution and NAS all in one (FreeNAS 9.3 wasn't working out for me so I switched to Linux, haven't really tried bhyve in 9.10 yet). I know the devs test FreeNAS in VMware, there was a blog post about virtualizing FreeNAS, and also during my research I found a complete guide on how to virtualize FreeNAS but I'm still running into an issue, most likely with PCI Passthrough of my HBA.

I have ESXi 6 installed onto a 150 GB WD Raptor (since USB drives were giving me issues) and have the remaining space turned into a datastore, which hosts FreeNAS 9.10. I had no issues installing FreeNAS, nor do I have issues running FreeNAS by itself (besides a few random errors about NFS and Samba errors in the config, both protocols still work fine though), the problem starts when I boot up another VM.

For example, I have allocated and locked 32 GB of RAM for FreeNAS 9.10 and I have my IBM ServeRAID M1015 HBA passed through to the VM, I boot up and there are no errors. I transfer a huge file over SMB and there aren't any issues. I create a 6 GB archive (from a folder that exists on the share) over SMB and there aren't any issues. Yet, as soon as I start up my Usenet VM (contains Sickrage, Couchpotato, etc...) FreeNAS starts throwing CAM status errors (ATA 41 IIRC), which are visible in the vmkernel logs of ESXi. Initially I thought this was related to NFS since it seems fine with SMB, so I commented out all the shares in /etc/fstab and rebooted but still got the same errors. The Usenet VM resides in an iSCSI zvol on FreeNAS so I figured maybe there was too much IO so I moved the Usenet VM to another Raptor, which is attached directly to ESXi, but still received the errors!

There shouldn't be any issues with RAM because I've only allocated 8 GB of RAM to the Usenet VM, which would leave 23 GB of RAM completely untouched and available to ESXi (would actually be around 30 available since usenet only uses about 1 GB). I've researched this a bit and it seems that other people have been having these issues for a while (PCI passthrough of an HBA), and some people recommended changing the driver used for the passthrough, but I don't know how to go about doing that. The CAM Status errors also appeared if I passed through the HBA before I had installed FreeNAS, they would appear while the OS is installing to the desired drive (not connected to the HBA).

I have yet to complete further testing (in progress now), but my initial conclusion is that there is a problem with PCI passthrough, but why would it work fine with SMB but barf up errors at other points which seemingly have nothing to do with it? All in all, it does work since I had 5 VMs (FreeNAS, Plex, Usenet, Usenet 2( which contains another plex server) and my webserver) running, Usenet was successfully doing what it should be doing, data was being transferred to the appropriate pools and was accessible from Kodi. It just looked like the process was getting bottlenecked due to the controller screwing up. Other things would screw up, which I'm not sure if they're related or not, such as services failing to start in the usenet vm (it would hang indefinitely after sudo systemctl start sickrage, even once it was outside of the zvol, it worked perfectly minutes before and nothing had been changed.)

I'm currently in the process of setting up an Arch Linux VM w/ZoL to see if the problem is BSD, but for some reason it won't let me outside of my local network. I think I've scrambled my brain after days of messing with this for 10-12 hours at a time....

Interesting... and not in a good way!

I've set up FreeNAS as a VM on VMware 6.0 U2 on two Supermicro servers (see my systems below), neither with anywhere near the amount of RAM you have available, and it's always 'just worked'...

Is the system in question the one in your signature? Based on a Supermicro X10SDV w/64GB RAM?

Have you installed the board's latest BIOS? Have you flashed the IBM M1015s with the latest P20 IT firmware? Did you also flash them with the boot ROM code? (I recommend against installing the boot ROM code for HBAs used by FreeNAS under VMware.)

You mentioned locking the FreeNAS VMs memory, which is good. But did you customize its CPU resources in any way? I set mine to 'High' with a 1GHz reservation (see image). How many vCPUs did you give to the FreeNAS VM?

You mentioned @Benjamin Bryan's excellent blog post... did you set up a separate 'virtual' storage network on its own switch the way he described? I use his approach with good results.

Good luck! (And please let us know about any solutions you find!)

 

[brando56894]

Yes, the system in question is the one in my sig, it's my home server. I haven't touched the BIOS on it, I just got it a few months ago and went with the "if it ain't broke, don't fix it" approach, which was hard for me to do since it always leads to these scenarios (there was nothing wrong with my previous ZFS on Linux setup, I just decided to give ESXi a try and stupidly deleted my Arch install when I thought everything was ok).

I flashed P20 to the HBA when I first got it over a year or so ago when I was still using FreeNAS and haven't touched it since. IIRC I didn't touch the Boot ROM.

I've given it 4 CPUs. I thought I had customized it but I didn't apparently. I have 15 GHz available, only about a total of 5 is used running all VMs. Shares for RAM are set to high, no reservations. Shares for the OS vHD (datastore on Raptor, connected to a SATA port) and vZIL (datastore on Intel S3700, connected to a SATA port) are set to High. I didn't set up a storage network like he did because it would have added too much complexity to my network and I don't need it secured off in it's own subnet since I'm the only one that does anything.

From the looks of further testing, it looks like this is indeed a problem with ESXi and not FreeNAS since I created an Arch Linux VM and passed through the HBA (while the FreeNAS VM was off) and it wasn't showing. After messing around with configs in ESXi for awhile and it still not showing, I decided to look through Arch's dmesg and it seems that it's suffering from the same timeouts that FreeNAS was, only FreeBSD is more tolerant of it it seems because it at least allowed me to access it, whereas in Linux I get nothing at all.

early on I get a bunch of these

Code:

[    0.854918] pci 0000:00:15.3: BAR 13: no space for [io  size 0x1000]
[    0.854919] pci 0000:00:15.3: BAR 13: failed to assign [io  size 0x1000]

then this

Code:

[    5.535783] mpt2sas_cm0: MSI-X vectors supported: 1, no of cores: 4, max_msix_vectors: -1
[    5.536637] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 65
[    5.536640] mpt2sas_cm0: iomem(0x00000000fd2fc000), mapped(0xffffc90003188000), size(16384)
[    5.536641] mpt2sas_cm0: ioport(0x0000000000006000), size(256)

[    5.650438] mpt2sas_cm0: Allocated physical memory: size(7445 kB)
[    5.650441] mpt2sas_cm0: Current Controller Queue Depth(3307),Max Controller Queue Depth(3432)
[    5.650442] mpt2sas_cm0: Scatter Gather Elements per IO(128)

[   35.688679] mpt2sas_cm0: _base_event_notification: timeout
[   35.688729] mf:

[   35.688730] 07000000
[   35.688732] 00000000
[   35.688732] 00000000
[   35.688733] 00000000
[   35.688733] 00000000
[   35.688734] 0f2f7fff
[   35.688735] ffffff7c
[   35.688735] ffffffff
[   35.688736]

[   35.688759] ffffffff
[   35.688760] 00000000
[   35.688760] 00000000

[   35.688765] mpt2sas_cm0: sending message unit reset !!
[   35.690458] mpt2sas_cm0: message unit reset: SUCCESS
[   36.124071] mpt2sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:8592/_scsih_probe()!

Here's the full log in case anyone is interested: http://pastebin.com/VpUYsEsq

 

[Mirfster]

Perc H200 FTW?

 

[brando56894]

I think I may have found a solution!! According to this thread, the problem stems from MSI/MSI-X (pseudo/emulated PC interupts that support more than limited hardware interrupts), once those are disabled in the guest VM it seems to work. Many people in that thread have reported success with Linux and one guy mentioned that's how he fixed it in FreeNAS. I just added mpt3sas.msix_disable=1 to my Linux kernel boot parameters and was able to list ALL of my drives with fdisk -l this time instead of only the vHDD! I'll give it a try with FreeNAS now and report back.

Perc H200 FTW?

Huh? LOL

 

[Mirfster]

Huh? LOL

Instead of IBM M1015. I use the Perc H200 cross-flashed to LSI 9211-8i IT Mode and have it passed through to a FreeNAS VM without any issues. No tweaks or modifications required.

 

[brando56894]

Looks like MSI-X was indeed the issue. I booted my Usenet VM along side FreeNAS and no timeouts so far!

Edit: and the errors started up again when I started up the other 3 VMs which are stored in an iSCSI zvol in FreeNAS. I'm going to move them to their own drive outside of the HBA and hopefully the timeouts will be resolved once an for all.

Edit 2: I removed all my VMs and datastores and made a sole datastore on my S3700 SSD. I currently have FreeNAS and 2 other VMs running with no issues at all!

 

[toadman]

What is sitting on ada0 in the Freenas VM? Is that the vmdk that you are booting with?

 

[brando56894]

What is sitting on ada0 in the Freenas VM? Is that the vmdk that you are booting with?

Yea that would have most likely been the boot drive, which would have been a raptor formatted with VMFS (datastore).

I seem to have solved my issue by using those tunables I mentioned in my first post since I have 5 VMs in total running with no issues.

Sent from my AOSP on dragon using Tapatalk

 

[toadman]

Ok, good, glad you solved it. I asked because I had seen a similar issue with vmdk access on a very slow datastore.

 

[brando56894]

I just had another odd issue which seemed to cause a ton of the CAM Status errors over night: I went to sleep at about 2 AM watching media from the NAS, I went to check on things when I woke up around 11 am and found that my sites weren't accessible. Three (Plex server sharing out minimal multimedia, FreeNAS and Usenet) out of my five VMs had "questions" stating that their disks didn't have enough space and that I should free space on the volume. First off I didn't know if this meant the vHDD or the datastore itself, to further complicate things both the vHDDs and the datastore had significant amounts of space left (FreeNAS has about 7 GB free on it's vHD, Plex has 9 GB available, and Usenet has 13 GB available), meanwhile the datastore that they reside on said it had about 40 GB available! So I just moved the FreeNAS VM to the Raptor that ESXi is installed to since that has nothing else on it and FreeNAS should be the least impacted since it largely runs in RAM. Lets see if this setup remains stable for more than 24 hours hahaha

 

[toadman]

It seems clear whatever datastore you were using is too slow. I suspect the errors are because of simultaneous access by your VMs to their vmdk files. That datastore may be failing. You mentioned you have the local datastore on a raptor where you moved your FreeNAS vm. What is the datastore you were using previously? It's local (i.e. DAS on the esxi host) or networked? If networked, is it iscsi or NFS?

 

[brando56894]

I had all of them on an Intel S3700 connected to the SATA 3 bus. I still have all the other vHDDs on there but FreeNAS is now on the raptor and everything has been stable for about the past 24 hours.

Sent from my AOSP on dragon using Tapatalk

 

[toadman]

Odd. I would think the S3700 would easily handle it. Unless you had some weird access density issue, but I can't see how that would happen for a datastore. You might check that S3700, cables, etc.

 

[brando56894]

Yea that's what I figured too. I guess 5 VMs and the ZIL (what I had originally purchased it for a year ago, it was far underused since it's about 100 GB) was too much I/O for it.

I'm currently drooling over an idea I have right now, just gotta get the money: 512 GB Samsung 950 Pro NVME attached to the M.2 slots on my motherboard and using it for 2 L2ARCs, all vHDDs and also install ESXi on it. It's only about $300. That would make it pretty beastly and wouldn't have any problems at all with I/O. Another $600 and I could max it out at 128 GB! Talk about a monster of a home server hahaha

I gotta add more storage to my pools, 10 TB in my main pool isn't enough haha

Sent from my AOSP on dragon using Tapatalk

 

[Spearfoot]

Yea that's what I figured too. I guess 5 VMs and the ZIL (what I had originally purchased it for a year ago, it was far underused since it's about 100 GB) was too much I/O for it.

You used a single SSD as both a datastore and a SLOG device? That explains the problems you've been having. A SLOG device should always be a dedicated device.

 

[brando56894]

Ah! But my pool doesn't do that many sync writes so it didn't think it would have been an issue, I've been checking on it often and it writes a few hundred KBs every few hours.

Sent from my AOSP on dragon using Tapatalk

 

[toadman]

You used a single SSD as both a datastore and a SLOG device? That explains the problems you've been having. A SLOG device should always be a dedicated device.

Well, it might explain it, but should be confirmed by looking at traffic data. I don't understand how it's possible to use it for an SLOG in a FreeNAS vm and as a datastore for ESXi unless the datastore is networked. And that could explain the issues he was seeing (admittedly it would just be exacerbating the problem of a non-dedicated SLOG), depending on the network config (esp if it's a 1Gbps, though if the vms are on the same host as freenas it could be a 10G virtual network).

 

[toadman]

Ah! But my pool doesn't do that many sync writes so it didn't think it would have been an issue, I've been checking on it often and it writes a few hundred KBs every few hours.

In which case you may not need an SSD SLOG at all? You could try repurposing it as a dedicated datastore for ESXi and put the vmdks on that thing?

 

[Spearfoot]

My guess is that he partitioned the SSD, then used one partition for the SLOG and another for the datastore. But think about the inherent problem of ZFS attempting to buffer data to the SLOG at the same time a VM is doing I/O when both are on the same device!