LSI 9300 PCIe passthrough: 9.3 boot stucks, but 10.1M1 works !!! NEED HELP !!! (Proxmox/KVM way)

[m00nraker]

Hi guys.

I'm new to this forum and I would like to jump on the train and wanted to setup a virtualized FreeNAS. Please don't put me in a hole and throw stones on me. English is not my native language, so I try to do my best. First I have to say, that I read a lot about setting up FreeNAS in the last weeks. I read all the recommendations and guides (e.g. key concepts, or "guide...to not completely losing your data") here and I hope I am aware what I'm doing. FreeNAS natively runs fine on my hardware but this is not my way.

So I decided to go the PCIe-passthrough way with an additional LSI HBA controller in IT-Mode.

I did also made a post in the Proxmox-forum, but I think the problem is related to FreeNAS itself and not to Proxmox as I thought before. I also hope that I don't miss a forum post that could solve my problem. Then forgive me.

Code:

Hardware:
Supermicro X10SLH-F
Intel Xeon E3-1275Lv3
32GB Samsung ECC-RAM (4x8GB sticks)
Host Bus Adapter LSI 9300-8i (in IT-Fimware mode, latest Bios and Firmware updated)
HDD: 2x Samsung 850 Pro 256GB SSD
HDD: 3x Seagate 4TB NAS HDD (= total 12TB)
Intel I350 Quad NIC (in addition to the 2 Intel I210 onboard NICs)
Digital Devices Cine S2 Twin-Sat-Card

# Both SSDs are attached directly to the Supermicro SATA onboad ports
# All other SATA Seagate 4TB drives are attached to the LSI 9300 HBA (IT-mode = Raid functions disabled).
# Supermicro BIOS settings: AHCI enabled for SATA, Intel-Software-Raid disabled for all 6 onboard-ports.
# LSI-Controller-BIOS: Adapter BIOS disabled (= all boot-functions disabled), so HBA only (no booting from attached drives).

• Install Proxmox 3.4 with ZFSonLinux as the bare-metal hypervisor for virtualization
  (I know ESXi is preferred here, but I fall in love with Proxmox and wanted to go the KVM/QEMU way)
• During Proxmox setup I configured both SSDs as a redundant Raid 1 ZFS-pool, so in this ZFS-pool all VM's will find their place alongside Proxmox itself, everything on SSD storage is under control by ZFSonLinux.
• Under Proxmox create a new VM for FreeNAS with FreeNAS-9.3-stable-ISO as the CD-ROM boot medium for the VM (I followed the guidlines here. FreeNAS gets 16GB RAM. For the first run, I don't configure any PCIe-passthrough, so this is a relatively normal setup.)
• After setup, FreeNAS-9.3-VM boots normally and eveything is fine.
• After testing around a little bit I stop the VM and get into the Proxmox config-file for FreeNAS-VM to enable vt-d PCI-Express passthrough for the LSI controller. I followed the PCIe-passthrough guidelines for Proxmox (wiki pages). I also enabled Intel iommu=on in Proxmox-Grub, blacklisted mpt3sas module on host, etc.

Code:

lspci on the Proxmox host:
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
...
01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02

So 01:00.0 has to be to passthroughed to the VM. For that this is the relevant part of the VM config-file:

Code:

hostpci=01:00.0,pcie=1,driver=virtio
machine: q35
...
scsihw: virtio-scsi-pci
virtio0: zfs-rpool:vm-100-disk-1,cache=writethrough,backup=no,size=6G

(BTW, for the onboard SATA controller/SSD I use virtio. That works fine, also for other VMs)

• When I now start booting the FreeNAS VM with enabled PCIe-passthrough ,I get some issues.
• The boot process starts normal, the LSI-Controller shows up during POST in VM-Bios (passthrough seems to work).
• Then after booting a while the boot process stopped when FreeNAS detects the onboard SATA controller/ports.

Code:

Output from the console boot screen (also see attached files)
...
...
ahcich0: AHCI reset...
ahcich0: SATA connect timeout time=10000us status=00000000
ahcich0: AHCI reset: device not found
ahcich1: AHCI reset...
ahcich1: SATA connect time time=0us status=00000113
ahcich1: AHCI reset: device found
ahcich1: AHCI reset: device ready after 0ms
ahcich2: AHCI reset...
ahcich2: SATA connect timeout time=10000us status=00000000
ahcich2: AHCI reset: device not found
ahcich3: AHCI reset...
ahcich3: SATA connect timeout time=10000us status=00000000
ahcich3: AHCI reset: device not found
ahcich4: AHCI reset...
ahcich4: SATA connect timeout time=10000us status=00000000
ahcich4: AHCI reset: device not found
ahcich5: AHCI reset...
ahcich5: SATA connect timeout time=10000us status=00000000
ahcich5: AHCI reset: device not found

Booting gets stucked then.

(I get the same result, when booting the VM into the installer-ISO with previously enabled PCIe-passthrough.)

So what's the problem? I really don't know. In the first run I thought I had some issues with my Proxmox setup (IRQs, settings, whatever). I tested around and I go through the Proxmox forum and all the wikis, read a lot, and so on. But with nothing I had success. Then I changed a lot in the Supermicro Bios-Settings. All I did, no success, even when switching from AHCI to IDE. Then I had the idea, to try the latest FreeNAS 10.1 Milestone. And voila: success. I configured the VM to use 10.1-ISO-Installer and I started the VM with passthrough enabled. All hardware gets detected at the first run, the onboard SATA controller, the passthrough'ed LSI and all attached HDDS. After finishing setup, the FreeNAS-10.1M1 boots normally and eveything was detected.

I think there must be an issue with the FreeNAS-9.3 stable.

Before I start a bug-report for 9.3, has anybody any clue what I can do to enable PCIe-passthrough with 9.3 to work with my setup or at least what I can do to workaround the problem? Is it really FreeNAS related or could it be possible, that a setting in my hypervisor setup has to be changed?
I read something about the SATA timeout in the forums, and other people also have this prolem, but not related to pcie-passthrough. But I only run into trouble, when PCIe-passthrough was enabled before. 10.1M1 doesn't have these problems. 10.1 is not for production. So this is not an option for me.

I attached some files with additional information:
# lspci -vv on Proxmox host
# FreeNAS /var/run/dmesg.boot for 9.3 without passthrough and for 10.1 with passthrough
# qm monitor output for running VMs (9.3 and 10.1)

Please, I need some advice, help, whatever....? Thanks for any tipps and don't anger about my basic english skills.

Kai

 

[mjws00]

Interesting that 10.1 functions nicely. Does vanilla freebsd 9.3 run nicely under proxmox with the 9300 passed through? Does 9.3 run nicely with a different hba passed through?

We don't see many successful proxmox rigs around here. So I'm interested to see if an expert will chime in. I'd be interested in isolating that hba as the drivers are fairly new. Also isolating BSD 9.3 generic issues with proxmox. You could easily be hitting a known incompatibility.

The good news is ESXi will run nicely on that gear. ;)

Sorry I couldn't help more. Your English is fine. The idea itself may have some fundamental flaws and a lot of unknown risks. Beyond testing the theoretical, I wouldn't trust data to this. The unknowns and potential for data loss seem to be showing themselves early, imho.

Interesting observations and troubleshooting so far.

 

[Zedicus]

in proxmox set the kernel for freenas to qemu64

on my LSI HBA cards i ended up wiping the bios.rom off of them as even disabling add-in boot rom support on the mainboard would not keep proxmox from occasionally trying to grab onto the HBA's bios and causing an error in the freenas VM.

(i run Freenas 9.2.x branch and im testing 9.3 branch on proxmox and with some minor tweaks it works beautifully.)

is your optical drive being handled as a virtual IDE drive? (defaults that way in proxmox) if yes, remove it and add a new virtual optical drive using virtual sata.

(i realize this post is old but KVM info is lacking)

 

[m00nraker]

Hi Zedicus, first thx for your reply. Yes, post is old, but I still really need help. Problem isn't fixed.

When I disable passthroughing of my LSI 9300 host adapter FreeNAS ISO boots fine and I can set it up in a VM. But I need the LSI passthroughed to the FreeNAS VM.

When I add these 2 lines to my VM config file, FreeNAS ISO stops booting:

Code:

hostpci=01:00.0,pcie=1,driver=virtio
machine: q35

So all in all I still haven't success in booting a virtual FreeNAS 9.3 ISO in Proxmox 3.4 with LSI 9300 pci-e passthrough enabled. I updated Proxmox to the latest kernel version, I updated to the latest Supermicro Bios and to the latest LSI adapter bios and firmware. I tried so many configuration options, but no success (also tried CPU type=qemu64).

is your optical drive being handled as a virtual IDE drive? (defaults that way in proxmox) if yes, remove it and add a new virtual optical drive using virtual sata.

No optical drives installed. Virtual sata one to load the ISO.

I disabled the LSI adapter bios, so that booting from LSI connected drives is not possible. My system boots from internal SSD connected to mainboard SATA controller. In the Supermicro bios I disabled option rom for pcie slot 6, where the LSI is installed.

So what exactly do you mean with:

i ended up wiping the bios.rom off of them as even disabling add-in boot rom support on the mainboard

When you get it running with pcie passthrough, can you please post your settings (special bios settings for this setup, your FreeNAS vm config file, hardware setup, ...)

Thank you man.
Kai

BTW: FreeNAS 10 boots fine with pcie passthrough enabled. So there must be an issue with 9.3. Confusing..

 

[Zedicus]

o.k. counter-intuitively we are not actually going to use pci-e passthrough. i know the card is on the pci-e bus but that is only a physical location. all devices on a computer are located in bus locations, even the on-board ethernet is on the 'pci bus' so it could be passed to a VM if needed.

proxmox is working on adding the headings and a newer kernel as an extension for PCIE VIDEOCARDS. that is the Q35 addition. it is not needed for HBAs, NICs, heck even sound cards and USB3 controllers can be physically pci-e and still pass through using just pci passthrough.

then on the HBA's themselves i ran a set of commands similar to doing a firmware update that prepped the cards for use as HBA's in passthrough. i started with
sas2flsh -o -e 6
this command wipes the cards firmware and bios, but leaves the SAS address.
then i did
sas2flsh -o -f 2118IT.bin
this command writes the firmware. normally they recomand a command that looks like 'sas2flsh -o -f 2118IT.bin -b mptsas2.rom' the -b with the rom file is the bios for the card. EVERY install i have seen using LSI products on KVM has required the bios to be removed from the card as no other attemted work around would allow the VM to boot. they all would hang at some point. similar i think to what you are describing but there are some other potential issues that need checked.

also i am still on the stock kernel in proxmox 3.3.

attached is a screen grab of my config because of the remote nature of my host.

http://pve.proxmox.com/wiki/Pci_passthrough
on that page only about the top 3rd of the instructions apply to passing through an HBA or NIC. actually stop when you get to 'PCI EXPRESS PASSTHROUGH'

 

[m00nraker]

@Zedicus
Thx a lot for your post. With that information I can get FreeNAS 9.3.x running on Proxmox in the first step. Setup detects the passthroughed LSI HBA and all connected drives, even with complete smart data. But I run into some other issues with that setup and I haven't a solution at this time.
I need some more time for testing and will reply again in a few days. Hope that you can come back for a little bit discussion about this topic.

One thing I would comment now is your way to erase the bios from the LSI HBA:

I didn't use sas2flsh. I used sas3flsh (DOS version on USB stick) for ersasing the flash memory and to flash back the firmware file without bios. Running this tool terminated with PAL error in DOS. Then I used sas3flash (Linux version). That tool started but erasing flash memory (... -o -e 6) failed with an error. Last but not least the only way was using the EFI version of this tool -> sas3flash.efi. To run it, an EFI-shell is necessary. In that shell you can mount a FAT16/32 USB stick with that sas3flash.efi tool and flashing worked properly without any issues. The only thing is, before flashing you have to disconnect all attached drives from the LSI HBA. Neither DOS or Linux version of that tool worked without issues. You can also google on this. Other users reported the same issues.

 

[Zedicus]

@Zedicus
...

I didn't use sas2flsh. I used sas3flsh (DOS version on USB stick) for ersasing the flash memory and to flash back the firmware file without bios. Running this tool terminated with PAL error in DOS. Then I used sas3flash (Linux version). That tool started but erasing flash memory (... -o -e 6) failed with an error. Last but not least the only way was using the EFI version of this tool -> sas3flash.efi. To run it, an EFI-shell is necessary. In that shell you can mount a FAT16/32 USB stick with that sas3flash.efi tool and flashing worked properly without any issues. The only thing is, before flashing you have to disconnect all attached drives from the LSI HBA. Neither DOS or Linux version of that tool worked without issues. You can also google on this. Other users reported the same issues.

that is very true, it seems flashing these cards is less than ideal on server hardware. i always pull the LSI HBA out and stick it in my desktop, and boot to a USB FREEDOS thumbdrive created with RUFUS. from there i copy the utilities i need to the thumbdrive and do my flashing. also you sas2flsh or sas3flsh is based on the controller so yes, use the appropriate version.

server hardware will sometimes prefer EUFI, or have onboard SAS controllers. or other things that will cause issues during the flash.

Proxmox is not an EASY road for hosting freenas but the completed system is worth the effort.

 

[topprospect]

@m00nraker,

I had your *exact* issues--9.3 freezes on boot when passing through my 9201-16i. (FreeNAS 10 M1 would boot, as you saw). Removing "machine=q35" and the "pcie=1,driver=virtio" options from the hostpci line allowed FreeNAS 9.3 to boot properly.


@Zedicus,

Wow, thank you for your information in this thread. It was invaluable to my bringup.


Thanks for debugging this in public so that others benefit!


Here are a few things additional things I learned along the way (hope they might help someone else):

1. I used cpu=host in Proxmox so that FreeNAS gets access to the AES-NI instructions necessary for hardware accelerated encryption.
2. I ended up not erasing the ROM in my 9201-16i but I did disable boot capability in the LSI ROM application. So far I have not seen any problems. Another thing for people to try is to add the "rombar=off" option to the hostpci line in their vm conf file. This should disable the ROM without having to zero it out.
3. If you use Proxmox 4.0b1, note that you'll need to run "echo pci_stub >> /etc/modules" and reboot. Otherwise you get a "TASK ERROR: can't unbind/bind to stub pci device" error message.