Is 'rombar=1' important for TrueNAS-VM

[saveZFS]

Hallo,
I try to passthrough a LSI 9300-8i from a Proxmox host to a TrueNAS-VM. The HBA is flashed to IT-Mode. And the IT-Mode is shown in the boot menu of the host.

If I set 'rombar=1', then I get an error:
"
Unable to load Avago Technologies MPT BIOS
MTP BIOS Fault 02h encountered at adapter PCI(01h,00h,00h)
Firmware Fault Code: 2667h

Press any key to continue ...
"
When I then press a key it starts and everything seems fine.

If I set 'rombar=0', then everything works fine without pressing any key.
But I'm a little bit insecured, because if I check in TrueNAS-VM, if the HBA is running in IT-Mode by sas3flash I can't see the IT-Mode.
I only see:
"
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
"
So the question is, if the HBA is also in IT-Mode, if 'rombar=0'?
And is it important for TrueNAS to see the whole firmware (rombar=1)?

 

[jgreco]

Welcome to the forums.

First, please note that vague questions like these are likely to result in low-quality responses that are not particularly helpful. You have posted a question along the lines of "my vehicle won't start". We don't know if you've got a car, truck, motorcycle, airplane, helicopter, or bicycle, or whether it's fueled by gasonline, diesel, hydrogen, or electric battery. The lack of any relevant detail makes this extremely hard to come up with good answers.

As a result, and as required by the Forum Rules, conveniently linked at the top of every page in red, we ask that you post a reasonably detailed hardware manifest of your system, versions of both Proxmox and TrueNAS involved, etc. For the hardware manifest, we don't usually need detail down to the exact brands, but telling us that you have a Supermicro X10SRL with an E5-1650v3 along with 4x 16GB sticks of DDR4-2400 ECC, etc.

Some general points here in response to your vague question:

a Proxmox host

Proxmox has not proven itself to be a reliably stable hypervisor platform for FreeNAS/TrueNAS. It is a new entrant into the hypervisor game and its PCIe passthru is still in the "experimental" stage. See this wiki:

PCI Passthrough - Proxmox VE

pve.proxmox.com

where it says

"PCI passthrough is an experimental feature in Proxmox VE"

It's recommended that you follow the guidance in

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

for proper virtualization.

Version 16.00.00.00 (2017.05.02)

This is the wrong firmware version, although to me it looks like you just typed "sas3flash" without any arguments, so that's the version of the flash utility, not the card.

Please make sure you have installed 16.00.12.00 on the card.

LSI 9300-xx Firmware Update

Hey Community, If you are using an LSI 9300 HBA with FreeNAS or the soon-to-be TrueNAS CORE, you may experience some performance issues causing the controller to reset when using SATA HDDs. After working with Broadcom, we’ve come up with a...

www.truenas.com

So the question is, if the HBA is also in IT-Mode, if 'rombar=0'?
And is it important for TrueNAS to see the whole firmware (rombar=1)?

This is ambiguous. It looks to me like what may be happening is that the PC BIOS ROM portion of things is barfing. There is ALSO a thing called the firmware, which is the program that runs ON the HBA's CPU to provide all the I/O capabilities. Both things can be referred to as "firmware". The card firmware is probably running just fine.

You can check the firmware version of the card by running "sas3flash -list" and looking in the output.

Code:

# sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

        Controller Number              : 0
        Controller                     : SAS3008(C0)
        PCI Address                    : 00:03:00:00
        SAS Address                    : 5003528-0-2ab1-eb01
        NVDATA Version (Default)       : 0e.01.30.28
        NVDATA Version (Persistent)    : 0e.01.30.28
        Firmware Product ID            : 0x2221 (IT)
        Firmware Version               : 16.00.10.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : LSI3008-IT
        BIOS Version                   : 08.37.00.00
        UEFI BSD Version               : 18.00.00.00
        FCODE Version                  : N/A
        Board Name                     : LSI3008-IT
        Board Assembly                 : N/A
        Board Tracer Number            : N/A

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.

The card CPU firmware MUST be running in order for the HBA to work. I suspect that what's broken for you is the PC BIOS firmware, which may be broken because you're using a dodgy hypervisor, or passthru is broken in some manner.

 

[saveZFS]

First my HW is:

• E1245v5
• 64GB ECC
• ASUS P10S-E/4L
• LSI 9300-8i

Please make sure you have installed 16.00.12.00 on the card.

Thank you for the tip with 'sas3flash -list'. - Works fine.
Firmware 16.00.10.00 is installed. The only diffrence to your HBA is the BIOS. In my case is 08.37.02.00.

It's recommended that you follow the guidance in

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

for proper virtualization.

Thank you.
I cam from ESXi and my master server is still running with it.
But for my backup server I wanted to try some freeware. Because ESXi Free is extremly limited in kind of backuping the VMs. :(
But I didn't know, that with PRoxmox I can't get a stable environment for my TrueNAS-VM. :(
So I think I must go back to ESXi! :(

 

[jgreco]

16.00.10.00 is probably fine. There are some minor bugs with specific devices. I think a lot of my cards are running .10. as well.

Because ESXi Free is extremly limited in kind of backuping the VMs. :(

Indeed. You can use a tool like ghettoVCB, or you can do backups from within the VM's. This is a deliberate shortcoming of ESXi. They want you to buy the licensed copy.

If you are doing this in an environment that could credibly be considered a home lab, you can pick up VMware User Group Advantage EvalExperience licensing for $200/year. This includes non-production licensing for most VMware products. For ESXi, I believe it's for three dual-socket hypervisors and a copy of vCenter Server.

 

[saveZFS]

ndeed. You can use a tool like ghettoVCB, or you can do backups from within the VM's. This is a deliberate shortcoming of ESXi. They want you to buy the licensed copy.

Is it possible, that in a few month probably Proxmox will be an recommended alternative for running a TrueNAS-VM?
Or is the way for Proxmox very long till it could be a recommended alternative to ESXi?

 

[jgreco]

that in a few month probably Proxmox will be an recommended alternative for running a TrueNAS-VM?
Or is the way for Proxmox very long till it could be a recommended alternative to ESXi?

People have been bothering me about this since literally the day after PCI passthru was announced on Proxmox.

Since I do not develop either product, my yardstick for this is actually YOU; YOU arrived here, and indicated you had problems. This has been semi-broken for several years, and it seems reasonable that this will be at least as difficult to permanently resolve as it was for ESXi, which is developed by a company with a large staff of incredibly talented programmers and systems people -- they developed their own hypervisor stack, after all. Proxmox is mostly bundling pre-existing elements together into a nice package, so, if you want the pessimistic view, it could be a significant amount of time before this is truly stable. The developers of Proxmox are probably not throwing a team of people at resolving PCI passthru bugaboos.

From a historical perspective, I wrote the "Please do not virtualize FreeNAS in production" document after ESXi was probably a stable and viable virtualization platform, but appropriate hardware was hard to come by. We kept seeing people trying desperate things to virtualize FreeNAS, and failing, usually because they were using older Nehalem or Westmere gear, where it sorta-worked-mostly-didn't. I wrote the "Absolutely must virtualize FreeNAS" article as a test, a suggested golden path for successful virtualization, and that mostly worked out okay.

If and when a golden path for Proxmox becomes obvious, I'm fine with documenting where the pain points are. However, I am not holding my breath. The problem with hypervisors, NAS, ZFS, etc., are that you are really stressing out all the components in maximal ways. None of these things are easy, and 99.99% isn't good enough.

 

[saveZFS]

None of these things are easy, and 99.99% isn't good enough.

In case of storage, even for a backup server, you are absolutly right.
I think I will stay by ESXi and maybe in a few years proxmox could be a valid alternative for my homelab!